This is the first meta-analysis focusing on elderly patients with mobility problems, physical disability and/or multi-morbidity. The aim of this study is to assess the effect of physical exercise therapy on mobility, physical functioning, physical activity and quality of life. A broad systematic literature search was performed in the databases PubMed, CINAHL, Embase, PEDro and The Cochrane Library. Relevant study characteristics were reviewed and meta-analyses using standardized mean differences (SMDs) were performed. The results show that physical exercise therapy has a positive effect on mobility (SMD final value: 0.18; 95% CI: 0.05, 0.30; SMD change value: 0.82; 95% CI: 0.54, 1.10) and physical functioning (SMD final value: 0.27; 95% CI: 0.08, 0.46; SMD change value: 2.93; 95% CI: 2.50, 3.36). High-intensity exercise interventions seem to be somewhat more effective in improving physical functioning than low-intensity exercise interventions (SMD final value: 0.22; 95% CI: −0.17, 0.62; SMD change value: 0.38; 95% CI: −0.48, 1.25). These positive effects are of great value for older adults who are already physically impaired. The effect on physical activity and quality of life was not evident and no definite conclusions on the most effective type of physical exercise therapy intervention can be drawn.