Psychological distress caused by decreased physical activity (PA) is a growing concern among the elderly due to public health measures since the coronavirus disease (COVID-19). We aimed to (1) assess how public health restrictions impact PA, subjective well-being (SWB), and health-related quality of life (HRQoL) of community-dwelling elderly, and (2) investigate risk factors that lead to a decline in PA. Self-administered questionnaires assessed the changes in PA, SWB, HRQoL . Multivariate logistic regression analysis was performed to identify significant associated risk factors for decreased PA. Of 165 participants (valid response rate , 41.3% ; mean age , 78.5 ± 8.0 years), 47.3% became less active , 23.0% became more active, and 29.7% maintained PA levels . There was a significant decrease in SWB at baseline and follow-up after COVID-19 restrictions in the less active group (p <0.01). Higher levels of moderate or strenuous exercise/sports activity at baseline (odds ratio [OR], 1.12; 95% confidence interval [CI], 1.01-1.24), and lower mental component HRQoL scores at baseline (OR , 0.96; 95% CI , 0.93-0.99) were associated with an increased risk of decreased PA. Public health restrictions impact the PA of the elderly, especially those who had higher levels of exercise/sports activity and lower HRQoL before COVID-19 . Decreased PA was strongly associated with lower SWB.