COVID-19 is an emergent disease with significant global concern, which might have a negative effect on quality of life. This study aimed to determine the impact of the COVID-19 pandemic on the quality of life of people (with or without COVID-19) attending primary health centers in A’Seeb Wilayat in Muscat, Oman. Methods: This cross-sectional study was conducted in four randomly selected primary health centers in A’Seeb Wilayat from 17 July 2021 to 31 January 2022. All Omani men and women, aged ≥ 18 years, regardless of their COVID-19 infection status, who were able to read and use online questionnaire, were included. The consecutive sampling method was applied. An online self-administered and validated Arabic version of the Short Form-12 was used to determine the impact of the COVID-19 pandemic on mental and physical quality of life. Results: A total of 701 participants were included with a mean age of 25.3 years. Two-thirds of the participants (n = 473, 67.5%) reported being physically affected by the COVID-19 pandemic (score of ≤ 50) and more than half (n = 392, 55.9%) had been mentally affected (score of ≤ 42). Univariate analysis revealed a significant association between physical impact and educational level, low family income, chronic diseases, and alcohol consumption. Gender, young age, being single, low income, and chronic diseases were significant risk factors for mental impact. Conclusions: Physical and mental impacts are very common during the COVID-19 pandemic. Several risk factors were identified. Public health programs need to be implemented to mitigate the negative impact of COVID-19 on quality of life.