This study aimed to examine the potential integration of generative AI in the mathematics curriculum as a supplementary tool for higher education students at the College of Science, Sultan Qaboos University (SQU). The study also aimed to find appropriate ways to use customized assessments to prevent academic dishonesty related to the use of AI tools like ChatGPT. A quasi-experimental design was employed using a convenience sample of 54 students. Mixed methods, including quantitative and qualitative approaches, were used to gather the study's data. Quantitative methods included in-class assessments (two exams and two assignments), while qualitative methods included observations and semi-structured interviews. The independent variable in this study was the use of ChatGPT by the experimental group in learning mathematics compared to the control group, which did not utilize ChatGPT, and the dependent variables included student performance. The quantitative analysis demonstrated a notable difference in performance between students who utilized ChatGPT and those who did not. Specifically, the experimental group outperformed the control group in general assessments, whereas the control group performed better in customized assessments. Moreover, the qualitative analysis of semi-structured interviews indicated that participants perceive ChatGPT as a valuable tool for enhancing their proficiency in basic mathematics. They expect ChatGPT's problem-solving ability to improve significantly when students accurately enter mathematical symbols and provide clear directions. Additionally, through observation, the researcher identified the obstacles that ChatGPT encountered when faced with particular mathematical problems, providing insights for educators on how to modify questions that generative AI tools like ChatGPT can solve into customized ones they cannot. This strategy could help deter academic dishonesty. However, the study acknowledges the need for further research on the long-term implications of integrating generative AI in mathematics education, focusing on advanced topics. Based on the findings, some educational implications and recommendations in this area are suggested.