This study aimed to investigate the effectiveness of teaching according to TRIZ theory in solving space geometry problems and in mathematical self-esteem (MSE) of eleventh grade students. The study followed the semi-experimental method and is applied on a random sample of eleventh grade students separated into two groups, an experimental group consisted of (22 students) taught according to TRIZ theory and controlled group consisted of (23 students) taught with conventional method of teaching during the second semester of the year (2017/ 2018). The researcher applie for his research, one is a test of space geometry problems which was been adjudicated for validity and reliability by a-Cronbach which reached (0.78). The second is a measurement of mathematical self-esteem (MSE) consisted of (30 questions) which is, also, adjudicated for validity and reliability using a-Cronbach which reached (0.79). That is besides using re-test and person correlation coefficient, which reached (0.74). Data processing was done, then, after making sure that the two groups are equivalent using independent samples t-test to study the significant differences between the average scores of controlled group students and that of their counterparts in the experimental group, besides counting Eita square. The results were like the following: 1-There is a statistic significant differences at the level (a=0.05) between the average scores of controlled group students and that of their counterparts in the experimental group in solving geometry space problems gone for the experimental group which were taught using TRIZ theory. 2-There is a statistic significant differences at the level (a=0.05) between the average scores of controlled group students and that of their counterparts in the experimental group in (MSE) measurement gone for the experimental group which were taught using TRIZ theory. According to the results, the researcher advised teachers to be trained and directed to apply TRIZ theory with their students in solving mathematical problems. He, also, suggested to do similar studies for the rest sections of mathematics.