In this thesis, we have studied the basic equations of fluid dynamics for nanofluid modeling, the fundamental aspects of nanofluids including nanoparticles, base fluids and their production processes. The applications of nanofluids in science, engineering and nanotechnology are widely discussed. A detailed literature survey is carried out to explore the free convection flow and heat transfer in a right-angle triangular cavity filled with nanofluids. A numerical investigation is presented to study Buongiorno's mathematical model for unsteady natural convection flow and heat transfer inside a right-angle triangular cavity filled with nanofluids. The inclined wall of the cavity is maintained at constant cold temperature whereas various combinations of the thermal boundary conditions at the vertical heated wall are considered. The bottom wall is insulated. The cavity is permeated by an inclined uniform magnetic field and the effects of Brownian motion and thermophoresis are incorporated into the nanofluid model. The governing nonlinear coupled partial differential equations are made dimensionless using a suitable transformation of variables. The simulation is carried out through the pde solver Comsol Multiphysics with Matlab interface which uses Galerkin weighted residual finite element method. Comparisons with previously published work are performed and excellent agreement is obtained. The effects of various model parameters such as Hartmann number, Richardson number, the aspect ratio of the triangular enclosure and inclination angle of the magnetic field on streamlines, isotherms and isoconcentrations have been displayed graphically. The heat transfer augmentation for various combination of the pertinent parameters as well as various thermal boundary conditions have been done in light of the average Nusselt number from the vertical heated wall. The obtained results are discussed from the physical and engineering point of views. The results show that the heat transfer rate can be decreased with the increasing values of the Hartmann number but it can be increased by increasing the Richardson number. On the other hand, the rate of heat transfer can be controlled by only changing the inclination angle of the magnetic field. The obtained numerical results also indicate that the variable thermal boundary conditions as well as the variation of base fluid have significant effects on the flow and thermal fields. Furthermore, it is detected that changing the aspect ratio leads to increase the average Nusselt number. Finally, it is observed that the average Nusselt number of Fe3O4-Engine Oil (EO) nanofluid is much higher than other types of nanofluids which are studied in the present analysis.