In the past decade, quick developments in nanotechnology have created quite a lot of prospect for the scientists and engineers to check up. Nanofluid is one of the amazing consequences of such progression. Nanofluids are engineered by suspending nanoparticles in traditional heat transfer fluids. Nanofluids are considered to offer important advantages over conventional heat transfer fluids. In the beginning of the thesis, the wide-ranging fundamental evolutions of nanofluids have been discoursed thoroughly by sketching out a gargantuan depiction of the diminutive biosphere of nanofluids through a brief review of some chronological foremost milestone, and potential applications and benefits of nanofluids. Also, different kinds of modeling and very important slip mechanisms of constructing heat transfer modeling of nanofluids have been discussed comprehensively. The amazing results of research and tremendous research opportunities have been found while reviewing. As a result, new mathematical equations have been established theoretically for electrical conductivity, and thermophoretic velocity, thermal diffusion coefficient as well as the mass flux equations in nanofluid which can be used for general transport of nanofluid modeling. The result of the thermal diffusion coefficient is justified by the experimental findings. One of the main objectives of this study is to investigate the convective heat transport within enclosures numerically. Hence, the basics of convective heat transfer are demonstrated briefly along with the standard fluid dynamics equations. A broad picture of convective heat transfer models in nanofluids that had been developed theoretically in recent years has been presented. The limitations of the existing models are manifested. A new mathematical model called nonhomogeneous dynamic model of nanofluids had been developed to tackle the limitations. For solving two-dimensional incompressible natural convective flow of nanofluids in an enclosure using nonhomogeneous dynamic model, the Galerkin weighted residual technique of finite element method is employed. The detailed procedures of this method over the nonhomogeneous dynamic model are discussed and calculated for the first time. In this study, considering research scope and practical applications, the circular enclosures have been simulated using the nonhomogeneous dynamic model, and the Galerkin weighted residual finite element method. The five series of the separate problems have been set up and successively solved. The problem of two-dimensional transient convective flow and heat transfer in quarter-circular, semi-circular, semicircular annulus and horizontal cylindrical annulus shaped geometry utilizing nanofluids have been inspected. Separate boundary conditions are adopted in each problem. Also, the heat transfer performances for different types of thermal boundary conditions have been examined. The numerical treatment of the singularity at the hot and cold junction has been studied. Different types of nanofluids have been considered for the analysis. The problem of convective heat transport within quarter-circle and horizontal cylindrical annulus has been investigated along with the oriented applied magnetic field. Different shapes of the thermal boundary of the geometry for the heat transfer performance have been analyzed. The effects of the sizes, shapes and amount of nanoparticle in the suspension and different pertinent parameters of the problem such as the thermal Rayleigh number, the solutal Rayleigh number, Hartmann number, magnetic field inclination angle have been analyzed. Comparisons of heat transfer enhancement are made with the numerical as well as the experimental data available in the literature. The results are displayed in terms of streamlines, isotherms, isoconcentrations, and the local and average Nusselt numbers. The results show that 1-10 nm size nanoparticles are uniform and stable in the suspension within the quarter-circular and semi-circular enclosure while 1-20 nm size nanoparticles are stable in an annulus shaped geometry. The external magnetic field and its direction control the flow pattern of nanofluid significantly. The average Nusselt number increases significantly, as magnetic field inclination angle, and Rayleigh number increases while it decreases with the increase of magnetic field parameter. The average Nusselt number increases significantly with the increase of the nanoparticle volume fraction as well as with different shape factors of nanoparticles, whereas it decreases remarkably with the increase of nanoparticles diameter. The dissemination of temperature gradient magnitude with respect to the solid volume fraction is reciprocally proportional to the dissemination of local Nusselt number. It is also found that the inner shape of the annulus significantly affects the thermal flow as well as the Nusselt number Finally, the thermal instability analysis for the natural convective heat transport in a horizontal nanofluid filled layer using nonhomogeneous dynamic model has been investigated. The solutions with respect to the wide-ranging parameters of the problems in the current study are tested whether they are stable or unstable. It is found that the system is eventually stable for the increasing values of thermal Rayleigh number and solutal Rayleigh number. The higher wave number assists the system to remain stable. Adding nanoparticles to base fluid increases the stable region of the system. The heat transfer of the system is unstable initially in a small period of time, and then it becomes stable in rest of times