The objective of this paper is to analyze the performance of the rooftop photovoltaic (PV) system, taking into account its shading impact on the rooftop surface and the eventual cooling load of the building. This paper tests three approaches of the rooftop PV system: zero tilt angle flat PV configuration; PV configuration with a monthly adjusted tilt angle; and PV configuration with a dual-axis sun tracking system. Each of the PV configurations is optimized to do the following: minimize the self-shading among the adjacent arrays; maximize the rooftop surface shading to curtail the cooling load; maximize the net energy yield; and minimize the net levelized cost of energy (LCOE) of the PV system. The existing building model is developed in SketchUp Pro. The model is simulated in an EnergyPlus environment to calculate the building's cooling energy consumption with different shading scenarios in various PV configurations. Various rooftop PV configurations are designed and simulated in a System Advisor Model (SAM) to analyze the effect of self-shading of the adjacent PV arrays on the PV performance. The optimal distance between the arrays (for maximum net energy yield and minimum net LCOE) is found to be 1.5 m. The net LCOE of the optimal scenario is 5.247 ¢/kW h and 4.112 ¢/kW h for monthly tilt and dual-axis tracking arrangements, respectively. The economic surplus of the optimized system is 0.422 ¢/kW h and 0.258 ¢/kW h for the monthly tilt and dual-axis tracking arrangements, respectively, as compared to the ground-mounted system. © 2019 Author(s).