From the temple grounds there is a beautiful view of the Great Seto Bridge and the Seto Inland Sea. The area has prospered as a port since ancient times, and is known as the Front Gate to Shikoku. It is a sacred site that has been served by many famous high priests and monks. It is known as Yakuyoke Utazu Daishi and is the only Shikoku pilgrimage temple that belongs to the Jishu sect of Buddhism. According to legend, Gōshōji was founded by Gyoki in 725. Gyoki carved a 55-centimeter statue of Amida Nyorai (Buddha of Limitless Light and Life), enshrined it as the principal image of the temple, and named the temple Bukkōzan Dōjōji. The name Dōjōji in the temple poem is a remnant of this. In 807 Kobo Daishi visited the temple, carved a statue of himself, and made a prayer to ward off evil from the temple. This wooden statue of Kobo Daishi is still widely worshiped as Yakuyoke Utaizu Daishi (Kobo Daishi for Warding off Evil).

Rigen Daishi Shobo (832-909), the founder of Daigoji Temple in Kyoto, is said to have practiced asceticism at this temple during the Ninju era (851-854). During the Kanwa period (985-987), a monk named Eshin Sozu Genshin (942-1017), who laid the theoretical foundation of Pure Land Buddhism, dedicated a painting of Shaka Nyorai (the Historical Buddha) and built the Shakado in response to a vision. In 1243, when Dohan Ajari of Koyasan, the author of Nankai Ruroki and the founder of the Chūin School, was exiled, he used this temple as his temporary home. Ippen Shonin (1239-1289), founder of the Jishu sect of Buddhism, stayed at the temple in 1288 and spread the teachings of Igyo and Pure Land Buddhism. This makes Gōshōji unique among the 88 pilgrimage temples of Shikoku in that it is home to both the Shingon and Jishu schools of Buddhism. The teachings of Shingon Sanmitsu and Jodo Igyo are still handed down from generation to generation.