Negoroji stands on Mt. Aomine, the main peak of Goshikidai. It used to be a gigantic temple with spacious grounds. Kobo Daishi noticed the presence of the Gochi Nyorai ( the Five Wisdom Buddhas) on the five peaks of Goshikidai, designated the temple as a place for esoteric Buddhist practice, and built a temple which he named Kezoin on Aomine. Later, when Kobo Daishi's nephew Chisho Daishi visited the temple, Chisho Daishi met Ichinose Myojin, the mountain's guardian deity, who told him to build temples in Bishamondani Valley, Rengedani Valley, and Goyadani, and to make a statue of Kannon Bosatsu (Bodhisattva Who Hears the Sounds of the World) from a tree in Rengedani Valley. Kobo Daishi carved a statue of Senju Kannon Bosatsu (Thousand-armed Bodhisattva Who Hears the Sounds of the World) and built Senjuin to enshrine the statue. Because the stump of this sacred tree emitted an aromatic fragrance, the names Kezoin and Senjuin were collectively called Negoroji (Fragrant Root Temple). Negoroji was a favorite of Emperor Go-Shirakawa (reigned 1155-1158). Later the temple was rebuilt by the feudal lord of Takamatsu and others, and at that time it was converted to the Tendai sect of Buddhism.

The legend of the temple is as follows:  a long time ago, there lived on Mt. Aomine a terrible monster, the Ushioni (Bull Demon), which ate people. The villagers asked Yamada Kurando Takakiyo, a master archer, to kill it. However, no matter how many times Takakiyo went into the mountains, the Ushioni did not appear, so Takakiyo prayed to the principal image of Negoroji for twenty-one days. On the last day, the Ushioni appeared and the archer hit him with an arrow in the mouth. He chased the fleeing Ushioni and found it dead in Jogafuchi, about two kilometers to the west. Takakiyo cut the horns off of the Ushioni and dedicated them to the temple. The horns are still preserved there. Images of the Ushioni are popular as amulets to ward off evil.