During the Heian period, Kūkai was traveling around Shikoku and discovered a hot spring here. The spring was known for its efficacy in curing all illnesses. In 815, Kūkai carved an image of Yakushi Nyorai and built a temple to house it a short distance from the current location. Over time, due to the benefit of the hot spring, the temple buildings came to stretch all the way to the nearby hills, and the sound of temple bells never stopped. But in the late 1500s, the whole area was burnt down by Chōsokabe Motochika. A century later, the temple was rebuilt at the present location.

Around the time that Chōsokabe was unifying Shikoku, the authorities in the Awa domain where Anraku-ji is located were concerned that there was no accommodation for pilgrims in the region, so they designated the temple as a place that would provide accommodation and meals to people passing through. Due to its healing hot spring, it was an obvious choice. Consequently, Anraku-ji has probably the oldest temple lodging in Shikoku. It has space for 250 people at a time.