The occupation of the Lavardin site dates to Prehistory because the excavations show the presence of axes, arrows, and tools of the Neolithic. Inhabited by Gauls, the fortified site of Labricinum, became a Gallo-Roman village, as evidenced by the traces of a temple of Mercury.
Towards the end of the 8th century, the region became Christianized by Richemer, a monk from Tours. He settles first near the primitive castle, Turris dominica, the Tower of the Lord before founding a hermitage further, thus giving his name to the commune of Saint-Rimay. However, it is Saint-Julien, whom the chronicles quote as the first founder of a church.
Lavardin is fortified at the end of the tenth century as part of a fortification line built in the Loir Valley to fight the Normans according to the will of Charles the Bald. The Count of Vendome, Bouchard then gives up this stronghold to the Count of Anjou Foulques Nerra. One of the vassals of the Count of Anjou, Hugues d’Amboise, godson of Hugh Capet, comes into possession of this fortress and this seigneury at the end of the tenth century by marrying the heiress.
His nephew Solomon founded in 1030 the Priory Saint-Gildéric. In the eleventh century, Solomon II built in this fortress the first stone dungeon to protect the peasants of the lordship. They lived outside wartime in the caves carved into the rock. Indeed, for several decades the Seigneurie de Lavardin is at war with that of Montoire. It is during a battle between Hamelin de Montoire and Aimeric de Lavardin in the plain of Villavard that the legend is founded at the origin of the pilgrimage of the Black Madonna of Villavard.
During the war between Henry II of England and Philip Augustus, Richard the Lionheart came to put in 1188, the seat of the castle of Lavardin. The King of France came to lend a hand to the militia of Montoire and Lavardin who allied themselves and they force the English troops to lift the siege.
Around 1380, John VII, count of Vendome launches the reconstruction of the castle. It hosts in 1448 King Charles VII and Agnes Sorel. He is welcomed in Lavardin by the Count of Vendôme Jean VIII who finished the restoration of the castle. Since Lavardin, the king manages the reconquest of Le Mans – which falls on March 15 – and signs a truce with the English. The Count de Vendome, disgraced by Louis XI, made Lavardin Castle his last home, dying there on 1477.
In 1581, more than 300 inhabitants would be victims of an epidemic (of plague) before being in the conflicts between Protestants and leaguers. In 1589 Henry IV sieges the Catholic garrison Lavardin and he must send the Prince de Conti to make a second seat and resolve the situation. It is as a result of this decision that the castle is dismantled (1589) and that the stones are used by the inhabitants for their constructions. The estate was sold and forgotten for centuries.
In the nineteenth century, Napoleon III plans to rebuild the castle, but he prefers that of Pierrefonds. During the War of 1870, the military genius blew up three arches from the bridge to block the enemy and the Germans plundered the village. The bridge is restored by the troops of Von Moltke.