Le Puy-en-Velay is a famously know historic town in the Haute-Loire department. I've been there a few days before Le Tour de France, and its beautiful! On top the highest point in the old city, there is a 22m statue made from 213 cannons taken at the battle of Sevastopol. The view from there is crazy, but its a steep walk so be prepared! The are around the city is mainly rural with a lot of natural points of interest, like the Cascade de la Beaume. Go there and open your eyes!