Map
The 'modern' town of Kampaeng Phet lies to the south of the old walled city and the historical parks which contain the bulk of the ruins. All of the accommodations are in the modern city or across the river from it, so if you're based in Kampaeng Phet, you'll be staying in the newer part of the city and commuting to the sights from there.
There are a few old temples on the west bank of the Ping River, which was a separate ancient city in its own right called Nakorn Chum, but for the most part they are not much different from the main ruins.
Kampaeng Phet doesn't get many international tourists, so there isn't much of an infrastructure to serve them. The hotels are a bit basic and there are very few restaurants with English language menus. There isn't much in the way of shopping either, at least not for souvenirs. There is a large fresh market off Charoensuk Road near the middle of the new city, which might be interesting to look around in.