The best way to travel around the city is by car, bicycle or motorcycle. There are taxis and buses (公車 gōngchē), but they are not so convenient for non-Chinese speakers.
There is a scooter rental shop next to the Tainan City TRA (slow train) station. Rentals cost around NT$600 per 24 hours. Whether a rental shop will check for a license varies from shop to shop.
All inner city bus routes pass through Tainan Station (train). There is a tourist information booth at the Station with friendly staff (English speaking) who can show you how to use the bus system. On Sundays there are two free sightseeing bus routes (88 and 99) which can take you to and back from all the major historical sites.
If you do take a taxi just make sure you have a map you can point at or the business card of the location you're headed. The taxi drivers are very helpful, but be aware that sometimes even Chinese speakers take roundabout ways.
One should take note that there are thousands of scooters and motorbikes packing the streets and if you injure someone while you are driving in Taiwan, the local laws require you to pay for whatever the person you injured cannot. Try getting your insurance company to write a waiver for you to be insured before driving in Taiwan.