Lots of things to do to keep you busy for a lot longer than a week:
Take a hour or more to wander through Beacon Hill park. It's just on the edge of downtown. I'm sure you'll stop by it on your HOHO tour.
Go on the little water ferry that circles the Inner Harbour. It will take you to Fisherman's wharf and residential area. Walk around at each of those stops.
See what's playing at the Belfry or McPherson playhouse while you're in town. Have an old school cocktail at Clive's Lounge in the Chateau Victoria before hand.
Visit the Empress Hotel for afternoon tea. If you can't make that, visit their gift shop to take home some fabulous tea.
Visit the Royal BC Museum. Next door you can visit Thunderbird Park for free and get a free tour of the British Columbia Legislature.
If you like history and gardens, visit Craigdorroch Castle and Hatley Castle, both built by the wealthy coal magnate, Robert Dunsmuir.
Plan sightseeing with your lunch. Sea Cider Farm and Ciderhouse in Sydney has cider tastings with gorgeous ocean views. You could bring a picnic and enjoy their scenic deck while you sample their unique ciders.
Make the longer drive to Sooke and plan a meal at Sooke Harbour House. Along the way, visit one of the spectacular wild beaches. Frech beach, China beach and Sombrio beach are about an hour from Victoria.
Further still is Botancial Beach near Port Renfrew but well worth it for the complete west coast experience.