I stayed at the Empress three years ago. Flying in by seaplane from Seattle drops you off within walking distance of some nice hotels, and the coach from the ferry does as well (I came in by ferry and left by seaplane). The city is so lovely and walkable, the museum is pretty good. I just wandered around town and found excellent food on outdoor patios. Definately going back some day.
Eta: I had business in Vancouver and then in Seattle, so I shoehorned this vacation in between. I only spent one night... which is how I could afford the Empress! I was very happy with my one night stay, but I did get most of both days on either side.