Although I wouldn't choose to do it, a day trip to Victoria is absolutely doable from Vancouver. I agree with those who've suggested the float plane idea as it's really very quick and easy. Harbour Air is what you'd be looking to book. You can hop a 35 min flight from downtown Vancouver to downtown Victoria, and Victoria is about as walkable as it gets. With a group of 4 this will be pretty expensive (maybe $1500 CDN round trip?), but it's a neat experience.
Doing a day trip driving onto the ferry has some advantages and is also doable, but it would make for a long day as others have stated. I do a similar day trip frequently as I live in the Gulf Islands. You would be driving against the flow of traffic, which sounds great in theory, except there's a counterflow lane setup in the Massey Tunnel which is an absolute disaster on the weekdays. Because of this you'd want to catch either the 5am or the 7am ferry to Victoria in order to avoid traffic. There is a great scenic drive from the Swartz Bay Victoria Ferry Terminal to downtown Victoria if you leave the highway and hug the coastline. The ferry trip would only be around $300 plus whatever your car rental expenses would be.
If you're flying in to YVR in the evening, you could consider renting a car and staying in Tsawwassen rather than downtown Vancouver for that night. There is a nice Coast Hotel that's 10 minutes away from the ferry terminal with no traffic worries.
If you want to continue on after exploring Victoria, you could drive the Malahat Highway to Nanaimo and take the ferry back to West Vancouver. On a weekday, you probably wouldn't want to be arriving back anytime from 3:30pm to 6:00pm (both the Massey Tunnel and the Lions Gate Bridge will be tough).
