Is there a golf package included at MFV. If so, I would actually consider an early spring or fall week there - it is close enough to home to just take a couple days off from work and actually commute the rest ot the week.
BTW, weather in SJ in mid-Nov can be very iffy. I have seen daytime 60's but also daytime 30's. It's just that in-between-seasons time that you can never predict. But casinos are inside, in-and-out of the shops at Smithville or Cape May, and I know my husband will golf every day as long as there's not snow on the ground to hide the ball!