I would say 1 week in Williamsburg with a day trip to Virginia Beach. If you have come that far you should dip your feet in the Atlantic.
I also think that a trip to DC would be in order. Especially with children. I personally could spend 2 weeks there between the Smithsonian and all the wonderful history and tours there. If you feel a week is too long you can do a metro ride to Baltimore. Also a lot of history. Take in a baseball game (Camden Yards is a fun time), see the aquarium. If you plan to do DC contact your local Congressmans office as they can get you tours that need to be pre-arranged.
And as long as you are out and about, dont miss Annapolis..Some history there too (George Washington used to visit to participate in the horse races there)