I live about 5 minutes away from Port Jefferson. I love going down port an hour or two. There isn't a ton of things to do, but if it doesn't take too much of an effort to get there, it could make for a nice afternoon. There are plenty of restaurants. I always wish they'd get some better shops in there. There are a few nice shops, but not enough to spend hours window shopping.
If you live close to the ferry and it is not a big trip, then go for it. It is always pleasant to ride the ferry on a nice day. There is a nice little theater in town called Theater Three. Check out their website (theaterthree.com) and see if there is a show you might enjoy while you're there. They usually are very nice productions and the tickets are reasonable. That would kill some time.
Then you could spend an hour or two walking around. There are ice cream shops, and a sweet shop.
There is an excellent brick oven pizza place called The Pie. That would be a great place for lunch. For dinner, Pace's Steak House is very good, although a little pricey. Pasta Pasta is a nice neuvo italian restaurant and of course there are plenty of little sea food joints.