Key West Ok for families
Let me say that The first time we went to Key West we were hesitant, but were pleasently surprised. The Galleon is at the end of Front St and is pretty quiet. They have a great pool and small private beach with float. There are also jet ski and boat rentals at their marina on site as well car and moped rentals also on site. There are plenty of families there and both our kids have been able to make friends. They are walking distance to most everything, I highly recommend to.
Don't let others scare you away. Honestly there are so many tourists on Duval that you have to look hard to find any kind of wild behavior. That's not to say you can't find it if your looking for it, just that if you have kids it's easy to avoid. There are lot's of family friendly activities, between the Mel Fisher's museum, the Hemminway house, the Butterfly house, the pirate museum, snorkeling trips, fishing trips and others you find lots to keep the family busy.
Go and have a great time. We have been several times during the summer and the kids love it.