We always go the second week of August because of school schedules and the fact that we own a fixed week. It is hot then, and if you don't like heat, you might not enjoy it as much.
Toward the end of August and beginning of September, there is more of a chance of tropical storms, but just about any other time is fine. I've been in April and it was delightful--warm in the daytime and comfortably cool in the evenings. The ocean water temp. is colder then than it is in August, though.