We have a family place there and a lot of family history.
It's not for everybody, but it has this:
1. Beautiful architecture. Unlike a lot of towns that have high tourism, which sport store front veneers to make the place look old and charming, Key West is actually fucking old and historical, and there is a lot of that to see. Like the Casa Marina, built by Flagler of Standard Oil fame, or the Southernmost House or the Little White House. For such a tiny place, a lot of things have happened there.
2. A great place for foodies, and always has been. There are great places to eat, from the very high end to amazing comfort food places and everything in between. If you're not good at the restaurant biz, you don't make it in Key West. There aren't a lot of 4x2 mile islands that pack more high-end food experiences, and this is one of them. There are even a few Michelin 5 stars, which is astounding for the size of the place.
3. The water experience is superior to California. Sorry guys: the Pacific ocean sucks ass this far up north other than to look at. Give me the Gulf side of Mejico and the Caribbean any day over Cali coast water. This one's not even fucking close. There aren't a lot of great beaches on Key West proper (though there are some), but the good stuff is all over the place and short boat rides away. A trip out to Fort Jefferson would be unforgettable.
4. Fantasy Fest is a mini mardi gras. You either like that or not. If not, don't go in October.
5. Key West, and Florida, are at its best weather-wise when it's completely miserable everyplace else. While I'd pick western US weather over southeastern US weather most of the year, from about late November to about March, it's hard to beat KW and south Florida. Come April, you want to get the hell outta there.
KW is worth a visit, especially in the winter. The town has gone through changes ... the rough and tumble fishing culture that Hemingway loved so much to a renaissance in the early 80s driven by an influx of a large gay community that really made KW a destination for European tourists to what has become a place at which kids from SEC schools like to converge for spring break. KW has always had a party culture, and if seeing a girl getting a tattoo on her ass on Duval Street isn't your cup of tea, it's easy enough to avoid.
It's not Hawaii and shouldn't be compared to it. Though I'd take the actual town itself over any town I've been to in Hawaii, and I'd rather be in a boat in the Keys over any beach I've been to on Maui. That said, I've been to some beaches on Oahu that I'd rate higher.