Your resort review rating methodology

[gravityrules]

@jp10558, bumping this thread from last year about resort ratings. I think there's some irony in you getting some pushback over a point or half point (is it 8, 8.5 or 9 ?) in rating the Welk considering the amount of thought and effort you put into this thread and your comparatively hard grading of other resorts. To the extent there was a consensus in this thread, it was that TUG numerical ratings aren't particularly reliable, at least not to 1 point of resolution. Too many examples of average or worse resorts with ratings in the 'eights'. "Read the reviews" was the mantra, and I agree with that advice. I'm still of the opinion that most of the TUG ratings are subjective satisfaction scores with a host of unstated assumptions baked into the rating (price paid being one of those). TUG reviewers for the most part seem to ignore the objectively oriented TUG ratings guidelines. jp10558, I have appreciated your reviews and the attempt to avoid falling into that pattern.

 

[jp10558]

@jp10558, bumping this thread from last year about resort ratings. I think there's some irony in you getting some pushback over a point or half point (is it 8, 8.5 or 9 ?) in rating the Welk considering the amount of thought and effort you put into this thread and your comparatively hard grading of other resorts. To the extent there was a consensus in this thread, it was that TUG numerical ratings aren't particularly reliable, at least not to 1 point of resolution. Too many examples of average or worse resorts with ratings in the 'eights'. "Read the reviews" was the mantra, and I agree with that advice. I'm still of the opinion that most of the TUG ratings are subjective satisfaction scores with a host of unstated assumptions baked into the rating (price paid being one of those). TUG reviewers for the most part seem to ignore the objectively oriented TUG ratings guidelines. jp10558, I have appreciated your reviews and the attempt to avoid falling into that pattern.

Thanks - I do think I deviated a little on the Welk, mostly based on this and other threads, to try and give an "estimated expected experience" rather than a "strict what I experienced". I kind of expect that if I went back I might well find the sauna working, the lockers working, a cleaned balcony and a fixed Internet (based on the router reset comment that just came in. TBH I never thought about complaining to maintenance about the Internet at a place.). Kind of like my 2 Massanutten Regal Vistas reviews - the second stay across 2 Units (all?) of the issues were not there at that time. However, now remembering that of course you can rate the location multiple times if you stay multiple times - I probably should have been harsher on The Welk. I did indicate just how impressed I was by the amenities I really enjoyed or would use a lot (Farmers Market, Pizza Hut, Shuttle on demand, large and comfortable unit). And as I talked about here earlier with Oak and Spruce - there probably should be something in there about whether you got value out of the resort over a basic hotel room - and in both cases I really did, in a way the otherwise "better" MarBrisa (if you count not having a messed balcony and fancier rooms) didn't - but it "won" because it just didn't have the thing at all (sauna, on demand shuttle, farmers market, pizza hut etc). And like I said, I'm not sure I should rate someplace higher because it doesn't have a sauna than one that has a sauna but it was under maintenance...

Thanks again for your compliments though, and this is why I also try and have a somewhat formulaic text review because of all the confusion assigning a number.

 

[TUGBrian]

id probably agree that there is little need for a half point difference in a 1-10 rating scale.

ill see if we can revamp it to make it a bit easier.

that said, I see plenty of resorts that get rated 10 that really shouldnt given our rating scale.... no knock on the resorts themselves as they are clearly very nice and anyone would be pleased with a stay there....but there is a clear difference between a really nice resort...and "world class, absolutely the best possible vacation someone could ask for" or similar which should be the criteria for a 10 rating etc.

especially when the review itself actually contains legitimate complaints!

no clocks or chargers on the night stand, tv remotes not working, finiky ac/heat, missing items from the kitchen.

while none of these would ruin my vacation or prevent me from staying at a resort again, it would (for me anyway) disqualify a resort from getting a 10 rating which should be a stay where you didnt have any complaints at all.

 

[jp10558]

id probably agree that there is little need for a half point difference in a 1-10 rating scale.

ill see if we can revamp it to make it a bit easier.

that said, I see plenty of resorts that get rated 10 that really shouldnt given our rating scale.... no knock on the resorts themselves as they are clearly very nice and anyone would be pleased with a stay there....but there is a clear difference between a really nice resort...and "world class, absolutely the best possible vacation someone could ask for" or similar which should be the criteria for a 10 rating etc.

especially when the review itself actually contains legitimate complaints!

no clocks or chargers on the night stand, tv remotes not working, finiky ac/heat, missing items from the kitchen.

while none of these would ruin my vacation or prevent me from staying at a resort again, it would (for me anyway) disqualify a resort from getting a 10 rating which should be a stay where you didnt have any complaints at all.

I think the half point is fine - but it would be nice if the 10 rating said "world class, absolutely the best possible vacation someone could ask for" and "a stay where you didnt have any complaints at all".

Like I said last time - something less ambiguous and more obvious would help. And a better guide - like here if you have any complaint you expect 9.5 to be a ceiling. If we have to stay with one scale here is my current suggestion:

10 - "world class, absolutely the best possible vacation someone could ask for" and "a stay where you didnt have any complaints at all". Good service from staff, housekeeping offered.
9.5 - Top Tier location. Furnishings are up to date. Has all major amenity categories on site
{water} Hot Tub, Pool, Sauna, Steam Room,
{energetic} Fitness Center, Hiking Trails, Skiing, Mountain Biking, Kayaking etc
{shopping} Marketplace, Restaurants, Farmers Market
{low impact}Ping Pong, Shuffle Board, mini golf, Cornhole etc
{educational}Golf, Tennis, Pickle-ball Training, Nature explanations, Water Yoga class etc
{events} Concerts, Movies, Bingo, Owner Get together etc
{crafting} Wine and Art, Tye Die T-Shirts, Make your own candles etc
Multiple activities scheduled and made easily accessible by notification on check-in or text or signage around resort public areas.
9.0 - Same as 9.5 but you had one or two issues that were resolved by staff OR some amenities missing or not available during your stay (i.e. under construction or maintenance).
8.5 - Good resort with a great location. Units are top quality and up to date. Has many amenity categories with good coverage.
8.0 - Same as 8.5 but is missing some amenity categories. You may have had an issue or two that were resolved by staff.
7.5 - An above average resort with either lesser units or lesser location or very few amenity categories. Like a good resort and location that only has a Hot Tub, Pool and Fitness Center and nothing else would be 7.5.
7.0 - Same as 7.5 but you had some issues, or it only has good units but a meh location and few amenity categories.
6.5 - Older resort or less luxury resort. Units still in good repair. Has many things to do and at least several amenity categories.
6.0 - Same as 6.5 but you had some issues. May only have one amenity category with limited items like just a Hot Tub. OK exchange, but could be better.
5.5 - Average resort that has little by itself to recommend coming here - maybe you like the location for some reason, or it is cost effective vs hotels in the area. Not a destination in itself at all. Units could be older but serviceable. Has at least several amenity categories. You'd be OK going back as a good price / value option based on location or amenities or events.
5.0 - Same as 5.5 except you had issues, or only has one amenity category with limited items like just a Pool.
4.5 - Below average resort. Units are older but serviceable. Only has one amenity category or location is very weak, or units don't have multiple expected things like no laundry, no kitchen, no pull out bed, missing things like cups or the like, etc. May be a bad location or "only good for sleeping in".
4.0 - Same as 4.5 but you also had issues, may not have been resolved. Furnishings are noticeably out of date. Is still OK to sleep in, but only just.
3.5 - Units are old, tired, run down, in need of refurbishment. Things in the unit are worn out / broken. Furnishings are noticeably out of date. Literally only good for sleeping in.Beds are worn out or not comfortable.
3.0 - Same as 3.5 but you had issues, may not have been resolved. You probably don't have a reason to go back.
2.5 - Units are old, tired, run down, in need of refurbishment. Many things in the unit are worn out / broken. No amenities on site. Bad location. Beds are worn out / bad. There is no reason to go back.
2.0 - Same as 2.5 but you had issues that were not resolved. Cleaning may be less than perfect. You may immediately want replacement cleaned sheets etc.There is no reason to go back.
1.5 - DO NOT GO. Cleaning was sub standard to not done. Units run down.
1.0 - DO NOT GO. So bad you left after a night or two - you found alternate lodging or went home early.

Basically - try to come up with sort of categories based on amounts of amenities or location or having or not having both with the whole numbers - good stays get the .5 as a bonus, the X.0 is if you had issues / complaints.

 

[TUGBrian]

I think the half point is fine - but it would be nice if the 10 rating said "world class, absolutely the best possible vacation someone could ask for" and "a stay where you didnt have any complaints at all".

Like I said last time - something less ambiguous and more obvious would help. And a better guide - like here if you have any complaint you expect 9.5 to be a ceiling. If we have to stay with one scale here is my current suggestion:

10 - "world class, absolutely the best possible vacation someone could ask for" and "a stay where you didnt have any complaints at all". Good service from staff, housekeeping offered.
9.5 - Top Tier location. Furnishings are up to date. Has all major amenity categories on site
{water} Hot Tub, Pool, Sauna, Steam Room,
{energetic} Fitness Center, Hiking Trails, Skiing, Mountain Biking, Kayaking etc
{shopping} Marketplace, Restaurants, Farmers Market
{low impact}Ping Pong, Shuffle Board, mini golf, Cornhole etc
{educational}Golf, Tennis, Pickle-ball Training, Nature explanations, Water Yoga class etc
{events} Concerts, Movies, Bingo, Owner Get together etc
{crafting} Wine and Art, Tye Die T-Shirts, Make your own candles etc
Multiple activities scheduled and made easily accessible by notification on check-in or text or signage around resort public areas.
9.0 - Same as 9.5 but you had one or two issues that were resolved by staff OR some amenities missing or not available during your stay (i.e. under construction or maintenance).
8.5 - Good resort with a great location. Units are top quality and up to date. Has many amenity categories with good coverage.
8.0 - Same as 8.5 but is missing some amenity categories. You may have had an issue or two that were resolved by staff.
7.5 - An above average resort with either lesser units or lesser location or very few amenity categories. Like a good resort and location that only has a Hot Tub, Pool and Fitness Center and nothing else would be 7.5.
7.0 - Same as 7.5 but you had some issues, or it only has good units but a meh location and few amenity categories.
6.5 - Older resort or less luxury resort. Units still in good repair. Has many things to do and at least several amenity categories.
6.0 - Same as 6.5 but you had some issues. May only have one amenity category with limited items like just a Hot Tub. OK exchange, but could be better.
5.5 - Average resort that has little by itself to recommend coming here - maybe you like the location for some reason, or it is cost effective vs hotels in the area. Not a destination in itself at all. Units could be older but serviceable. Has at least several amenity categories. You'd be OK going back as a good price / value option based on location or amenities or events.
5.0 - Same as 5.5 except you had issues, or only has one amenity category with limited items like just a Pool.
4.5 - Below average resort. Units are older but serviceable. Only has one amenity category or location is very weak, or units don't have multiple expected things like no laundry, no kitchen, no pull out bed, missing things like cups or the like, etc. May be a bad location or "only good for sleeping in".
4.0 - Same as 4.5 but you also had issues, may not have been resolved. Furnishings are noticeably out of date. Is still OK to sleep in, but only just.
3.5 - Units are old, tired, run down, in need of refurbishment. Things in the unit are worn out / broken. Furnishings are noticeably out of date. Literally only good for sleeping in.Beds are worn out or not comfortable.
3.0 - Same as 3.5 but you had issues, may not have been resolved. You probably don't have a reason to go back.
2.5 - Units are old, tired, run down, in need of refurbishment. Many things in the unit are worn out / broken. No amenities on site. Bad location. Beds are worn out / bad. There is no reason to go back.
2.0 - Same as 2.5 but you had issues that were not resolved. Cleaning may be less than perfect. You may immediately want replacement cleaned sheets etc.There is no reason to go back.
1.5 - DO NOT GO. Cleaning was sub standard to not done. Units run down.
1.0 - DO NOT GO. So bad you left after a night or two - you found alternate lodging or went home early.

Basically - try to come up with sort of categories based on amounts of amenities or location or having or not having both with the whole numbers - good stays get the .5 as a bonus, the X.0 is if you had issues / complaints.

it does say that for the rating of 10 =)

Your resort review rating methodology

How do you determine your 1 to 10 rating of a resort when submitting a review? To me the template implies using a more objective standard (10 is world class resort if I'm remembering correctly) but I see a lot of very numerically high reviews of resorts that are in no way world class. I think...

tugbbs.com

 

[DaveNV]

Jumping in a bit late here, but there was a time when RCI asked vacationers to review the resort they stayed in, for purposes of determining the award level of the resort within the RCI system. (I don't know if they still do that.) But there was always the thought that owners at that resort would rate it artificially higher, trying to bump their resort in the RCI rankings. Do Tuggers do that?

I think on Tug, some reviewers are either hypercritical of a place, (the old "Lumpy pillow syndrome" we've all heard of) or they gloss over things and try to give everything an inflated rating, thinking if they're critical, people might think badly of them. "It wasn't THAT bad, so I need to rank it higher."

For me, every review I've ever written was based on my real world experience, and if I was satisfied with the place. Was it clean, was it comfortable, was I inconvenienced by the staff or other guests? Noise? Slamming doors and screaming kids? Would I stay there again? If not, why not? Self check: Am I being objective, based on my actual experience? The numbers I've given are definitely based on my personal criteria, and how I'd rate a resort based against other resorts I've stayed in. I tried to make the wording of my comments reflect the accuracy of how I felt, staying there, which is why I tried to write my review as soon after check out as possible, while the feelings were still fresh. Some places were exceptional. Others, I couldn't stay far enough away.

The most shocked I've ever been is when I had an average or less experience with a resort that was rated really high - I'd wonder what I'd done differently, to have such a less than stellar experience. (My stay at The Manhattan Club was kind of awful. Who got it wrong? The reviewers, or was my stay a one-off?)

Not sure there will be a perfect point-scoring system that works for everyone, because everyone has a different criteria of what they consider to be important.

The text below is supposed to be real. (I found this on Reddit, but I've read it elsewhere, too.) This is kind of a hoot:

Actual complaints received by the travel agency "Thomas Cook Vacations"​

1. "They should not allow topless sunbathing on the beach. It was very distracting for my husband who just wanted to relax."
2. "On my holiday to Goa in India, I was disgusted to find that almost every restaurant served curry. I don't like spicy food."
3. "We went on holiday to Spain and had a problem with the taxi drivers as they were all Spanish."
4. "We booked an excursion to a water park but no-one told us we had to bring our own swimsuits and towels. We assumed it would be included in the price."
5. "The beach was too sandy. We had to clean everything when we returned to our room."
6. "We found the sand was not like the sand in the brochure. Your brochure shows the sand as white but it was more yellow."
7. "It's lazy of the local shopkeepers in Puerto Vallarta close in the afternoons. I often needed to buy things during 'siesta' time -- this should be banned."
8. "No-one told us there would be fish in the water. The children were scared."
9. "Although the brochure said that there was a fully equipped kitchen, there was no egg-slicer in the drawers."
10. "I think it should be explained in the brochure that the local convenience store does not sell proper biscuits like custard creams or ginger nuts."
11. "The roads were uneven and bumpy, so we could not read the local guide book during the bus ride to the resort. Because of this, we were unaware of many things that would have made our holiday more fun."
12. "It took us nine hours to fly home from Jamaica to England. It took the Americans only three hours to get home. This seems unfair."
13. "I compared the size of our one-bedroom suite to our friends' three-bedroom and ours was significantly smaller."
14. "The brochure stated: 'No hairdressers at the resort.' We're trainee hairdressers and we think they knew and made us wait longer for service."
15. "When we were in Spain, there were too many Spanish people there. The receptionist spoke Spanish, the food was Spanish. No one told us that there would be so many foreigners."
16. "We had to line up outside to catch the boat and there was no air-conditioning."
17. "It is your duty as a tour operator to advise us of noisy or unruly guests before we travel."
18. "I was bitten by a mosquito. The brochure did not mention mosquitoes."
19. "My fiancée and I requested twin-beds when we booked, but instead we were placed in a room with a king bed. We now hold you responsible and want to be re-reimbursed for the fact that I became pregnant. This would not have happened if you had put us in the room that we booked."

My two cents.
Dave

 

[TUGBrian]

absolutely right, itll never be perfect and examples of how its wrong will always exist. that said im always up for suggestions to improve things!

 

[easyrider]

not against changing the criteria at all.

though i dont think we could switch from 1-10 to 1-5 without some severe issues there on the back end...but who knows.

perhaps just making the existing rating system more visible on the review page vs it being something you have to click on to view would improve things.

Who ever came up with the criteria list was really thinking. The one to ten ratings work perfectly because the criteria for each number designates a lack of or higher level of amenities and experiences, imo. It's easier to classify a resort with a great location but a bit out dated with 1 to 10.

If it went 1 to 5 it wouldn't matter much.

Bill

 

[easyrider]

Jumping in a bit late here, but there was a time when RCI asked vacationers to review the resort they stayed in, for purposes of determining the award level of the resort within the RCI system. (I don't know if they still do that.) But there was always the thought that owners at that resort would rate it artificially higher, trying to bump their resort in the RCI rankings. Do Tuggers do that?

I think on Tug, some reviewers are either hypercritical of a place, (the old "Lumpy pillow syndrome" we've all heard of) or they gloss over things and try to give everything an inflated rating, thinking if they're critical, people might think badly of them. "It wasn't THAT bad, so I need to rank it higher."

For me, every review I've ever written was based on my real world experience, and if I was satisfied with the place. Was it clean, was it comfortable, was I inconvenienced by the staff or other guests? Noise? Slamming doors and screaming kids? Would I stay there again? If not, why not? Self check: Am I being objective, based on my actual experience? The numbers I've given are definitely based on my personal criteria, and how I'd rate a resort based against other resorts I've stayed in. I tried to make the wording of my comments reflect the accuracy of how I felt, staying there, which is why I tried to write my review as soon after check out as possible, while the feelings were still fresh. Some places were exceptional. Others, I couldn't stay far enough away.

The most shocked I've ever been is when I had an average or less experience with a resort that was rated really high - I'd wonder what I'd done differently, to have such a less than stellar experience. (My stay at The Manhattan Club was kind of awful. Who got it wrong? The reviewers, or was my stay a one-off?)

Not sure there will be a perfect point-scoring system that works for everyone, because everyone has a different criteria of what they consider to be important.

The text below is supposed to be real. (I found this on Reddit, but I've read it elsewhere, too.) This is kind of a hoot:

Actual complaints received by the travel agency "Thomas Cook Vacations"​

1. "They should not allow topless sunbathing on the beach. It was very distracting for my husband who just wanted to relax."
2. "On my holiday to Goa in India, I was disgusted to find that almost every restaurant served curry. I don't like spicy food."
3. "We went on holiday to Spain and had a problem with the taxi drivers as they were all Spanish."
4. "We booked an excursion to a water park but no-one told us we had to bring our own swimsuits and towels. We assumed it would be included in the price."
5. "The beach was too sandy. We had to clean everything when we returned to our room."
6. "We found the sand was not like the sand in the brochure. Your brochure shows the sand as white but it was more yellow."
7. "It's lazy of the local shopkeepers in Puerto Vallarta close in the afternoons. I often needed to buy things during 'siesta' time -- this should be banned."
8. "No-one told us there would be fish in the water. The children were scared."
9. "Although the brochure said that there was a fully equipped kitchen, there was no egg-slicer in the drawers."
10. "I think it should be explained in the brochure that the local convenience store does not sell proper biscuits like custard creams or ginger nuts."
11. "The roads were uneven and bumpy, so we could not read the local guide book during the bus ride to the resort. Because of this, we were unaware of many things that would have made our holiday more fun."
12. "It took us nine hours to fly home from Jamaica to England. It took the Americans only three hours to get home. This seems unfair."
13. "I compared the size of our one-bedroom suite to our friends' three-bedroom and ours was significantly smaller."
14. "The brochure stated: 'No hairdressers at the resort.' We're trainee hairdressers and we think they knew and made us wait longer for service."
15. "When we were in Spain, there were too many Spanish people there. The receptionist spoke Spanish, the food was Spanish. No one told us that there would be so many foreigners."
16. "We had to line up outside to catch the boat and there was no air-conditioning."
17. "It is your duty as a tour operator to advise us of noisy or unruly guests before we travel."
18. "I was bitten by a mosquito. The brochure did not mention mosquitoes."
19. "My fiancée and I requested twin-beds when we booked, but instead we were placed in a room with a king bed. We now hold you responsible and want to be re-reimbursed for the fact that I became pregnant. This would not have happened if you had put us in the room that we booked."

My two cents.
Dave

Pretty funny Dave.

Bill

 

[gravityrules]

I suspect the aggregate average of all TUG reviews for all resorts is 8 or over. There aren't many 'average' or below reviews. Is calling a resort average considered as an insult? It seems we effectively have a 5 point scale. Has it become like so many surveys where a merchant will request only the highest rating or no rating at all?

I think a separate resort category star rating (1-4 based on previous discussions) would be helpful, then the review ratings could be a satisfaction score (which I suspect that is what it is for most reviewers).

 

[TUGBrian]

id agree, the only time i see ratings below 5 is when a person had a truly AWFUL stay...bordering on having to leave the resort to go somewhere else etc.

although I also have seen it in the past where the low rating of 1 or 2 was given because of how they were treated at the sales presentation or by a salesperson, which sadly isnt fair for the rating of the resort itself!

all in all I think the words in the review matter more than the numerical score...for me anyway.

 

[gravityrules]

@TUGBrian The review rating of a resort is an average but is it over all reviews ever submitted on TUG or is it limited to a certain timeframe, such as just over the last 10 years? Or perhaps another number?

 

[TUGBrian]

its only the past 5 or so years (id have to check the code for the actual number)...nothing over 10 years for sure.

 

[gravityrules]

Reason for asking ...

There is a resort where I looked up reviews/ ratings on Tripadvisor. The rating was 4.3 (on a 1 to 5 scale). Seems fine, right? But if you look at the last 20 reviews, which cover about the last 2 years, the average is 2.35

Danger Will Robinson!

Resort in decline is the correct assessment here (confirmed by my personal experience with the resort).

The 'advertised' rating of 4.3 over 660 reviews is misleading in this case. I don't know how many years those reviews go back, perhaps all the way to 2005 when the resort was first opened.

 

[gravityrules]

Che

its only the past 5 or so years (id have to check the code for the actual number)...nothing over 10 years for sure.

Checked ratings for 2 recently reviewed resorts, the displayed rating seems to be averaged over reviews for the last 7 years.

 

[easyrider]

I think many of the reviews ratings are more of a satisfaction rating.

You have a good point here. My ratings are bad when things like cleanliness , broken items, bugs, bad attitudes and a bad vibe. The bad vibe is usually having to wonder if my catalytic converter is safe in the parking area. I've stayed at crappy to others reviewing resorts and thought it a solid 5 or 6. I've also stayed at 7-9's and thought these were really over rated. I kind of focus on the reviews wordings because often times the writer will reveal what was good, bad or in-between.

Bill