The place for all things wine, focused on serious wine discussions.

Points! Do amateurs do them as well as the experts?

Moderators: Jenise, Robin Garr, David M. Bueker

no avatar
User

Robin Garr

Rank

Forum Janitor

Posts

21612

Joined

Fri Feb 17, 2006 1:44 pm

Location

Louisville, KY

Points! Do amateurs do them as well as the experts?

by Robin Garr » Fri May 25, 2018 9:58 pm

A fascinating data dive by Mark Schatzker and Richard Bazinet compares Cellar Tracker scores against the experts and comes up with some fascinating results.

CORRELATION2.jpg

Vox wrote:Why amateur wine scores are every bit as good as professionals:
If amateurs lacked wine expertise, we would expect to see little or no correlation with professionals. We saw just the opposite.
https://www.vox.com/platform/amp/2016/1 ... s-amateurs
You do not have the required permissions to view the files attached to this post.
no avatar
User

Patchen Markell

Rank

Wine guru

Posts

947

Joined

Sat Jan 16, 2016 11:18 am

Location

Ithaca, New York

Re: Points! Do amateurs do them as well as the experts?

by Patchen Markell » Fri May 25, 2018 10:17 pm

Okay, granted, it’s been 26 years since I studied statistics and multivariate regression, and granted also that I’m half a bottle of Mount Eden Chardonnay to the wind, but I’m pretty sure that there’s nothing at all surprising about this: there should be a stronger correlation between CT averages and individual critics’ scores than between the scores of any two individual critics, even on the supposition that both the CT users and the pros are scoring by random draws. Doesn’t say a thing about the relative “skill” of experts and amateurs.
cheers, Patchen
no avatar
User

Tim York

Rank

Wine guru

Posts

4925

Joined

Tue May 09, 2006 2:48 pm

Location

near Lisieux, France

Re: Points! Do amateurs do them as well as the experts?

by Tim York » Sat May 26, 2018 2:12 am

This just seems to say that tastes differ. The biggest divergence is with Jancis R who has a more European palate than the weight of CT users and the other two "experts". It's my guess that if it were possible to attempt correlation with the views of RVF's critics, the divergence with the CT average would be even greater.

It is a source of constant irritation to me that web wine sellers trumpet Parker and Suckling scores on their sites. The higher the number, the less likely I am to be interested in the wine without a well written corroborative TN which allows me to calibrate their views against my own palate. I'm more likely to be impressed by a Jancis score on the rare occasions when they quote one.
Tim York
no avatar
User

David M. Bueker

Rank

Riesling Guru

Posts

34251

Joined

Thu Mar 23, 2006 11:52 am

Location

Connecticut

Re: Points! Do amateurs do them as well as the experts?

by David M. Bueker » Sat May 26, 2018 9:41 am

None of the big names do anything for me anymore.

There was a time when I still trusted Jancis’ recommendations, but she is regularly flubbing these days. I think her palate/nose is going.

Anyway, I only think of scores as rough guides regardless of who provides them, and have seen so much of the palates difference effect that 99.23975281% of the time I don’t even see numbers next to notes.
Decisions are made by those who show up
no avatar
User

Robin Garr

Rank

Forum Janitor

Posts

21612

Joined

Fri Feb 17, 2006 1:44 pm

Location

Louisville, KY

Re: Points! Do amateurs do them as well as the experts?

by Robin Garr » Sat May 26, 2018 10:33 am

David M. Bueker wrote:None of the big names do anything for me anymore.

There was a time when I still trusted Jancis’ recommendations, but she is regularly flubbing these days. I think her palate/nose is going.

Anyway, I only think of scores as rough guides regardless of who provides them, and have seen so much of the palates difference effect that 99.23975281% of the time I don’t even see numbers next to notes.

Agreed. I did think it was fascinating, though, to see the differences among the three critics. And I think Tim is on target about Jancis' Euro-palate vs the Americans.
no avatar
User

David M. Bueker

Rank

Riesling Guru

Posts

34251

Joined

Thu Mar 23, 2006 11:52 am

Location

Connecticut

Re: Points! Do amateurs do them as well as the experts?

by David M. Bueker » Sat May 26, 2018 10:35 am

Versus specific Americans. There are some (e.g. David Schildknecht and Allen Meadows) who have more European palates.
Decisions are made by those who show up
no avatar
User

Robin Garr

Rank

Forum Janitor

Posts

21612

Joined

Fri Feb 17, 2006 1:44 pm

Location

Louisville, KY

Re: Points! Do amateurs do them as well as the experts?

by Robin Garr » Sat May 26, 2018 11:05 am

David M. Bueker wrote:Versus specific Americans. There are some (e.g. David Schildknecht and Allen Meadows) who have more European palates.

Right, and you and me. And many others.
no avatar
User

Tim York

Rank

Wine guru

Posts

4925

Joined

Tue May 09, 2006 2:48 pm

Location

near Lisieux, France

Re: Points! Do amateurs do them as well as the experts?

by Tim York » Sat May 26, 2018 11:42 am

David M. Bueker wrote:Versus specific Americans. There are some (e.g. David Schildknecht and Allen Meadows) who have more European palates.


Those specific Americans' tastes seem to me in sympathy with those of the "average" American wine lover represented by the average scores of the largely American CT population. I think that there is no accident about that. Those critics' commercial success is tied to their being acceptable to that "average" taste. I sense that Jancis' recommendations have evolved New World-wards in recognition of the inclination of the UK market in that direction.

Meadows and Schildknecht address more limited connoisseur audiences.
Tim York
no avatar
User

Robin Garr

Rank

Forum Janitor

Posts

21612

Joined

Fri Feb 17, 2006 1:44 pm

Location

Louisville, KY

Re: Points! Do amateurs do them as well as the experts?

by Robin Garr » Sat May 26, 2018 12:28 pm

Tim York wrote:Those critics' commercial success is tied to their being acceptable to that "average" taste.

I have a chicken-and-egg thing going on now, though: Does RMP follow that national average taste, or did he shape it?
no avatar
User

David M. Bueker

Rank

Riesling Guru

Posts

34251

Joined

Thu Mar 23, 2006 11:52 am

Location

Connecticut

Re: Points! Do amateurs do them as well as the experts?

by David M. Bueker » Sat May 26, 2018 12:50 pm

It’s more likely he was a significant factor in shaping it.
Decisions are made by those who show up
no avatar
User

Steve Slatcher

Rank

Wine guru

Posts

1047

Joined

Sat Aug 19, 2006 11:51 am

Location

Manchester, England

Re: Points! Do amateurs do them as well as the experts?

by Steve Slatcher » Sat May 26, 2018 12:55 pm

On the subject of chickens and eggs, I wonder how much of the correlations are because CT users know how the wines were scored by the critics...? Or because they "just know" that the wine is good/bad because of the reputation for it created by the critics.
no avatar
User

Robin Garr

Rank

Forum Janitor

Posts

21612

Joined

Fri Feb 17, 2006 1:44 pm

Location

Louisville, KY

Re: Points! Do amateurs do them as well as the experts?

by Robin Garr » Sat May 26, 2018 2:05 pm

Steve Slatcher wrote:On the subject of chickens and eggs, I wonder how much of the correlations are because CT users know how the wines were scored by the critics...? Or because they "just know" that the wine is good/bad because of the reputation for it created by the critics.

Possibly so, Steve, although the article does address that at some length and, sort of, dismisses it.

It’s possible all those enthusiasts on CellarTracker already knew what Robert Parker and other experts said about each wine and just parroted their scores. That said, when you consider that a 2007 Seghesio Family Vineyards Zinfandel Sonoma County was rated 1,406 times on CellarTracker, for an average score of 90.5, does it really seem likely that 1) they had all read the expert scores, and b) they were consciously or unconsciously swayed as a result?
no avatar
User

David M. Bueker

Rank

Riesling Guru

Posts

34251

Joined

Thu Mar 23, 2006 11:52 am

Location

Connecticut

Re: Points! Do amateurs do them as well as the experts?

by David M. Bueker » Sat May 26, 2018 2:37 pm

The authors of the study do not properly address the fact that point scores do not mean anything like what they used to mean. 90 is now damning with faint praise for a significant portion of raters. 93 or 94 is pretty much the low bar for “worth drinking,” and people give 89-91 points to wines where the note claims it was undrinkable.

CT averages are essentially meaningless, as is any perceived correlation.
Decisions are made by those who show up
no avatar
User

Jenise

Rank

FLDG Dishwasher

Posts

42547

Joined

Tue Mar 21, 2006 2:45 pm

Location

The Pacific Northest Westest

Re: Points! Do amateurs do them as well as the experts?

by Jenise » Sat May 26, 2018 3:00 pm

David M. Bueker wrote: and people give 89-91 points to wines where the note claims it was undrinkable.


I've seen that, and it flummoxes me all to hell.

Have you ever looked at an average rating to see what effect a low score has in bringing down an average? I did it once, and IIRC 80 will lower the mean score but 79 won't--CT lets you enter a low score but in fact it doesn't take any score under 80 into account. Not one to give scores much (unless I see a need to contribute to make an impact on the average score shown), but I used to give 79 pts to wines that were below the minimum of average. There's just no way to numerically identify a substandard wine.
My wine shopping and I have never had a problem. Just a perpetual race between the bankruptcy court and Hell.--Rogov
no avatar
User

David M. Bueker

Rank

Riesling Guru

Posts

34251

Joined

Thu Mar 23, 2006 11:52 am

Location

Connecticut

Re: Points! Do amateurs do them as well as the experts?

by David M. Bueker » Sat May 26, 2018 4:20 pm

Anything below 80 gets lumped in with flawed.

Of course anything below 85 does not get published by the Wine Advocate anymore, so that’s also an incomplete data set.
Decisions are made by those who show up
no avatar
User

Steve Slatcher

Rank

Wine guru

Posts

1047

Joined

Sat Aug 19, 2006 11:51 am

Location

Manchester, England

Re: Points! Do amateurs do them as well as the experts?

by Steve Slatcher » Sat May 26, 2018 4:48 pm

Robin Garr wrote:Possibly so, Steve, although the article does address that at some length and, sort of, dismisses it.

Thanks for pointing that out, Robin. I admit to not having read the article when I made the comment - mainly because I didn't spot that the link at the bottom of your original post was the source of the graphic. Not a good excuse I know, but there you are!

But the proposition the article (quite reasonably) rejects is that all CT users had read the expert scores. My equally reasonable argument would be that some CT users had read the scores or knew the reputation of the wine, and that could well be the reason for some of the correlation. How much though, is of course impossible to say.

Personally I hold no strong views on the "wisdom of crowds" vs "ignorance of the masses" debate apart from to say that everyone sufficiently motivated to publish a tasting note and score is entitled to their opinion. Equally, everyone must chose who, if anyone, they pay attention to.

Who is online

Users browsing this forum: ClaudeBot, Google [Bot], TomHill and 0 guests

Powered by phpBB ® | phpBB3 Style by KomiDesign