Jon Shepherd: On horses, zebras and why projections matter (unless they don't)

When I was earning my doctorate, a professor wished to explain the difference between us research-based toxicologists and the resident physicians who were sitting in on our lectures. She told us the difference between our groups is that physicians look for horses while researchers go on safari for zebras. What she meant was that for physicians to do the job they do, they need to diagnose based on what is most probable. For researchers, our own nature is to press the boundary and to find the scenarios that do not fit the norm. For those reasons, these two groups can be potentially quite dismissive toward the other.

Last week, there was a bit of a brouhaha over comments made by FanGraphs' Dave Cameron about this recent incarnation of the Orioles. Briefly put, he noted that although this club has on whole defied expectations of projections that it was highly unlikely that this performance was truly based on measurable talent. From an analyst's point of view, there is nothing controversial about this statement. This sort of overperformance has been seen only once in the past 10 years and that was a half-decade run by the Angels. The two clusters are interesting because it indicates continued performance, which is how skill shows up in the numbers. However, clustering does not ensure the presence of a skill. With a sample size as large as Cameron's, these clusters are about what you would expect if the data is normally distributed. To think anything else would be wishing for zebras, which is not what Cameron does.

Response from many Orioles fans was one that is difficult to articulate in a logical way. Much of the discussion revolves around: (1) projection models are worthless, (2) Dave Cameron is trying to save his bread and butter and (3) how dare they call it luck.

None of these contentions are based on evidence. The models Cameron describes are based on evidence and are challenged with it each year. Camden Depot actually showed that these models have usefulness this past spring, and studies have shown that projections are more reliable than partial season performances, and through our own interactions, we do not know of a single major league organization that does not use modeling to help inform decision-making. This also pokes holes in the second complaint because any model with glaring flaws would be relegated out of use. Simply put, the data science field will not put up with someone juggling coefficients in order to make a pretty model.

Regarding luck, try not to get upset about the term. Many fields, many industries have peculiar terms for things. Luck means that there is deviation from the expected that currently cannot be explained. The etymology dates to the early 2000s when data science was trying to more forcefully make its way into front offices. Language use was often somewhat flamboyant in order for it to be noticed. Over time, the basic work done then has become more nuanced and uncertainty is now more universally recognized. However, the term "luck" is still used because it is familiar and a network of research is connected through this term. Yes, it is a poor term because it can connote a cheapness to success, but to complain about it borders the conversation toward English class as opposed to baseball.

In the end, the distance between analysts and fans is a bit like the horses and zebras idea. For an analyst developing and enhancing a projection model, the goal is for it to fit the data to describe the population. To do that, you focus on what is common, the horses. They are not overly concerned with performance outliers, the zebras. Yes, the models need to test themselves when unexpected clusters appear, but it is understandable that two blips might not constitute a true need for intellectual investment when there are other research needs. This is not disrespect or discounting. It is simply how research is done.

It is also important to remember that Dave Cameron's role is to speak intelligently about the use of these tools. His role is not to make Orioles fans feel better about their team or to suggest that there is skill behind their unexpected performance marks. His role is not to tell you that you should sit back and enjoy the wins. His role is simply to describe the herd.

Jon Shepherd blogs about the Orioles at Camden Depot. Follow him on Twitter: @CamdenDepot. His thoughts on the O's appear here as part of MASNsports.com's continuing commitment to welcome guest bloggers to our little corner of cyberspace. All opinions expressed are those of the guest bloggers, who are not employed by MASNsports.com but are just as passionate about their baseball as our roster of writers.