Jon Shepherd: When stats fail

One of the early thoughts that emerged through the primordial sabermetric movement was that the previous year's numbers racked up at Triple-A was as meaningful as those earned at the major league level when trying to determine what would happen in the upcoming season. That matured into more refined ways to project performance, such as Clay Davenport's Minor League Translations. That approach looks at park factors, the performance of players crossing over from one minor league level to the one above, and weighs specific kinds of baseball events. It works amazingly well for the total population of players moving up to the next level. However, when these translations are used, we tend to use them for specific players. In other words, we are using a population effect and directly applying it to an individual. There can be major issues with applying that large-scale trait to a higher-resolution level. Perhaps you have experience with Body Mass Index (BMI). The BMI was developed to better describe the health of a population by using only height and weight, two common descriptors found in medical records. For a health industry that loves simplicity, it began being used for individuals and applied to health incentives by some companies for insurance purposes. Personally, I am listed as overweight coming in at 6-foot-3 and 210 lbs. I run about 15 miles a week and compete in several half-marathons a year. When I go to the department stores to buy pants, I have trouble finding a pair long enough to go with my waist. You may have experienced something similar, but the take-home here is that great population metrics will fail under some circumstances for individuals. A baseball example would be PECOTA's projection of Matt Wieters' rookie campaign in 2009. This was probably the most publicized failure for this system when it suggested historical stardom using the Davenport Translations mentioned above with Wieters' 2008 minor league performance. The model projected Wieters line as .311/.395/.544 and being worth nine wins above replacement. Instead, he hit a respectable .288/.340/.412 for the Orioles and being worth about a half-win above replacement. The problem was apparent that even sophisticated performance scouting will have issues. Now, let us look at a more common way of performance scouting. It is called going to Bowie's or Norfolk's statistics page for who is hitting well and then claiming that those players should replace individuals at the major league level. Over the years, statistical Icarus has been played by Lou Montanez, Oscar Salazar, Josh Bell and Joe Mahoney. Caleb Joseph has enjoyed being at the tip of the fan's tongue for the past two seasons with some irate that he was available last fall in the Rule 5 draft. All of these guys have hit well, earned time in the majors and have shown to be not well-equipped to start at that level as well as struggling to remain on the bench. So, why are these statistical lines confusing? For one, there is always the ever-looming specter of small sample sizes. With respect to hitting, you need about 400 plate appearances to have anything potentially useful to say statistically about performance. When you do have that body of work, other factors may sneak in. First, always give due diligence to reviewing what scouts are saying about the player. It is possible for statistical lines to be built upon a hitter feasting on the majority of the league while severely struggling against the top class of pitching that is destined to move forward. Second, older and more polished players are more likely to succeed at any given level, but are less likely to have room for improvement. This means an individual who excels at Double-A one year may have their talent level stagnate the following season while their competition is still maturing. Third, the possibility of a fluke season is not zero after accumulating 400 or so plate appearances, so weird things can certainly happen. Try to remember this if you find yourself pining for Brian Ward as a replacement for Nick Hundley or Caleb Joseph. Remember the pifalls of performance scouting when your fingers rush to call for sending down Jonathan Schoop to be replaced by Jemile Weeks. Question what it means when Dariel Alvarez or Mike Yastrzemski dominate their respective leagues. Recognize that for all the hitting David Lough did at Triple-A Iowa, that the scouting report on him during all of those years looks a lot like the player who is currently struggling at the plate. And certainly just because a strict statistical focus fails from time to time, do not ignore performance. It is a tool to go in your tool box along with traditional scouting. They key is trying to figure out when is the right time to use one tool over another. Jon Shepherd blogs about the Orioles at Camden Depot. Follow him on Twitter: @CamdenDepot. His thoughts on the O's appear here as part of MASNsports.com's continuing commitment to welcome guest bloggers to our little corner of cyberspace. All opinions expressed are those of the guest bloggers, who are not employed by MASNsports.com but are just as passionate about their baseball as our roster of writers.