Mark Twain said, “There are three kinds of lies: lies, damned lies and statistics.” Statistics are certainly useful but can be manipulated, especially when taken out of context. A city Governor might promote success by quoting the number of violent crimes in the city being down 10 percent in the past year. But what if, in the first two years of the Governor’s term, violent crimes rose 30 percent?
One of the most competitive battlefields for getting to the heart of statistical analysis, is baseball. This was exemplified in the 2011 movie Moneyball. The movie is an account of the Oakland Athletics baseball team’s 2002 season and their general manager Billy Beane’s attempts to assemble a competitive team. The biggest hurdle he faced was the capital advantage bigger baseball teams had and therefore their ability to purchase the best players. The Oakland Athletic budget was around $44 million as compared to the New York Yankees who had around $125 million.
Beane met Peter Brand, a young Yale economics graduate with radical ideas about how to assess players’ value. Instead of using traditional measures such as batting average, runs batted in and stolen bases, Brand came up with different set of measures that were dismissed by the ‘experts’ (and some decision makers within the Oakland Athletics as well). Beane and Brand persisted, finding they could identify talent that had been rejected by the larger outfits and therefore at a much lower cost. Although Oakland Athletics did not win the series, they did go on to reach higher levels than ever before and Sabametrics was born.
Sabermetrics is now used by most baseball teams as the methodology to identify the talent of the future. What does this story tell us? simply that we cannot rest on our laurels when it comes to measurement analysis. The world is ever changing and therefore the way we measure things has to change. What worked last year may not work this year. There is no thing as a perfect strategy and no thing as a ultimate measurement system. Together with the review of actual data there must be a regular review of the measurement system. At least annually do a sanity check to establish you are measuring the right things and doing the right way.
