This is a review of the home plate umpire research conducted by David J. Hunter, applying convex hull metrics.
Boos have been ringing out across baseball diamonds, criticizing umpire’s call judgments since the beginning of the game. Pitch-tracking data provides a method to evaluate and train Major League Baseball umpires and evidence suggests that since the installation of the equipment umpire accuracy has improved. Today, players and managers accept variations in calls across umpires as long as each umpire is consistent in their strike zone.
The rule book cites a rectangular strike zone is to be used by umpires. However, in practice, the strike zone has more rounded corners with pitches on the corners of the rectangle being called balls. In addition, pitches off the plate opposite from the batter are more likely to be strikes than pitches off the inside of the plate. This suggests that strike zones differ between left and right-handed batters.
New metrics are being created to evaluate the consistency and accuracy of an umpire’s calls over the course of a game. First, the requirement for a rectangular strike zone is dismissed, and variations based on the handedness of the batter are permitted. As factors such as the starting pitcher can influence an umpire’s strike zone, consistency is measured within a game and averaged over all games in a season.
Ideally, each umpire should establish his strike zone and follow it consistently throughout the game, no matter who is up to bat. Four different metrics are proposed for evaluating the consistency of calls relative to the established strike zone of the umpire.
The first two metrics are rectangular metrics. The first one looks at the smallest rectangular region that contains all of the called strikes. Any pitches called balls within this zone are said to be inconsistent. The one-rectangle inconsistency index is defined as the number of inconsistent balls divided by the total number of called balls. This index is easy to determine but is very sensitive to a single outlier. It also does not account for multiple bad calls in the same location. These issues are addressed by using more rectangles. In this case, inconsistent balls are weighted according to how many rectangles they are contained within. The more rectangles they are contained within, the greater their weighting.
The last two metrics are convex hull metrics. As strike zones established by umpires are not rectangular in nature, these metrics relax that assumption. Instead, a consistent zone is assumed to be based on the idea that any pitch landing between two called strikes will also be called a strike and therefore the established strike zone is convex. Similar to the one-rectangle index, the convex hull index can fail to account for multiple bad calls in the same location. To combat this, the location of called balls is used to define a called ball region. Rather than counting called balls within the established strike zone, the area of overlap between the called-ball region and the convex hull of strikes is measured.
All four metrics are sensitive to a single outlying called strike and to a bad call of a ball in the middle of the strike zone. This is done so that umpires who make slightly inconsistent calls are not penalized to the same degree as those who make clearly bad calls. However, as a result these metrics are also sensitive to the number of pitches called. As the number of called pitches increases, the chances of making a poor call also increases.
A kernel density estimation is a more accurate method for determining the borders of a consensus strike zone. It can also be used to assess conformity and zone size of individual umpires. Kernel density also works well with small sample sizes.
Using one rating systems allows umpires to be ranked. Such a ranking could provide a basis for salaries, rehiring, and termination. While the evidence suggests that MLB umpires are typically quite accurate and consistent, mistakes can happen, as umpires, after all, are only human. Strike zone inconsistency could be due to game circumstances such as the strike count. The age and experience of the pitcher may also have an impact. These possible factors could be analyzed to see if any are true for an umpire, giving the umpire an opportunity to improve his call making, and avoiding changing his strike zone in any of these circumstances.
Umpires will never be 100% accurate and that is just part of baseball. However, ensuring as much accuracy as possible results in a game being won or lost based on the player performance and not the umpire’s calls.
Analytics Used: Rectangular Metrics, Convex Hull Metrics, Kernel Density Estimation
Find out how Sports Analytics Expert Victor Holman can give your team the competitive advantage.
How mature is your team’s analytics program? Take the Sports Analytics Maturity Assessment.
Discover the Groundbreaking Sports Analytics Application and Framework coaches and sports analysts are talking about!
Learn all about sports analytics in Victor Holman’s Sports Analytics Blog.