Every game watched results in someone inevitably saying, “the umpire screwed us.” Most fans acknowledge that calls go both ways most of the time, but do they really believe that? Does it all even out over the long run?

With Statcast tracking everything via radar, the precision with which umpires are being tracked has continuously increased, with robot umpires being tested in the low “independent” leagues. This year robot umpires will be used in the Low-A Southeast League.  One of the side effects of MLB effectively moving to own all of the minors was the ability to decide how the leagues would be run.

When Pitch f/x came on the scene, and umpires were going to be graded, people studied the results.  It was widely discussed a decade ago how umpires are only 85-90% accurate but have consistently improved after getting the feedback. These data were created from a different system than MLB uses today, which provides accuracy, they claim, to 0.1 inches.

With today’s tracking, we have been blessed by @umpscorecards.  On Twitter, after every game, these cards are posted letting you see how your game was impacted by the home plate umpire:

As you can see, the Mets won despite a small advantage to the Marlins from behind the dish.  It highlights the importance of the count on a batter. It also allows for taking a deep breath on how your team did or did not get jammed by the officiating.

For this game, you can see the Inside the Zone Accuracy was right where it has been from the previous decade. However, the accuracy average, like fastball speeds, have increased.

In the previous years, the best umpires were in the 90% range, and now we can see that is where the worst umpires are.  There is more work to do here, but it appears the longer tenured umpires may be more entrenched in “their” strike zones, rather than the correct strike zone.

It is still early in the season, so each umpire will regress to their mean, but the average is up considerably. One likely contributor is the accuracy of the assessment – the umpires were not quite as bad as previously measured.

What has this meant for the Mets?

Not much. The Mets have been fortunate enough to fall in the meaty part of the curve. What is shocking is the spread from top to bottom. 65% of the time the Red Sox are benefitting from umpiring, while the Tampa Bay Devil Rays are getting jammed.  Again, it is early in the season, and this is not accounting for park effects (yet), so we will see if these numbers regress.

What happens when you merge this data?

Most teams still “even out”. 2.5 runs over 40 games is not very much. The Mets have gotten just about the service from umpires as any team might expect.  The Red Sox may have had a big tilt in number of games with an advantage, but the advantage in those games did not add up to as much as other teams.

Then there is the advantage the Rangers have gotten. A win is about 10 runs, so the umpiring has moved the needle for the Rangers by at least that, and at the same time have taken that away from the White Sox.

Tracking exactly how umpires treat the best pitchers, Jacob deGrom for example, is still in the tiny sample size area, but definitely worth checking.

There is more to do on this area, specifically to matching pitchers and umpires, an underutilized lever, in the name of “player comfort”. At Baseball Prospectus, there was a report on umpire tendencies. Teams know who the upcoming umpires are, and the umpire rotation is well known. Teams could adjust their rotations (slightly) to take advantage of pitcher/umpire matchups and should a particularly poorly matched umpire is behind the plate, use that day for your bullpen game to resent your rotation.

With the injuries the Mets have had this season, this is not really an option, as almost every game is a AAA/bullpen game.

9 comments on “A detailed look at umpires and missed ball/strike calls

  • MikeW

    Thanks for writing this article. It is great. I dint know this data was available for public consumption.

    • Brian Joura

      I second what Mike says above.

      It’s all well and good to say that umpires have a 94% overall accuracy. But overall accuracy isn’t really the issue, is it? I mean 75% of the pitches my daughter could call right. It’s those pitches on the edges and the corners that are really the issue. By their own grading system, Timmons missed 11 pitches in the game. How many pitches were in the same distance to the corner/edges did he get right? What’s an acceptable number for an ump to miss?

      This study is a few years old but contains the data we’re looking for:

      “That figure includes the obvious calls where the pitches are right down the middle or way outside. When Moscowitz (sic) narrowed his analysis to pitches that were within two inches, either way, of the corners of the plate, the umpires got the call wrong 31.7 percent of the time – nearly one of every three pitches!”

      https://www.star-telegram.com/sports/spt-columns-blogs/gil-lebreton/article105378146.html

      To me, this is completely unacceptable. I’m sure it’s gotten better since the study. But it needs to be that 94% accuracy that they tout for all pitches and there’s no way that it is.

      Also, how are you calculating run values from these?

      Summing up the right versus wrong calls is interesting – and useful – but it operates under the assumption that all blown calls are equal and we know that’s not true. And it’s also the case of how many more pitches did a guy have to throw because of the blown call? Did May have to throw a dozen more pitches after the missed strike call that should have ended the inning? If so, that’s the difference in him being available to pitch the next day, which is something that even 100% accurate run values can’t calculate.

      • ctdial

        Glad you enjoyed it. Thanks for the observation; the older articles also use the older measurement technique, which seems to be a little lower in accuracy.

        The run value of a ball and strike are well established. I didn’t calculate these run totals; this is done by the ump scorecard guy. I haven’t checked but the run value could absolutely be considering the base-out state, and other factors you are concerned about.

        • Brian Joura

          Yeah, I did some extra digging and found out that not only is it base-out but it is also count-specific. I don’t believe they incorporate leverage but they did not say one way or the other.

    • ctdial

      Thank you. I have been following the umpire reports for 20+ years, but the Ump Scorecards are just amazing.

  • Wobbit

    I hate when they miss calls, but it is part of the game. I can’t wait to see what the machine does to the game… might make it better. Pitchers and catchers will find ways to cheat.

  • T.J.

    Good stuff Chris, thanks. It does seem inevitable that technology calls balls and strikes. That is fine by me, as I think allowing technology to make the call and having a home plate umpire make the call will seem the more or less the same…

    Obvious bad calls will be reduced, which is good, but baseball has bigger and likely more complicated with the game.

    • Brian Joura

      When we introduce the automated strike zone, it may seem the same but it won’t be the same.

      There are relatively few pitches missed but we have no way of knowing ahead of time which pitches will be missed and how crucial the circumstances will be. When watching a game with the robo umps, we’ll say – of course the ump would have called that pitch right. And these umpire cards show they do that between 89-96% of the time. But all it takes is one bad call at the wrong time.

      There were 248 pitches in last night’s game and there are somewhere around 15 that were called wrong. Maybe none of those 15 had an influence on this particular game. Or maybe they did and we just have no idea. The best we can do is say what would happen on average.

      And they do an excellent job of digging into both the base-out matrix and pitch count differences. It’s unclear to me if there is a leverage component going on here, too. It’s one thing to call a two-strike pitch a ball with two outs and the bases loaded in a game the team is losing, 15-2 versus missing that same pitch when the team is winning, 1-0.

      One other note – the consistency chart means nothing to me. It was one thing in the past – when we didn’t have the technology – to value consistency. But now that we do have the technology, the only thing that should matter is accuracy.

      If the first base umpire called everyone whose foot was in the air preparing to touch the base when the 1B caught the ball safe – he would be consistent. But he would be wrong. And if the home plate umpire is consistently calling pitches outside the strike zone in a particular area as strikes (or vice-versa) he’s just as wrong. Umpires are there to enforce the rules, not to make up their own as they see fit on a particular day.

      • T.J.

        Brian,
        Yes, all true. The impact will be huge and a positive. My comment was coming from the perspective of aesthetics. I am seeing the computer making the call, but a continued need for a home plate ump for other calls – HBP, check swings, balks, plays at plate, etc. So, the ump gets a little indicator, red light/green light, and makes the call that the computer tells, only overriding for an obvious malfunction.

        Overall, the on the field product will look the same, to the TV and in person audience. The enhanced accuracy should improve the game flow slightly since it should result in less arguments/delays, etc. Aesthetically, there are bigger issues for the game…pace of play, lack of action, shifts, strikeouts, length of games…all of these seem more challenging to “fix”.

        So, for all the points listed above, I agree…let’s do it. Someone tell the commish!

Leave a Reply

Your email address will not be published. Required fields are marked *

The maximum upload file size: 100 MB. You can upload: image, audio, video, document, spreadsheet, interactive, text, archive, code, other. Links to YouTube, Facebook, Twitter and other services inserted in the comment text will be automatically embedded. Drop file here