Introducing an MLB Pitch Type Prediction Tool

Which pitch type is coming next?

This question lies at the core of modern baseball and has taken on an extreme level of importance – so much so that teams have proven to be willing to cheat to obtain an answer.

The Houston Astros’ sign-stealing scandal was, of course, the most significant indicator of how uniquely important identifying the next pitch type thrown has become to MLB players. But it isn’t just the players who benefit from such knowledge; it’s also fans like me, who have enjoyed trying to predict which pitch was coming next for nearly as long as we’ve been watching baseball.

MLB Pitch Prediction Tool

With that in mind, I decided to create a simple web application that is fun for fans to use while watching MLB games. Along with a probability distribution representing the next pitch type, the application also displays the current game state (innings, outs, score, count, etc). You can view the app here: MLB Pitch Predictor

For those of you who have the same questions I would have, like how it works or how accurate it is, read on.

How Does the MLB Pitch Type Prediction Tool Work?

The application uses a k-nearest neighbors machine learning model to predict the odds of the next pitch type. This type of algorithm (often referred to as a KNN model) is a lazy learner, which means it does not pre-train a model before making predictions on the prediction features. Rather, it uses some type of distance function to determine how similar the prediction features are to features in a stored dataset.

In this case, the model identified the k most similar prior game situations to the current one. Predicting a pitch type in a 2-2 count with no outs in the third inning, for instance, would rely heavily on the pitch types thrown in similar situations in the past for said pitcher.

To determine how similar two game situations were, the model simply checks which features are shared between the two game situations and creates a similarity score for each past pitch using weights. For instance, the weight for strikes was 1.948. That means 1.948 points would be added to a past situation’s similarity score if the current situation matched said number of strikes. The feature weight for the last pitch type, for example, was 1.03.

The optimal weights were identified using Bayesian optimization and used 155 randomly selected MLB games from May 1st through August 25th to identify the feature set that resulted in the highest-calibration model. More on that later. All in all, the optimal parameters were found based on roughly 50,000 pitches.

How Accurate is It?

To measure the effectiveness of the model, we care not only about the accuracy – how often the highest-probability pitch type is thrown – but also about calibration.

Let’s say the model predicted a 55% chance of a fastball in 100 different instances. If the model is well-calibrated, roughly 55 of those pitches will be fastballs. Calibration differs from accuracy because it tells us how accurate the probabilities themselves are, rather than simply how often the highest-probability pitch type is thrown next.

To evaluate the model, a naive estimator was used to predict roughly 100,000 pitches from randomly selected games (excluding the parameter identification dataset). The baseline estimator had no knowledge of the situation and therefore used the pitcher’s overall usage rates to predict the next pitch every time. This means it predicted their most commonly thrown pitch every time.

The KNN model outperformed the baseline in both calibration and accuracy. For every pitch predicted by the KNN model, a randomly selected pitch type-probability pair was chosen (e.g., Slider, 19.6%) and placed into one of ten bins based on the predicted probability. The same process was used for the baseline estimator, and the rate at which said pitch type was thrown next, as well as the overall predicted probability for each pitch type prediction, was calculated for each bin.

The KNN model was more well-calibrated than the baseline estimator overall, as well as for pitches with an estimated probability in the 1-10%, 21-40%, and 51-70% ranges. The KNN model also achieved 44.25% accuracy, while the baseline estimator had a 41.99% accuracy. While it may not seem like a large difference, that is a statistically significant difference due to the extremely large test sample size.

Conclusion

Predicting the next pitch type is a great way to stay engaged and learn more about the pitchers we’re watching. This tool provides another useful, fun avenue for insight into how pitchers are attacking hitters.