Is "fixing goaltending" as easy as replacing the goalies?

Be sure to join the Discord channel to talk hockey, and everything else, with our writers and subscribers.

---

By CJ Turtoro (@CJTDevil)

The story of the year for the New Jersey Devils has been bad goaltending.

I’ve started using the hashtag #JustTheGoalie to drive home the point that it’s my opinion that we’d be exactly where we want to be, if not slightly ahead, if only the goaltender position panned out as planned.

That has been true for a long time. I wrote in December that the Devils had two problems: special teams and goaltending.

Since then, the power play has hit a stretch of puck luck, and the penalty kill has ascended towards the top of the league in terms of preventing dangerous opportunities. So now it’s really #JustTheGoalie.

Evolving-Hockey breaks team performance into goal components. That includes offensive and defensive shot quantity, offensive and defensive shot quantity, shooting, and saving.

The Devils skaters have produced a -4.5 goal differential in terms of shot quantity and quality, but their shooting adds 10.3 goals so they’re actually a +6 goal differential team in terms of their skaters. However, their goaltending has cost them 33 goals. That’s why they find ourselves in the spot they do. Again, #JustTheGoalie.

When we refer to “goaltending” in the public analytics space, we are often just referring to the difference between xGA (expected goals against) and GA (actual goals against). The assumption here is that xGs capture most of the information about when goals should be scored, and so any difference is attributable to the goalie.

There is a lot that isn’t in xGs, though. We don’t have screens, setup passes, the location of the setups, the movement of the actual shooter before the shot, the speed of the shot, the path of the shot, etc.

So if a team, or the players on a team, are uniquely poor at limiting the danger of all of those factors, the xGA count may underestimate the difficulty of those shots. If that’s the case, then “fixing the goaltending” might prove to be much harder than just replacing the goalies.

We could possibly improve our understanding of how much a goalie’s teammates are helping if we could isolate their impact on the differential between the xGA and GA stats.

Estimating Player’s Marginal xG Prevention

I decided to take a pass here at assessing the ability of players to consistently overachieve their defensive xG results. As my metric I decided to use what I’ll refer to as RAPM_dGA – the differential between a player’s GA/60 and xGA/60 impacts according to Evolving-Hockey RAPMs. RAPMs isolate for much of the context of usage, so this should help to control the impacts of any difference in usage that may impact their raw dGA numbers. For this piece I’ll be focusing ONLY on even-strength goaltending because special teams goaltending is noisy as shit.

As an initial pass, I used a Marcel projection (we’ve demonstrated that here before), which uses a simple linear regression model to predict a players RAPM_dGA from their previous three years’ RAPM_dGAs.

There was a surprising amount of signal with all three years being significant positive predictors of the 4th year. All-told the projection explained 1.5% of the variation in dGA/60 for skaters with 500+ minutes – 2.8% for defenders and 1.3% for forwards.

As is usually the case with hockey – especially so with defensive, and especially so with goaltending – there’s still a lot of variation here, and the shallowness of the line means a limited range of impact for individual players. The result of the impacts rarely exceeds three hundredths of a goal per hour (one goal every 2000 minutes).

Rosters are composed of 18 skaters, though. And many of them play 1000+ minutes a season. In a team that’s absolutely loaded with this type of player, it wouldn’t be insane to find a team whose skaters saved 5+ goals (about +0.015 per player).

Among goalies with 10 games, that’s the difference between Robin Lehner (+5.6, 17th in NHL), and Stuart Skinner (+0.9, 28th in the NHL). So, it’s not going to flip the leaderboard on its head, but it’s not nothing.

Does this ever happen? Are there coaches and GMs that select for this sort of thing or is it randomness and whatever skill players have will likely wash out on the team-level? 

Isolating Goaltender from Skater Impact

The main problem with the approach above is that the difference between a skater’s RAPM_xGA and their RAPM_GA could very well be impacted by the goaltender.

If you have a player that’s played three years in a row (largely with one goaltender), it’s certainly possible that the goalie is responsible for some of it and doesn’t deserve to have the credit taken away. 

In order to assess this, we should compare the predictivity of the skater-based projections with a similar goalie model. In order to determine how much the goalie benefits from an entire roster, we need to aggregate in some way. 

I took a TOI-weighted average of the players on a team and multiplied it by five (5-players on the ice) to get an estimate for a team’s average skater-produced per hour for a team. The graph below shows that distribution, ranging mostly from -0.03 to +0.03 marginal goals prevented per hour. 

Next comes the fun part. We’re going to try to predict dSv% (save percentage over expected) using both the team’s history and the goalies history to see who has more impact. 

We will build a similar model for the goalies as we did for the players, but with one small correction. For the players, it was reasonable to replace empty seasons with zero because this isn’t a skill likely to be too strongly selected for. For goalies, it’s all they do. I toyed around with what place-fillers to use for goalies and at dSv% of -1% turned out best.

So now we have two Marcel models for a goalie’s current-season dSv% – one using the goalies personal history, and one using that of his collective teammates.

With just the goalie’s history, we got an r-squared of about 1.5%. With just the skater’s history, we got a 1.7%. And when we combine them it looks as though the information they offered to the model didn’t overlap much at all. In this zero-intercept model both were highly (and comparably) significant independent of one another. 

So if we use just the skater-based coefficient from this model to make our prediction, that should produce an “expected dSv%” that is almost entirely separated from the goalie’s skills. When we apply it to the full database of goalies (TOI>1000), we can see that appears to be precisely what it does. 

The skater prediction does correlate with the results (left), what’s leftover no longer correlates with skaters (middle), and the adjustments make minor changes to a largely unchanged metric (right). 

It appears as though this model behaves the way in which we’d hope it would. The final step is to see what the results are. 

How Much of the Devils “Goaltending” is on the Goaltenders?

Fortunately for the Devils, it appears that their skater-induced expected save percentage is fairly mundane. Judging by the players on this roster, a goalie’s dSv% should see a dip of a tenth of a percent – the 22nd worst impact in the NHL.

So, for the most part, the Devils bad goaltending actually is mostly bad goaltending, according to this metric. In terms of the Devils’ skater breakdown, there’s likely a fair mix of expected and unexpected players on this ranking.

There is a quartet of players at the bottom of the list entered the season as some of the most likely players to on-ice Sv% below expected in the league (all were in the bottom 100 skaters). And, before you go crazy about Siegenthaler – our 2022 golden boy discovery – take a look at his RAPM impacts since entering the league.

Not only did he deserve that prediction entering the season, but he’s actually confirmed it during the season. I wanna remind everyone here that this is not a metric that means he is a bad defender – he isn’t.

What the metric means is that he may be a little more responsible for the poor on-ice save percentage that he’s observed than previously expected. It’s not about labeling players good/bad, it’s about informing our perception of an area we would’ve previously labeled as “luck” or “goaltending”.

Concluding Thoughts

First, with regards to the Devils, I think it’s perfectly reasonable to expect them to acquire a new goaltender and for that goalie to perform exceptionally well.

We’ve seen it happen for stretches in the past on this team already, we just need to find someone to do it consistently. If we get one (or two) for next season, I think we’ll be able to prove next year that this season was #JustTheGoalie

With regards to potential future research, if anyone is interested, the link to the github gist is right here. Some things to consider improving on in no particular orde:

1. Use a shot-based metric for skater rather than per60 for more direct comparison to goalies

2. Use anything more advanced than the linear (Marcel) model

3. Split up models into defenders and forwards

4. Use more than 3 seasons of history

5. Experiment with different “default” values of metrics for missing seasons

6. Remove coach and rink effects from players

7. Anything else you can think of! 

Thanks all for reading!