Comparing impact estimates for New Jersey Devils skaters

CJ used multiple models to interpret how various Devils fared in 2021-22.

Follow @InfernalAccess on Twitter

Be sure to join the Discord channel to talk hockey, and everything else, with our writers and subscribers.

---

By CJ Turtoro (@CJTDevil)

Something I like to do every season is look at how different models interpret New Jersey Devils players’ seasons.

My two favorite examples are from Evolving-Hockey and Hockeyviz; I like juxtaposing them because the latter implements more deliberate use of something called “priors” whereas the former does not. 

A prior is an informed assumption about what a given metric is likely to be. We then update that estimate as new evidence is collected.

In the case of Hockeyviz, the impact on on-ice expected goals for a rookie is assumed to be that of a typical rookie (slightly below average) and then every subsequent year it is assumed to be wherever they left off the previous season.

In the Evolving-Hockey model, everyone starts at zero at the beginning of a season and works to distinguish themselves as the season goes on.

There are benefits and detriments to both of these approaches, but I always think it is informative to look at the players who have the most disagreement in their metrics to learn about what the models might be seeing under the hood.

In this piece, I’m going to give the top-5 Hockeyviz-prefered guys and top-5 Evolving-preferred guys and then link to the whole list so you can browse it for yourself!

Top-5 Hockeyviz-Prefered Players

When you’re using a prior, you have to pay attention not only to what a player’s previous history is, but also the history of those with whom they played.

So, while you may be tempted to loop Zetterlund and Studenic together in terms of why the models disagree (due to their limited sample), Zetterlund’s divergence is actually more akin to that of Jesper Bratt.

Both of those players had Pavel Zacha and Nico Hischier as their most common linemates. And both of those players came into the season with fairly negative priors – Zacha because of a career being misused at center, and Nico because of two seasons of injuries. Due to this, the model likely undervalues their linemates, and therefore, potentially, overvalues them.

In terms of P.K. Subban and Dougie Hamilton, what’s interesting is that they came into the season with opposite expectations. You wouldn’t assume both would be preferred by the same model. But when we unpack it we can see it actually likes them for different reasons.

In Dougie’s case, he came in with such a strong positive prior, that this one down season wasn’t enough to drag him into the negatives. The model with a prior preferred him.

With P.K., however, it had more to do with who he played with. His primary defensive partner was Ty Smith, who had terrible 5v5 impacts last season. Thus, the Hockeyviz model decided that Subban deserved more credit for carrying that pairing because it remembered Smith’s 2021 season and gave that even more weight than Subban’s own personal priors.

Top 5 Evolving-Prefered Players

The Evolving-Hockey strategy isolates only this season’s performance, which should benefit people aiming to rebound from rough previous seasons and those whose linemates had good previous seasons. 

Cases of the “putting the past behind them” type players are McLeod and Gauthier, both of whom have established over their 100+ game careers that they are a drain on their team’s offense. Hockeyviz assumed that they were mostly to blame for the continued offensive struggles during their shifts (giving more positive ratings to Bastian and Vesey as well), whereas EH spread the love around more between the whole 4th line.

In the case of the other players, though, there are two things at work. First of all, there’s an additional quirk of comparison that EH seems slower to update thanHockeyviz – it takes longer for a player to separate too far from 0.

So when players are very bad – as Geertsen, Holtz, and Okhotiuk were – the slower update means they don’t have as much time to accrue negative value. 

The other quirk is the fact that the NHL is what’s called an “apex” league, meaning only the best players in the world play in it. That makes the talent distribution skewed left. As such, most players are “below average” – just as most people have below average wealth because of the Bezoses and Musks of the world.

When there’s a new player, it’s assumed they’ll be slightly below 0 as opposed to exactly 0. For instance, Dmitri Samorukov is the least-played player in the database having played just 2 minutes for the Oilers in this, his first NHL year. Yet, he already has a -0.04 offensive impact and a +0.08 defensive one for a -0.12 net xG impact.

In this sense, in addition to moving from their prior more quickly than in Evolving-Hockey, the Hockeyviz estimates also start lower. Small-sample guys with bad starts to their career will, therefore, look much worse in these estimates – hence Holtz, Geertsen, and Okhotiuk’s inclusion.

Big Picture Differences

The specific players above do offer us a window into some of the distinctions between how different models treat different players. But sometimes we can miss the forest through the trees by looking at individual examples. This turns out to be one of those cases a bit because, when looking at the full list of Devils Players and their total impact, you notice that all the players at the bottom of the list are way lower in the Hockeyviz model. That “apex” penalty drags the worst players down substantially when you look at the whole list. I made a tableau for y’all to play around with here.

Acknowledging distinctions like these are essential to understanding analytics. It’s important to remember that the more sophisticated a model gets, the further it is from an “objective fact” and the more room for subjective prioritization is allowed.

Not in the “I like this guy more than that guy” sense, but in the sense that different people have different intuitions on how to handle data.

Where will you set your prior? How much data will you use? Will you weight it all equally? What regression technique will you use to isolate impacts? What variables will you consider? 

These impacts are from two different sites with two different xG calculations, and two totally different approaches to isolating skater impact.

Sure, they both agree Jack Hughes is good. But which young players have a future and which veterans should they replace next season? Who has turned a new leaf and who was an aberration. 

These modelers offer different legitimate approaches to answering those questions. Time will tell which proves more prescient. My experience is that, in these situations, the answer is almost always "a little of both".