No breakthroughs in hockey analytics? Brian Burke has that wrong

Analytics approach helped build the Big Red Russian Machine and also pushed Roger Neilson to come up with best metric for fairly rating individual players

Toronto Maple Leafs general manager Brian Burke said a gloomy mouthful about the statistical analysis of hockey players at this past weekend’s MIT Sloan Sports Analytics Conference.

Essentially, Burke argued, there’s not been any real breakthrough in hockey through the use of analytics. While he didn’t say that such a breakthrough would never come about, that was clearly the tenor of his remarks.

But Burke is incorrect on both counts.

Of course, you don’t have to believe me on this. After all, I’m just a fan. I don’t go to the rink much, just watch games on TV. I was never much of a player, just a pond hockey guy mainly. I’m a hockey nobody.

And Burke? He’s the very definition of a hockey somebody. l certainly recognize and admire his expertise in negotiation, in team-building, in man management. But I still can’t agree with him in regards to hockey analytics.

In case you missed Burke’s comments, he started out by saying that NHL teams will look for any edge. If they find something, they’ll grab it and keep it secret.

“But there has not been a statistical breakthrough that I’ve seen that has been effective in hockey,” he said. “Baseball was made for this. But in hockey, statistics are more like a lamp-post to a drunk. They are useful for support, but not for illumination.”

Burke went on to say the Leafs get all kinds of research papers coming in trying to sell him on new advanced stats. “Everyone is looking for this breakthrough, for this Moneyball-type breakthrough. Grad students send me papers, and how many do we get a year? About 30. And I don’t read them but someone on my staff does. We actually keep them, we don’t throw them away, and I have yet to see anything that has value in terms of an alternative way or a more of a progressive way to evaluate players.”

Now it could be that Burke and his team have, indeed, found something useful in hockey analytics on their own and he’s simply keeping it secret, which would be a shrewd move. Certainly all the baseball and basketball pro analytics experts were extremely careful not to reveal too much about their best practices during their panels at the MIT Sloan conference.

So Burke might just be acting cagey here.

Or maybe he hasn’t fully thought through the lessons of hockey history.

Without question, there’s been a few massive breakthroughs in hockey analytics, and one such breakthrough is just waiting for an NHL team to properly exploit it for all it’s worth.

The first major analytical breakthrough: the Tarasov school

A profoundly analytical approach to hockey — including the analytical use of statistics — was at the core of one of the game’s greatest and enduring success stories. The studied, scientific approach pushed by Anatoli Tarasov, the father of Soviet hockey, took a nation of non-hockey players in the late 1940s and turned the Soviet Union into Canada’s chief rival for world hockey supremacy by the 1960s, an astonishing accomplishment.

Tarasov was fanatical about the need for puck possession and pushed his team to focus on this aspect of the game, partly by the simple act of counting up passes in a game.

In his book, Road to Olympus: Russian Hockey Secrets, Tarasov wrote of his team’s breakthrough at the 1963 World Championships. “It is noteworthy that in general … our teamwork was considerably above our main contenders. In the game against the Canadian team, the players of the USSR squad made 110 passes, while the Canadians made 60 passes; in the game against Czechoslovakia we made 106 passes, they made 70; in the game against Sweden we made 49 more passes than they did. … This is an indication of quite stable habits and a high culture of playing, a correct understanding of the game by the Soviet players.”

There’s little arguing with the success of Tarasov’s puck possession game. Even Burke would have to admit it.

I should say, though, that from a Canadian perspective, the focus from an analytical perspective has not been so much on passing and puck possession as it has been on scoring chances.

In our national school of hockey, we’re obsessed with creating scoring chances on the attack and preventing such chances on defence. Canadians will do so by whatever means necessary, employing all kinds of strategies:chip-and-chase,intimidation, the trap when we have a weaker team, firewagon hockey when we’re strong.

With our intense focus on scoring chances, it’s no surprise that almost all NHL teams, including the Edmonton Oilers, now track scoring chances, an advanced hockey stat if there ever was one.

The scoring chance stat gives a team valuable context.

A team can get good bounces and win a few games even if it is consistently getting out-chanced, but only a team that consistently out-chances its opponents will win consistently.

For example, this year when the Oilers were a decent team at the start of the year, they were about even in scoring chances for and against. From mid-November to mid-February, when the team was awful, they were getting badly out-chanced. Now that the Oilers are again playing a bit better, and winning a bit more, they’re again out-chancing their opponents.

If an NHL team doesn’t track its scoring chances, it will not have a solid indicator about the real level of its play. If it’s winning but getting out-chanced, it might not fully realize that things are actually a bit awry. If it’s losing but staying even with scoring chances, it might not realize things aren’t so bad as they seem.

If Burke’s staff isn’t tracking scoring chances with the Leafs, he’s missing out on some basic information, the kind of detail that might help him and his staff make the right decisions based on the best information.

The real breakthrough: Neilson individual scoring chances

Roger Neilson

The real breakthrough in pro hockey came in the 1970s, when a group of Canadian university and college coaches, partly spurred by Tarasov’s analytical approach, developed all kinds of new tactics and approaches to hockey. Among these new methods was a new stat, Roger Neilson’s advanced scoring chance metric to more fairly and accurately rate the play of individual players.

Neilson counted up the scoring chances for and against his team, but the crucial piece was that he also counted up which players contributed to each scoring chance and which players made mistakes on scoring chances against.

If you were on the ice but made no contribution to the chance, you didn’t get a plus mark. And if you were on the ice for a chance against but made no mistake, you didn’t get a minus.

Neilson’s revolutionary concept solved a number of issues. First, it was a plus-minus that focused on an utterly critical aspect of winning hockey games, the scoring chance.

Second, scoring chances are more numerous than goals, so it gave Neilson a better sense of who had actually played well in a game and who had not. If a player just made one good play and it resulted in a goal scored, but made five mistakes that could easily have resulted in goals against, his Neilson scoring chance plus-minus of -4 would show that.

At the same time, if a player was always out on the ice with really good teammates, he wouldn’t necessarily get numerous plus marks for all their good work, not unless he actually contributed to it. On the defensive end, a good defensive player might be out constantly with weak players, but unless he also made mistake on scoring chances against, he wasn’t penalized in his plus/minus column for their mistakes.

A revolutionary, much fairer and far more accurate plus/minus system was thus created. Since then, all kinds of hockey coaches, including former university-turned-pro coaches such as Dave King and Billy Moores of the Oilers, have rated their teams with this Neilson individual scoring chance plus/minus system.

An opportunity for an NHL team prepared to work hard

One downside is that it takes a lot of work to come up with Neilson plus/minus numbers. In today’s game, the NHL doesn’t do the work. You have to do it yourself.

Every single scoring chance in every single game must be recorded and dissected. It’s also evident that the better the hockey expertise of the person doing the recording and dissecting of each scoring chance, the more fair and accurate the plus/minus number will be for each player.

I don’t know if any NHL team yet breaks down Neilson individual scoring chance numbers for the entire league. It would take a massive amount of effort to get these numbers on all scoring chances in all 1,230 NHL regular season games.

You would need dozens of raters, all of them with a large amount of hockey knowledge. You would have to create sound, sensible standards for what is and isn’t a chance, and for what is and isn’t a contribution or a mistake, then you’d have to train your raters to judge scoring chance plays in a consistent manner.

You would have to set up a video centre to watch the games and a computer tracking program. You would have to introduce stringent methods of quality control, to ensure as much as possible that the raters were following the same definitions and standards.

In a way, though, that’s just the details. The real work has already been done. The breakthrough, the sound theoretical framework for this kind of operation, is already in place, set forth by Neilson long ago.

The $300,000 or $400,000 that it would cost a team like the Leafs to do this work would arm a GM like Burke with extremely valuable knowledge every time he went to make a trade or a signing.

He would have a great indicator of how many scoring chances his target had created and what he’d done to create those chances.

The GM would also have a great indicator of how many scoring chances the target player had made mistakes on and what those mistakes were.

In fact, through a proper video linking and archive system, it should be possible for the GM to rapidly review each and every scoring chance his target player had been involved in that season.

A GM armed with such insight, such knowledge, would have a tremendous competitive advantage over colleagues who were relying on less rigorous approaches and less comprehensive, fair and accurate stats.

Through his scoring chance database, the GM would be able to better identify those players who do the most to help their teams win, then try to pick up a few of those players in trades and signings.

The problem is too much analysis of existing data, but not enough good data

In closing, I want to make it clear that I’m not suggesting that Burke is missing something in those 30 research papers he gets every year. I suspect there are things of value in many of them, but that there probably isn’t some great method for ranking players in them.

Most of this research work mines the existing data, then calculates it in a new way in order to rank players. But the existing data doesn’t record scoring chances and only mechanically and crudely assigns plus and minus marks to players on goals and shots at net, whether they deserve the plus or minus marks or not.

The present data collected by the NHL is woefully inadequate. It doesn’t even track if a shot is screened, and who that screener was. It doesn’t track if a shot is a one-timer, and where that pass came from. It most certainly doesn’t track scoring chances.

Essentially, it fails to track the most important things in the right way, so no matter how you slice and dice the existing numbers, it’s not going to rank NHL players fairly and accurately enough for a GM to trust it, even if he’s able to understand all the impressive mathematical slicing and dicing that went into the creation of these ranking systems.

The real issue with hockey analytics is the scarcity of good data. It’s going to take a ton of hard work, knowledge and resources to start recording the truly significant events in a hockey game. Some teams are already hard at it. More are getting involved.

Me, I’d like to see fans do it, so we can all have access to the data, not just NHL teams. I’ve already started doing it myself on the Edmonton Oilers here at The Cult of Hockey.

But until this hard and extensive work happens, I don’t see a great deal of progress being made.

P.S. For a more optimistic take on the breakthroughs that the existing stats can provide, here’s a post written by the excellent hockey analyst Gabriel Desjardins of Behind the Net.

I should say that I like many of the new stats, such as QualComp, QualTeam, ZoneStarts and PDO. Again, my main concern is with all the new plus/minus systems, which fail to match Neilson’s system.