The Analytics Trap, by Tom Jelinek

The modern mind is as averse to precise ideas as it is enamored of precise numbers.   Hilaire Belloc, ~1938

​A common management philosophy says you shouldn't rely exclusively on having smart people.  They might leave, leaving you with nothing.  Instead, you need to turn their wisdom into a formula anyone can replicate, with sufficient training.  Of course, wisdom is a way of weighing conflicting information, and understanding which details matter more, case by case.  Formulas only work if the same details always matter equally.  Two people can be armed with the same numbers, and the same historical context, but one will make the sage decision, the other the foolish decision.  The formula-driven computer program will make the predictable decision.

If Belloc were alive today and followed hockey, he might have said that the modern mind is enamored of numbers, and averse to words.  When asked which team is better, the analytics specialist will cite their team Corsi, or XGF (Fenwick normalized for shot location).  While it acknowledges that Auston Matthews will always have a better shooting percentage than Freddie Guathier, it is fundamentally uncomfortable building individual players' numbers into its models.  Too good a shooting percentage, or too good a goalie's save percentage, and in particular the combination of both, is cited as proof of luck.  That tactics can influence both is ignored entirely.

Hockey has long tracked goals and assists, which provide useful info about a player, but everyone seemed to understand that there was more to the picture.  On a poor team, a player will score less.  Put Bernie Nichols on Wayne Gretzky's wing, and he'll score seventy goals; far out of proportion to the rest of his career.  And while scoring goals is wonderful, winning also requires suppression of goals against.  And not every good goal scorer is very good at preventing goals against.  Some are considered hot potatoes in that regard: You'd want them on the ice when behind by a goal late in the game, but never with a lead.  But since ideas are radioactive, someone had to put a number to it, and thus was born the plus/minus statistic.  Simple in principle: When that player is on the ice, how much do we end up scoring, versus how much do we get scored on?  And as soon as that stat was born, questions came up.  How about power plays and the penalty kill?  Those are treated as special circumstances, so we limit the tally to five on five situations, but we lose information on shorthanded goals, which I argue count for two in determining the momentum of a game.

Plus minus produced baffling results with some regularity, and players would sometimes go from being deep in the minus one year, to very positive the next.  As the number of head-scratching data points accumulated, the statistic slowly fell out of favor.  And I don't mean gently falling into disuse.  Analytics nerds today despise plus minus worse than Herpes, which is strange, since the underlying concept remains important, despite the stat's limitations: With that player on the ice, how likely are we to be scored on, versus score on the other team?  Other strikes against plus minus include that Mario Lemieux, in one of his most productive offensive years, was a net minus for the season.  So does this mean the stat is useless, or are people letting exceptions (for which there are often good explanations) tarnish the whole concept? 

Lemieux's Penguins at the time were a terrible defensive team, and he was uninterested in backchecking.  With him on the ice as much as thirty minutes per game (want to guess why he had no energy for backchecking?), they gave up significantly more goals than they scored, and did not win the Stanley Cup until they rectified their defensive problems.  Later, I'll discuss why it's possible for a player to have both a highly positive Corsi number and a strong negative number on plus/minus, and for both to be relevant.

The case against plus/minus was further bolstered by the realization that the number was often more relevant to the team than to an individual player, as one player can't change the for/against balance on his own.  As time went on, the analytics nerds began to react to plus/minus in the same way as a teenage boy who first discovers his father is not a world-class genius.  Well then, he must be a complete idiot, he decides, only realizing years later that there are degrees of being right or wrong.

The teenage mindset of hockey pundits was clearly hungry for a new father figure, to replace the idiot they rebelled against.  And their search first led them to baseball, which had been transformed by analytics as chronicled in the movie, Moneyball.  That's right, it was a movie that changed the thinking.  Ask any analytics nerd about the roots of the movement, and it's guaranteed the movie will come up.  And for those who still believe in the Easter Bunny, here is a second disappointment: Movies often embellish facts, to make for a better story.  The story, for what it's worth: A team with a small budget is trying to compete against the big-budget teams, and asks what is the best way  to value a player.  It eventually comes to reject the importance of things like runs batted in, since those can only come in bunches when teammates first get on base.  Eventually, they come up with on-base-percentage as the key statistic.  Its strength is that it turns the outcome of every at-bat into a binary outcome: Either he makes an out, or anything else (which puts him on base - that includes home runs).  Using the formula, the team in the movie enjoys great success, and the legend is born.  The movie was loosely based on the Oakland Athletics, who relied on that statistic and achieved modest success.

Hockey then seized on a development by former goalie Jim Corsi, who like all goalies, suffered from the tension of closing out games with his team pinned in their end, after having spent the majority of the game in the other end.  If only we could play the last two minutes in their end, must be the wish of every goalie.  Corsi wanted to put a number to that phenomenon - which side of the rink is the play in - and came up with the number that bears his name today.  He tallied attempted shots, whether they made it to the net, or not, as proxies of play being in one end or the other.  The team without the puck cannot attempt shots, so the correlation is quite good. 

The analytics nerds seized on Corsi like wild dogs on bacon.  It had several virtues, in their estimation.  In the first place, it had a larger sample size than goals or shots on goal.  After all, shots attempted are quite a large number, and they mean you have the puck near the other team's net.  Larger sample size means the likelihood of flukey measurements drops.  Quality of team, teammates, and opposition is still a problem, as it was with plus minus, but that did not dissuade the numbers geeks.  They would break down an individual players' numbers into finer and finer compartments, with various teammates, where sample size starts to become limiting, in order to keep the number relevant.  But there's one thing they won't admit, but that I suggest plays an outsized role: Corsi does not have many accumulated years of confounding data, like plus minus did.  It's new, so it hasn't been disproven.  That makes it the perfect father figure they've always longed for.

Corsi has one thing going for it: If all other things are equal, meaning the skill quotient of both teams, tactics, and luck, then Corsi is a good predictor of the outcome.  After all, if you're in their end all the time, you should score more, and win more.  An attempt was made to neutralize the effects of luck, with the use of PDO.  That's a statistic that adds together a team's save percentage, and shooting percentage.  Across the league, that number should be 1.00.  In other words, of all the shots taken, those saved and those that score have to add up to the number taken.  So if a team has a high PDO, it is deemed to be luck, and is expected to regress towards 1.00.  Likewise, if too low, it should regress upwards.  Recognized exceptions include a hot goalie, having a team stacked with skilled shooters, or both.  But only to an extent, beyond which, it is deemed luck.

​The cumulative effect of the adoption of these analytic numbers is that the nerds have closed their eyes to confounding factors, and I'll now discuss several.
 
1.  Tactics. 
A team that has its defense bail out early on the opponent's blue line, and has a forward aggressively apply back pressure will not keep the puck in the opponent's end as regularly.  This should go without saying - an aggressive pinch will often succeed, where an early bailout will leave that route open to the opponent.  This dictates fewer shot attempts for, and more against, because the odds increase that the opponent takes the puck down to the other end.  Likewise, a team that aggressively pinches at the blue line will boost their shot attempts for, and cut down on shot attempts against, by virtue of the puck staying in the opponent's end. 
So should this calm the tense goalie at the other end, late in the game?  Hah!  It doesn't happen late in the game.  The team with the lead adopts the tactic of falling back, and allowing the opponent out of their end.  Is this an error?  To answer that, we have to look at the downside of a strategy designed to boost the team's Corsi number.
If the aggressive pinch could reliably produce a Corsi of 100%, then there would be no issue.  If all the shot attempts are at the other end, your own goalie can have a cup of coffee on the ice without fear.  But of course, that's not how things work.  A lop-sided Corsi might be 60-40.  In other words, one team has 60% of the shot attempts, the other team, 40%.  And therein lies the rub.  A team playing back will give up more shot attempts, but they will be there to limit the damage.  Some teams defend by putting all five skaters into their own slot, thus clogging passing lanes and shot lines.  Good use of the stick, and they can prevent any shot attempts from actually getting to the net.  So while the number of shot attempts might be high, the success rate on those shot attempts will be low.  The current New York Islanders are an excellent example.  They are not appreciated by the analytics crowd, because they give up a lot of shot attempts, but they consistently allow fewer goals than others.  Conversely, the team that pinches aggressively may boost their shot attempts, but when the puck actually does get past their defenders, it invariably produces odd-man rushes, or outright breakaways.  And those have disproportionately high scoring percentages.  The defensive-minded team will give up a lot of shot attempts but allow few goals, while taking fewer shots, but those they get will tend to be high percentage shots, when the opponent over-commits.  But when the analytics nerd looks at the PDO of the two teams, he will call the defensive team lucky, the offensive team unlucky.  That is of course false, because the tactic determines the result.
As if recognizing this in some way, the analytics nerd will attempt to normalize his numbers with score-effects.  It stems from the observation that the leading team will adopt defensive tactics, the trailing team, offensive tactics.  But it takes a bit of judgment to realize it's the tactical adjustment that is responsible, and that teams may choose such tactics by default, regardless of the score. 
 
2.  Rush Versus Cycle.
Teams that rely on the rush for offense, like the Leafs prior to the current season, would have very high shooting percentages, and were outshot in most of their victories.  It is probably impossible to produce thirty odd man rushes in a single game, since the opponent adjusts.  But produce five, and a couple will turn into goals, usually tipping the balance of the game.  By contrast, a team that relies on the cycle to produce chances has to rely on brute force as well as quickness, to gain and hold the puck, to move it to the dangerous areas, where a player has to out-muscle defenders to create a little time and space, and take a shot on net.  It is difficult to routinely produce offense using this system, and a fortunate deflection is often needed.  Those teams will produce more shots and fewer goals than those who rely on the rush.  But if an opponent takes away the rush, they often open themselves up for extended possession in their zone, where the cycle kicks in.  A successful team ultimately needs both ways of producing offense.  It can then adjust to the tactics used by their opponents, taking whichever approach is offered to them.  But where a matchup relies heavily on either rush or cycle, the analytics will contain a bias, making one team look bad, the other good, when in reality it could be the reverse.  The result might be the better descriptor of the game than a number like Corsi or PDO.
 
3.  Skill divergences.
A team composed mostly of puck-moving defensemen will spend the majority of their time in the opponent's end, and produce strong Corsi numbers.  Their PDO may not be the best, but that will be attributed to bad luck.  But again, it's not luck as much as it is tactical choice, driven by the composition of the blue line.
By comparison, a blue line dominated by shutdown defenders not great at moving the puck will not force the play to the other side nearly as much as the first example.  But they will be physical, and make operating in their end difficult for their opponents.  They may adopt a defensive strategy, and will allow a larger number of shot attempts.  But those shot attempts will produce few goals.  At some point, forwards will spring loose on odd-man rushes, and will score at a high percentage.  The team will be called lucky based on their PDO.  But again, it stems from tactics, dictated by team makeup.
Defensive skills get no attention at all, anymore.  Because there's no handy number to quantify them.  The rationale given is usually some version of noting that nobody who did not score in junior ever became a top shutdown defender in the NHL.  Scoring is synonymous with skill, and if you lack hockey skills, you won't make the NHL.  That is extended to the point of error, when the argument goes that the higher the skill level as reflected in offensive metrics, the better the player will be in all situations.  Why it's in error is because it disregards the ten years it often takes for a top defender to hone his defensive skills and good judgment, never mind the strength he needs to be dominant near his net.  A puck mover with flair realizes points are going to get him his next contract, so learning the defensive side is often not a priority. 
It's telling that after last year's loss, Tampa picked up Luke Schenn, who at that point looked washed up.  He's a shutdown defender rather than a puck mover, and yet they understood he brought something they needed.  He has skills that turned out to be valuable after all, even if they're not of the same variety as Erik Karlsson. 
 
4.  Gaming the system.
When teams think they have a secret tactic not available to their opponents, they will look to game it for their benefit.  So if every team has the epiphany that we can't be scored on if we have the puck, they will adopt tactics to give them maximum puck possession time.  Whereas before, puck possession might have had value as a way to compare teams because they were unaware that they were being measured for it, now it's an objective in itself.  And any time a predictive factor turns into a lever to be pulled, it loses its value.  Ask any experienced stock trader - a winning formula quickly gets copied, and when everyone does it, it can't be a winning formula any longer. 
 
5.  Suspicion of the eyes.
Kyle Dubas supposedly said that your eyes are liars, when asked about what is ridiculed as the eye test.  How did he know this, other than the fact that he could cite examples where the conclusions reached by the eyes did not agree with his numbers?  The question is not merely rhetorical; he has a department he calls R&D, which suggests he thinks they engage in research.  But research involves changing a variable and testing its effect on the outcome; difficult to do in something as complex as a hockey game.  What his department is likely doing is called data mining.  That is, analyzing any metric they can lay their hands on, and looking for correlations with wins.  And I should point out that data mining is considered bush-league by legitimate statisticians.  Basically, you can find all sorts of spurious trends in a mess of data, and they mean nothing by themselves.  It's like picking letters at certain intervals in the Bible, or Shakespeare, and claiming to find hidden messages.  It's not denying the value of those texts to point out that this is a misuse.

​In the end, success requires wisdom, and analytics doesn't furnish it.  You need to be able to watch a game, see the places where the lineup fails, and fix those places.  And you need to be able to do it in defiance of some number that suggests a player is a gem, when it's clear that's not the case.  And that means trusting what you see, and stubbornly sticking to it.  Someone who does it right time after time is worth quite a bit of money.  By contrast, if wisdom was reducible to a number, no executive would be worth more than a respectable middle class salary.  Ask yourself this:  Since Lou Lamoriello went to the Islanders and Kyle Dubas took over the Leafs, what do the fortunes of both teams suggest in terms of which approach works?