Evaluating Goalies - Part I

First off, I very much appreciated the response after my first post (in a long time) last week. The comments were insightful and I'm ready to do it again.

Today I want to look at how to evaluate goalies. I want to think of this as a two-part discussion. First, I want to simply use logic, reasoning, and first principles in trying to decide which statistic provides the best indication of a goalie's talent. Next week, for the second part, I'd like to run some year-to-year correlations for different goalie stats - ES save percentage, wins, GAA, and shutouts - and see which appears to have the highest correlation.

At the outset, I think there are two particular things that make evaluating goalies so difficult. First, the difference in talent among goaltenders today is pretty small. A goalie with a Sv% below .900 is likely not going to be in the NHL too long, yet a goalie with an additional three one-hundredths in terms of Sv% (.930) is universally considered an elite NHL tender. Tom Awad wrote a great deal about this a while ago, and if you look at the confidence intervals for the 40 goalies he surveyed, you can see how little they vary. Indeed, in another article on goaltender performance, Tom concluded that:

However, the amount of year-to-year variance [in save percentage] due to luck is greater than the spread among NHL goaltenders.

Given the minimal differences among NHL goalies, and the fact that luck can play such a powerful role from season to season, we're naturally led to affix great significant to minor things, like short-term performance or "clutch" saves.

The second reason evaluating goalies is so difficult is because of team effects. No one questions that the skaters in front of a goalie can affect his save percentage. On a particularly bad night where the team is sluggish and the defensemen lack focus, it's likely that the shots they give up will be of a higher quality. Those higher quality shots will depress the goaltender's save percentage because their higher quality means more are going in. The big question is whether this is true on the team level from season to season. So while the Blue Jackets may be really awful one night, are they consistently so bad that team effects actually have a significant and measurable impact on the goalie's save percentage in the long-run?

Many are inclined to say no, and there is some evidence to support this. Vic Ferrari's look at how defensemen affect save percentage is one small piece of the puzzle. But there are other equally well done pieces which provide some basis to think there are team effects on save percentage -- see this piece by JLikens. Overall, this debate is unsettled, and I don't want to open it up because I don't have the insight to provide a useful answer to this question.

So with that in mind, let's use some logic, reasoning, and first principles to understand which goalie stat is better. I think there are four stats in the mix: wins, GAA, shutouts, and even strength Sv% (ESSv%). I use even strength save percentage instead of total save percentage because Eric T over at Broad Street Bullies did some great work and found that:

[I]n the long run, after 150+ games, even strength save percentage is the better predictor of a goalie's future success [as compared to total Sv%].

Lets start with shutouts. I start here because I think this is easy. Shutouts are very uncommon (sample size problems), and the difference between giving up no goals in a game and one goal in a game would probably be attributable exclusively to chance. In a twenty game span, lets say goalie A has 3 shutouts and a .900 ESSv%, and goalie B has 0 shutouts and a .925 ESSv%. Who do you choose? I've never heard anyone suggest goalie A, precisely for the reason that those games in a which a shutout happened are such a small fraction of the goalie's overall performance. Combined with the luck argument, I think it's safe to conclude that shutouts are a poor way of evaluating goalie talent.

Next up is GAA. I'm also inclined to think this is a poor measure of goalie performance. Goalies have no control over how many shots they face, as that is almost entirely a product of the skaters in front of them. One might suggest that's not entirely true because goalies could use rebound control to affect how many shots they face. The reasoning might go that poor goalies that give up lots of rebounds create more shots against for themselves, while good goalies who control their rebounds limit the amount chances they face. The contrarian goaltender looked at this problem and found rebound control at the NHL level was a minuscule component of the goalie's performance. Thus, whether a goalie faces 45 shots in one game and 20 in the next has as much to do with him as the usher in section 435.

Given this, it's an easy inference to see that GAA is bad. The reason is that GAA is heavily dependent on shot volume, since even a goaltender sporting a .950 Sv% is going to have a high GAA is he faces a lot of shots against. Given that goaltenders have absolutely no control over shot volume, using GAA as a measure of goalie talent doesn't seem to make much logical sense.

The next one is wins. This is big. There are many on the web who think measuring goalie performance by wins makes sense. This is because winning is all that matters at the end of the day, and the goalie who wins the most must be the best. Whether there are other objections to this line of reasoning, I see one big problem with it: preventing goals is only half the equation to winning hockey games -- you also have to score goals.

Goalies have no control over how many goals their team scores. Even if a goalie has a stellar game and only gives up two goals against, he won't get a win if his team can't score more than two. Penalizing the goaltender who had a stellar game because the guys in front of him didn't play well doesn't make any sense if we're trying to figure out how good this goaltender is.

A corollary argument to using wins to evaluate goaltenders tends to look at how "clutch" a goalie tends to be. I'm usually skeptical of such claims for a number of reasons. First, the contrarian goaltender has a nice post explaining why clutch play doesn't really tell us anything. Second, it hinges on the idea that the goalie can just turn this clutchness on whenever he needs to, since if he can't turn this clutchness on when he needs to make a clutch play, then he isn't a clutch goaltender. But this implies that some goalies take nights off or decide not to give it their all on certain occasions. I find this tough to believe. Moreover, if a goalie is truly clutch, how can he ever lose important games? Even a "clutch" goalie like MAF loses big games. The fact that all goalies lose games (some important and some not) means there really isn't such a thing as clutch performance.

I think the following thought experiment helpfully illustrates what is really going on. Take a 40 game span. During the first 20 games, goalie A is struggling with a .900 ESSv%, but he's fortunate to make the playoffs because he plays on a really good team. Then over the next 20 games, which all occur in the playoffs, he regresses upward (which is not unusual) by having a .930 ESSv% over that 20 game span. With that stellar save percentage and a good team behind him, goalie A wins the Stanley Cup. "This guy is so clutch," says Mike Clutchbury. "I'm gonna sign him to a 15 year contract."

But wait a second. Over the 40 game span, goalie A has a rather average ESSv% of .915. That's not terribly impressive. But for those who say "F&%# numbers," we can still see this is messed up if we just reverse the order of events. Say, instead, that through the first 20 games, goalie A plays above his head with an ESSv% of .930. Then he regresses downward (which is not unusual) and sports a .900 ESSv% over the next 20 games. Even with his good team, those numbers aren't good enough to win in the playoffs, and goalie A loses. Everyone is going to say "this guy isn't clutch," and he just "can't win when it counts."

Yet goalies have no control over when they go hot and cold. And using clutch play as a measuring stick for goalies highlights the problem with this: whether someone is clutch can simply turn on the pure luck of getting hot at the right time, something we know is not an actual skill.

That brings us to ESSv%. Perhaps unsurprisingly, this is my goalies statistic of choice. Many other very prominnet hockey bloggers agree. The goalie has a lot of control over how many pucks he stops, and by only looking at the rate at which he stops pucks, we ignore things he has little control over like goals scored and shot volume. Of course he doesn't have total control over this -- if team effects are truly pronounced in the long-term, a goalie's save percentage will be affected accordingly. But the point is that the goalie has way more control over this stat than any of the ones surveyed above. That this is most representative of individual talent means it should be the go-to stat when trying to make an individual determination about a goaltender's talent.