Jeff Zatkoff and Confidence Intervals

When the Penguins announced they placed Jeff Zatkoff on waivers yesterday, the reactions were mixed. Some people were upset because they thought Zatkoff was good, and others were happy because they thought he sucked. He admittedly has a small sample, so drawing firm conclusions is difficult. But in his 20 NHL games last year, Zatkoff finished with a .912 overall save percentage, which is better than Fleury's career average.

Nevertheless, what the Penguins did to Zatkoff ultimately doesn't matter one way or the other. This is so because we have no idea what kind of goalie he'll end up becoming. I took a look at his save percentage in the twenty games he played this year and came up with a 95% confidence interval for his play. At the outset, I thank a twitter user named Megan who sparked this idea after I read her outstanding piece on confidence intervals for many of the goalies in the NHL.

I use 95% confidence intervals because that is the typical benchmark for statistical significance. What that says is that there is a 95% chance that Zatkoff's true talent lies within the lower and upper bound of his 95% confidence interval.

The chart below contains those numbers for his 20 regular season games.

Pay attention to the upper and lower bound on that chart. You'll notice how enormous the gulf is between the two data points. What that tells us is intuitive: Zatkoff's sample is too small to reliably predict what kind of NHL goalie he'll be. The upper bound of his confidence interval suggests he could be the next Dominik Hasek; the lower bound suggests he could be worse than replacement level.

All this points to is that I don't think it's right to suggest that Zatkoff was definitively good or definitively bad. Superficially, his numbers are in line with Marc-Andre Fleury, but it's unclear whether he can keep up that level of production in the NHL. What the Penguins did was make an educated guess that it's unlikely Zatkoff's true talent lies toward the top of his confidence interval.

They have some reason to think that's the case. Zatkoff was 26 years old when he started this season. It's unusual for a goalie who has the true talent to be an NHL starter to begin his career so late. I used hockey reference to get the names of all goalies in the last 25 years who started their rookie season between the ages of 25 and 27.  There are a lot of duds on that list. But some of the more prominent names to play 10 or more games in their rookie season appear below.

Of course, that's not to say Zatkoff will end up like one of the better goalies on this list. It just means that we're unsure at this point given his small sample. Greiss has more impressive numbers than Zatkoff so far, so it makes a lot of sense to give him the job and see if he can take the starting role away from Fleury.

The Penguins made a reasonable bet yesterday by placing Zatkoff on waivers. Time will tell if it pays off.