Win Shares & EWA: A Comparison
Posted by Neil Paine on April 1, 2009
After we linked to a Kevin Pelton post a few days ago that compared his WARP stat to John Hollinger's new PER-based EWA metric, we thought to ourselves, "Hey, we've got our own 'wins created'-type metric! Why don't we compare it to EWA?"
So here we go. In case you didn't already know, Win Shares uses Dean Oliver's offensive and defensive ratings to determine a player's contribution in marginal points on offense and defense; these numbers are then linked to wins by dividing by the league's marginal points per win. We don't want to toot our own horn or anything, but it's pretty accurate: when we look at all NBA teams since the 1977-78 season, the average absolute error for WS vs. "real" wins is 2.72 wins.
In his article, John didn't provide the number of games an entire hypothetical team of replacement-level players would win, but simple math indicates that EWA would put a whole team of replacements at about zero wins over an 82-game season. In fact, it looks like the replacement level is set below absolute zero -- when one adds up the league's total EWA, it actually exceeds the number of games that have been won this season.
That's not very good news for EWA's accuracy in predicting team wins. In fact, here's the average absolute errors for each season when we use both EWA and WS to predict team wins:
+---------+-----------+----------+
| Year | EWA Error | WS Error |
+---------+-----------+----------+
| 1978 | 4.17 | 2.67 |
| 1979 | 5.48 | 2.77 |
| 1980 | 7.16 | 3.52 |
| 1981 | 5.68 | 2.68 |
| 1982 | 6.39 | 2.24 |
| 1983 | 7.41 | 2.50 |
| 1984 | 5.21 | 2.96 |
| 1985 | 5.02 | 3.13 |
| 1986 | 5.01 | 3.67 |
| 1987 | 4.70 | 2.73 |
| 1988 | 4.46 | 1.79 |
| 1989 | 4.50 | 2.41 |
| 1990 | 5.52 | 3.45 |
| 1991 | 5.36 | 2.10 |
| 1992 | 4.88 | 2.85 |
| 1993 | 4.85 | 3.15 |
| 1994 | 5.67 | 2.78 |
| 1995 | 4.84 | 3.55 |
| 1996 | 5.77 | 2.45 |
| 1997 | 7.38 | 3.46 |
| 1998 | 6.93 | 2.58 |
| 1999 | 4.29 | 1.78 |
| 2000 | 6.50 | 3.29 |
| 2001 | 5.21 | 2.57 |
| 2002 | 4.83 | 1.95 |
| 2003 | 5.13 | 2.19 |
| 2004 | 7.10 | 2.50 |
| 2005 | 6.36 | 2.44 |
| 2006 | 5.73 | 3.61 |
| 2007 | 4.99 | 2.67 |
| 2008 | 5.02 | 2.69 |
| 2009 | 5.91 | 2.24 |
+---------+-----------+----------+
Cumulatively, EWA checks in with an average error of 5.56, which is more than twice that of Win Shares. And if you look at the data, there isn't a single season in which EWA outperforms WS.
Individually, there are also significant discrepancies between WS and EWA. I don't know about you, but my first reaction to EWA was, "do the EWA totals look sort of... high?" Here are the top ten EWA seasons since 1977-78 (we used a constant repacement level of 10.82 for all players rather than making the position adjustment, but it doesn't really make a difference in the data):
+------------------+---------+-------+-------+
| Player | Year | EWA | WS |
+------------------+---------+-------+-------+
| Michael Jordan | 1988 | 34.41 | 20.35 |
| Michael Jordan | 1989 | 32.90 | 19.08 |
| Michael Jordan | 1990 | 32.39 | 18.68 |
| David Robinson | 1994 | 31.99 | 18.94 |
| Michael Jordan | 1991 | 31.42 | 19.84 |
| Shaquille O'Neal | 2000 | 31.20 | 18.70 |
| Michael Jordan | 1987 | 30.95 | 16.05 |
| Kevin Garnett | 2004 | 29.93 | 18.10 |
| LeBron James | 2006 | 28.83 | 16.13 |
| Michael Jordan | 1993 | 28.81 | 16.48 |
+------------------+---------+-------+-------+
As you can see, EWA is routinely crediting superstars with almost twice as many wins as WS does. Both systems seem to deal in total wins, which means the extra wins that got transferred to the stars had to be taken away from somebody... So where did those wins come from?
Players like Bruce Bowen...
+-------------+---------+-------+------+
| Player | Year | EWA | WS |
+-------------+---------+-------+------+
| Bruce Bowen | 1997 | 0.01 | 0.01 |
| Bruce Bowen | 1998 | 0.13 | 2.28 |
| Bruce Bowen | 1999 | -1.43 | 0.01 |
| Bruce Bowen | 2000 | -1.78 | 1.13 |
| Bruce Bowen | 2001 | -3.65 | 4.02 |
| Bruce Bowen | 2002 | -2.25 | 1.91 |
| Bruce Bowen | 2003 | -2.20 | 5.06 |
| Bruce Bowen | 2004 | -3.44 | 5.30 |
| Bruce Bowen | 2005 | -1.67 | 5.95 |
| Bruce Bowen | 2006 | -2.32 | 6.05 |
| Bruce Bowen | 2007 | -4.50 | 3.96 |
| Bruce Bowen | 2008 | -4.62 | 4.19 |
| Bruce Bowen | 2009 | -3.45 | 2.08 |
+-------------+---------+-------+------+
...and Shane "Moneyball" Battier:
+---------------+---------+-------+------+
| Player | Year | EWA | WS |
+---------------+---------+-------+------+
| Shane Battier | 2002 | 4.79 | 4.82 |
| Shane Battier | 2003 | 4.93 | 6.36 |
| Shane Battier | 2004 | 4.38 | 5.96 |
| Shane Battier | 2005 | 4.95 | 7.58 |
| Shane Battier | 2006 | 5.49 | 8.64 |
| Shane Battier | 2007 | 1.78 | 8.87 |
| Shane Battier | 2008 | 1.20 | 8.01 |
| Shane Battier | 2009 | -0.58 | 4.19 |
+---------------+---------+-------+------+
...and good old Mark Eaton (remember him?):
+------------+---------+-------+------+
| Player | Year | EWA | WS |
+------------+---------+-------+------+
| Mark Eaton | 1983 | 0.17 | 0.87 |
| Mark Eaton | 1984 | 1.61 | 3.88 |
| Mark Eaton | 1985 | 3.84 | 5.60 |
| Mark Eaton | 1986 | 1.73 | 4.02 |
| Mark Eaton | 1987 | 0.32 | 3.64 |
| Mark Eaton | 1988 | -1.75 | 4.19 |
| Mark Eaton | 1989 | -0.26 | 6.39 |
| Mark Eaton | 1990 | -0.29 | 5.18 |
| Mark Eaton | 1991 | -1.69 | 5.39 |
| Mark Eaton | 1992 | -1.78 | 3.79 |
| Mark Eaton | 1993 | -1.18 | 1.89 |
+------------+---------+-------+------+
That's just a small sample of the players who are undervalued by EWA. And what do these 3 guys have in common? In a word, defense. That's how they made a living in the NBA -- they played tough defense. And PER doesn't really care much about defense, at least not beyond defensive rebounds, blocks, steals, and personal fouls.
That's not necessarily an indictment of PER, because by JH's own admission it's not designed to take into account anything except what's included in the box score. But maybe that very fact means that trying to convert it to wins is a mistake -- EWA clearly doesn't correlate well to wins at the team level, which in turn calls into question how well it can measure an individual player's "wins added."
As Justin noted in a discussion we were having yesterday, "PER is a nice summary stat, kind of like OPS, but it misses quite a bit. It should not be used to get to player wins, just like you wouldn't want to calculate wins above average using OPS and plate appearances." I think that's a good analogy, and not just because I'm excited about baseball season starting up -- PER can definitely tell you about a player's per-minute box-score contributions, but extrapolating it out to wins just doesn't look like the best way to utilize the metric.
April 1st, 2009 at 8:50 am
Neil,
You refer to player WS/ews 'predicting' team wins, but you're actually just comparing same-season totals -- right?
This year, you have WS-Wins error as 2.24 . The Pythagorean-Wins 'error' is about 1.69 . So one might presume the WS-Pyth error should be closer to .55 . Is that a more reasonable result to know?
April 1st, 2009 at 11:39 am
Neil,
It's cool to see the relative accuracy of the WS method in terms of win determination.
How does it stack up to Dave Berri's Wins Produced?
When I look at things like PER, EWA, or WP I wonder if we wouldn't be better served developing a system where stats are valuated differently for different positions. It's not as clear cut as in some sports, but there is definitely a distribution of labor in the NBA. While it's great to acknowledge and reward versatility, maybe a PG, whose role is specifically to set up teammates, should get a little extra credit for assists and have smaller impact associated with his rebounds or blocks, which the team is typically going to rely on big guys to provide. Just as a matter of emphasis in the metric.
Or maybe I'm way off-base. I just look at stats like WP and find it odd that in order to even the playing field there need to be position adjustments done after the fact, as though guards require a handicap to produce wins at the same level as bigs do. Yes rebounding and efficient scoring are paramount, but how important is the space provided by a shooter or a clean lead or entry pass to getting that big man his high percentage shot opportunity?
Then we get back to the whole issue of assigning roles and positions in the first place, and it gets to be an even messier concept.
April 3rd, 2009 at 6:08 am
You had an earlier posting that was very enlightening about how defensive rebounding stats don't differ that much. You plug any big man into the center or power forward position and he'll get at least 6 or 8 rebounds per 36 minutes, most of them defensive. In contrast, Moses Malone got lots and lots of offensive rebounds that nobody else would have.
Still, some people are just bad rebounders. Dennis Rodman talks about how he could tell when the shot left the shooter's hand where the carom would wind up. In contrast, in my inglorious career as a 6'4" playground player who routinely got shut off the boards by low-flying stocky 5'10 guys, I seldom knew where the ball would bounce after already bounced.
Here's an idea for a topic: Who was the worst rebounder (relative to size) of all time? Some of seven-footer Brad Sellers rebounding statistics are astonishingly bad, but there was probably somebody worse out there.
Another possibility is to compare shotblocking to rebounding and look for the biggest contrasts. You'd think they correlate well, but I was a fine playground shotblocker despite not being able to jump and being pretty clueless. I think Mark Eaton would come up in any list of people who could block but not rebound real well.
April 3rd, 2009 at 11:33 am
Steve - That's a really interesting point about being able to plug in any 4 / 5 to get that rebound range. That's one of those instinctive things that raises my hackles when I look at WP (though I really like Dave Berri's thoughtful approach to things) - as an indicator of team wins it's very strong but as a rater of individual contribution it may over-reward highly replaceable stats.
I think with the blocks / rebounds correlation experiment you'd find Rodman and Barkley would be majorly off the curve in the other direction (as opposed to the Eaton's the world) because they were never much in the way of big number shot blockers but they have two of the highest rebound rates of any 30+ min/g players in NBA history.
On Dennis's anticipation of bounces off the rim, he used to study film of teammates and opponents to find where the ball would come to when they normally missed from each area of the court. It was a science to him.