So, Who’s This “Replacement Player” I Keep Hearing About, Anyway?
Posted by Neil Paine on March 31, 2009
This past week, ESPN.com's John Hollinger rolled out a new stat that compares a player's PER (Player Efficiency Rating) to that of a "replacement-level" player, in an effort to incorporate minutes played into an evaluation of the player's worth. Said JH:
"The idea behind Value Added is to take the difference between a given player's performance and that of a 'replacement level' talent -- the type of guy who might be sitting at the end of a team's bench, or perhaps in Sioux Falls -- and multiply that difference by the number of minutes that player played. The result shows, theoretically, how many number of points the player added to his team's bottom line on the season.
VA is very useful for award voting in particular, because it allows us to compare players with disparate production and minutes -- say, one player who was brilliant in 60 games and another who was merely good but played all 82 -- and figure out which performance was more productive."
This is actually just the latest in a long line of "value over replacement"-type metrics, which actually had their genesis in baseball -- in the mid-90s, Sabermetrician Keith Woolner invented VORP as a method of "correctly valuing durability and playing time versus rates of production". Since that time, it's basically become the de rigeur way to bridge the gap between counting stats and rate stats, since it essentially reflects the economic reality of pro sports (the talent pool is quite large and there is a certain level of production for which you can pay the bare minimum).
Baseball's replacement level is relatively easy to find, given that the definition is "the expected level of performance an MLB team will receive from one or more of the best available players who can be obtained with minimal expenditure of team resources to substitute for a suddenly unavailable starting player at the same position". Because baseball has very clearly-defined positions, you just look at the aggregate performances of the backups at those positions, and there's your replacement level.
But, as we often lament in the statistical community, basketball ain't baseball. There aren't clearly-defined positions, for one thing -- just about everyone can play at least 2 positions in a pinch, and some can play 3 or even 4. Also -- and perhaps consequently -- the delineation between "starter" and "backup" isn't so clear (for instance, an All-Star-caliber player like Manu Ginobili isn't technically a starter). So you can't simply look to all "non-starters" when determining the replacement level, at least not like you can do in baseball.
In addition, starters are "replaced" with a different caliber player than bench guys. If Dwyane Wade is injured, his replacement is somebody like Daequan Cook, who is much better than your typical 10-day contract/NBDL call-up; likewise, if Cook is hurt, somebody like Luther Head or Yakhouba Diawara can slide in. It's only when you get down to the Dorell Wrights of the world that you truly start to fill minutes with NBDL-caliber talent, and that's about 3 degrees of depth-chart separation from Wade himself. So it doesn't really reflect the reality of basketball to simply measure the difference between Wade's production and the production of an NBDL player in the same minutes, because he's not actually being directly replaced by the NBDL guy.
I've suggested in the past that what's sometimes called "chaining" could be the solution for this problem, but I'm not even entirely sure that method accurately reflects the reality of player replacement in the NBA. So I guess my question is, what do you think? Is the concept of "value of replacement" as valid in basketball as it is in baseball, or is the different nature of the two sports (roster-wise) a major stumbling block toward devoping "VORP" for NBA players?
March 31st, 2009 at 8:45 am
For a metric like Hollinger's PER, how much does the accuracy of the replacement level really matter? As long as it is calculated the same way every season, won't it give the same baseline for the comparisons to work and keep the number range the same? I guess it boils down to an "accuracy" vs. "precision" argument. Go high school science! Of course if the numbers could be both precise and accurate that would be great, but for the purpose of the advanced stats that use it, I would think that consistency would outweigh exactness.
I do think you've got a helluva good point with the fact that it's so hard to determine definitive positions and even true starter quality. Guys like Manu & Terry, who are clearly among the top 3 or 4 players on the team, are going to skew things in a way that baseball (at least the position players in baseball) doesn't have to deal with. And even some great players are difficult to classify by position. Tim Duncan comes to mind immediately.
March 31st, 2009 at 10:46 am
How high/low you set the replacement level can matter a lot in that trade-off between counting stats and rate stats, though. Generally, the lower you set the R.L., the more you're going to emphasize counting stats -- which makes sense, given that counting stats technically have a "replacement level" of absolute zero. Conversely, the higher you set the level, the more you emphasize the player's rate of production (for instance, setting average = 0 considers everyone below average to have negative value). And if you took it to some really absurd length like "Value Over Kobe Bryant" or something, practically everyone in the NBA would be negative, how much would just depend on how close your rate was to Kobe's.
March 31st, 2009 at 11:39 am
I actually emailed JH about this the other day. My problem with value over replacement player is that your not replacing LeBron with the 12th guy on the bench if he gets hurt. You are definitely not replacing him with an NBDL type talent. I like the idea of chaining based on the depth chart. Perhaps look at something like Value over First Replacement (VOFRP) which could average all of the first guys on the depth chart at that position (i.e. the theoretical replacements for a starter if they go down). I do think it's also true that as long as the methodology is consistent, the baseline for the comparisons will be similar.
If you wanted to take it another step, you could simply determine a league average Value Added and a league average value added by position and measure VORP based on that as opposed to using the 12th guy on the bench as a proxy. I suspect that the rankings will be similar regardless (guys like Kobe, D Wade, and LeBron are going to look great on this regardless of how its calculated. Their just good)!
March 31st, 2009 at 12:32 pm
Neil, I would say by the time you get down to Luther Head and Yakhouba Diawara you're already getting into replacement-level talent. Diawara was signed for a little over the minimum and Head, after he was waived by the Rockets, is surely making the minimum. That makes them pretty freely-available too.
March 31st, 2009 at 12:34 pm
I don't see the problem comparing people to the average. You won't get a value over replacement, you'll get a "how far does someone deviate from the average" stat, but that seems quite useful to know.
Alternately, you could go down a standard deviation from the average if you want a lower-performing baseline. Also, basing it on standard deviations would be a pretty intuitive way to look at just what the number means. Of course, I don't know what the distribution looks like in the NBA. Probably has a wicked tail at the top end with the Lebrons, Chris Pauls, Dwight Howards, and Wades of the world sticking out like a sore thumb.
March 31st, 2009 at 1:12 pm
I suppose that's true, Kevin. Head's performance in Miami has been much better than what you'd expect to get from a replacement, but he was pretty close to the replacement level in Houston, and Diawara's really right on it. Then again, both players have seen some significant playing time in their careers -- this year, it's especially been Diawara despite his performance (which, granted, was better last year). That's why I felt they were a notch above pure D-League-level players, because if they were that bad, you'd think they'd see fewer minutes.
March 31st, 2009 at 1:26 pm
Rashad, the problem is that average players aren't easy to find. They create value for their teams by playing instead of replacement-level players. A metric that measures players against average doesn't give them the credit they deserve.
Neil, it's my opinion that the D-League standard is a little too low for what freely-available talent really means. Training-camp invitees or pretty much anyone playing for the minimum would seem to be freely available to me. That's the standard I used in my replacement level manifesto several years ago:
http://web.archive.org/web/20040325075607/http://www.hoopsworld.com/article_7557.shtml
March 31st, 2009 at 2:57 pm
That's a good piece, Kevin, I hadn't seen that before. I guess this calls for a study of what PER your typical minimum-salary player is able to put up...