Danny Crawford and the Mavs
Posted by Neil Paine on April 19, 2011
ESPN had an interesting news story today about Danny Crawford's history with the Dallas Mavericks in the playoffs:
The gist is this:
"The Mavs have a 2-16 record in playoff games officiated by Crawford, including 16 losses in the last 17 games. Dallas is 48-41 in the rest of their playoff games during the ownership tenure of Mark Cuban, who has been fined millions of dollars in the last 11 years for publicly complaining about officiating."
At the risk of running afoul of the Wyatt Earp Effect again, what is the probability that this has happened due to chance alone?
In his book Mathletics, Wayne Winston finds that the final margin of victory in an NBA game can be approximated by a normal random variable with a mean of the point spread and a standard deviation of 12. Using that knowledge and the handy chart ESPN provided at the bottom of their story on Crawford, we can calculate the probability of Dallas winning each of their Crawford-officiated games since 2001:
| Round | Year | Date | Team | Site | Opp | Result | DAL PTS | OPP PTS | DAL Pt Diff | DAL Line | p(DAL W) |
|---|---|---|---|---|---|---|---|---|---|---|---|
| 1st Round | 2001 | 4/28/2001 | DAL | Home | UTA | W | 94 | 91 | 3 | -5.5 | 67.7% |
| Conf Semis | 2001 | 5/14/2001 | DAL | Road | SAS | L | 87 | 105 | -18 | 9 | 22.7% |
| Conf Semis | 2002 | 5/13/2002 | DAL | Road | SAC | L | 101 | 114 | -13 | 3.5 | 38.5% |
| 1st Round | 2003 | 4/27/2003 | DAL | Road | POR | L | 79 | 98 | -19 | -2 | 56.6% |
| Conf Semis | 2003 | 5/6/2003 | DAL | Home | SAC | L | 113 | 124 | -11 | 2 | 43.4% |
| Conf Semis | 2003 | 5/15/2003 | DAL | Road | SAC | L | 109 | 115 | -6 | 5 | 33.9% |
| Conf Finals | 2003 | 5/23/2003 | DAL | Home | SAS | L | 83 | 96 | -13 | -1.5 | 55.0% |
| 1st Round | 2004 | 4/26/2004 | DAL | Home | SAC | L | 92 | 94 | -2 | -6.5 | 70.6% |
| 1st Round | 2005 | 4/25/2005 | DAL | Home | HOU | L | 111 | 113 | -2 | -7 | 72.0% |
| Conf Semis | 2005 | 5/9/2005 | DAL | Road | PHO | L | 102 | 127 | -25 | 7 | 28.0% |
| Conf Semis | 2006 | 5/7/2006 | DAL | Road | SAS | L | 85 | 87 | -2 | 4 | 37.0% |
| Conf Finals | 2006 | 6/1/2006 | DAL | Home | PHO | W | 117 | 101 | 16 | -6.5 | 70.6% |
| NBA Finals | 2006 | 6/13/2006 | DAL | Road | MIA | L | 96 | 98 | -2 | 5 | 33.9% |
| NBA Finals | 2006 | 6/20/2006 | DAL | Home | MIA | L | 92 | 95 | -3 | -6 | 69.1% |
| 1st Round | 2007 | 4/27/2007 | DAL | Road | GSW | L | 91 | 109 | -18 | -3.5 | 61.5% |
| 1st Round | 2008 | 4/29/2008 | DAL | Road | NOH | L | 94 | 99 | -5 | 7 | 28.0% |
| Conf Semis | 2009 | 5/3/2009 | DAL | Road | DEN | L | 95 | 109 | -14 | 6.5 | 29.4% |
| 1st Round | 2010 | 4/23/2010 | DAL | Road | SAS | L | 90 | 94 | -4 | 3.5 | 38.5% |
So let's simulate that set of games 10,000 times and see how often Dallas has each possible record.
| Record | Count | Prob |
|---|---|---|
| 18-0 | 0 | 0.0% |
| 17-1 | 0 | 0.0% |
| 16-2 | 2 | 0.0% |
| 15-3 | 13 | 0.1% |
| 14-4 | 58 | 0.6% |
| 13-5 | 168 | 1.7% |
| 12-6 | 466 | 4.7% |
| 11-7 | 957 | 9.6% |
| 10-8 | 1538 | 15.4% |
| 9-9 | 1932 | 19.3% |
| 8-10 | 1881 | 18.8% |
| 7-11 | 1487 | 14.9% |
| 6-12 | 913 | 9.1% |
| 5-13 | 389 | 3.9% |
| 4-14 | 160 | 1.6% |
| 3-15 | 31 | 0.3% |
| 2-16 | 4 | 0.0% |
| 1-17 | 1 | 0.0% |
| 0-18 | 0 | 0.0% |
Yep, out of 10,000 simulations, Dallas ended up with a 2-16 record or worse...
5 times.
That's 0.05% of the time. Now, maybe the zombie corpse of Wyatt Earp is going to eat my brain for saying this... but given their point spreads, it seems like there's very little chance that Dallas' poor performance in Crawford-officiated games has happened due to random variation alone.
April 19th, 2011 at 4:11 pm
Yes if you read ex-NBA ref Tim Donaghy's book "Personal Foul" he reported Crawford bragged that Dallas had lost 15 of the 16 playoff games he had officiated to that time. Remember that Dwyane Wade shot 46 free throws in the last two games of the 2006 Finals (25 in game 5 and 21 in game 6 I believe). Included was a phantom foul at the end of game 5 that gave Miami a 1-point win. and another one at the end of a very close game 6 where he pushed off Nowitzki, who never touched Wade. Very interesting...Stern and Cuban have obviously had a lot of issues and it seems clear that the Mark's outspokenness has hurt him.
April 19th, 2011 at 4:48 pm
Is there a possibility that while the record is not due to chance, it is also not due to a particular vendetta by Crawford? Is it possible that Crawford's particular style (whether he's lax with physicality down low and strict with it on the wing, or whatever) just clashes with the styles of the Mavs' main players, who were mainstays through those losses (Terry and Dirk in particular)?
Would there be anyway to test that? Could we look to see if the Mavs are consistently under-performing in one specific area in those games?
Lots of questions there, I know...
April 19th, 2011 at 5:54 pm
Right, this just means there's something unique to Dan Crawford-officiated games that causes Dallas to play worse than Vegas (or anyone else) would expect.
It's not necessarily evidence of a scandal; really, does anyone think the NBA would be so obvious if there was some kind of conspiracy against Dallas? A more likely explanation is what you suggest -- that there is probably something endemic to both the Mavs' playing style and the way Dan Crawford calls a game that makes them more prone to lose.
April 19th, 2011 at 5:56 pm
Wyatt Earp!!! This kind of analysis really is flawed and the stats sports community needs to be more aware of it, because it really does influence the way people think about Danny Crawford, the Mavs, and the refs in general. Basically, if you look at enough team-by-ref records there are bound to be weird outliers like this. The only reason someone focused on this one is because they noticed it after the fact (post hoc). It's not appropriate to then go back and test it using the analysis you did above. You would either need to track the outcome of Crawford-Mavs games going forward OR you need to correct it by the number of possible comparisons you could have made (i.e., all of the ref-by-team combinations). I bet if you went out and dug into all of the possible combinations there really would be several other outliers out there that are close to or even more extreme than this one. The hoops stats community really needs to be more aware of this because it shows up all the time!!!
April 19th, 2011 at 6:02 pm
@Ben: Does it? If it shows up all the time, what the other examples of it? What other referees show such extreme results in situations like this?
April 19th, 2011 at 6:10 pm
Very interesting. I would add as a factor intimidation on the part of the Mavs. If they know Crawford is the ref, given the past record, that creates a distracting amount of stress. Especially since the Miami finals, him being in the game would impact the players psychologically. In addition to having to beat the rival, you have to beat the ref. That alone is enough for the NBA to avoid placing certain refs in certain situations (when the record is irregularly against or in favor of certain teams). I think the comparison of the ref's record with a team vs. all the other refs' records of the team should be a constant assessment. If a ref has more than a couple teams where their reffed games are different from the teams regular record, investigation is merited. It doesn't have to be necessarily about corruption, or the implication of such -- although Donaghy's quotes about Crawford are scary. It's just a bad idea to have the refs impact the course of play prior to the game in any way, regardless of whether bias or past bad luck are involved.
April 19th, 2011 at 6:15 pm
Regarding the theories that his reffing "style" conflicts with the Mav's style of play, I'd have to disagree. Over the course of this record, the lineup of the Mavs has one consistent factor - Dirk. Different lineups and coaches have had different styles of play, and I don't think Dirk alone would result in this kind of streak. More likely it's either psychological - or else maybe there really is some bias...
April 19th, 2011 at 6:34 pm
Read the Donaghy book. He makes a compelling case for certain refs having biases vs. specific teams, coaches, players and in Cuban's case, owners. The fact that Donaghy wrote Crawford "bragged" about Dallas' record in playoff games he reffed is pretty suggestive. Assuming his reporting is true, which I tend to believe, seems pretty damning to me.
April 19th, 2011 at 6:35 pm
#4 - Look, I even anticipated this might be a case of our old friend Wyatt in the post. But I still don't know if it meets the conditional aspect of the fallacy.
Part of the point about the non-notability of Earp winning so many consecutive duels was the fact that his probability of winning duel D + 1 was contingent on him winning duel D; that is, Wyatt's probability of winning any given duel in his streak was contingent upon him winning the previous duel (otherwise he'd be dead). It's survivorship bias. But I'm not sure Dallas' probability of losing their 14th Dan Crawford-reffed game was contingent upon them losing the 13th, etc.
Unlike the Wyatt Earp example, each game here is essentially an independent event, barring some psychological effect like lemonbrigade suggests in #6. The fact that each event is conditional is what defines the Wyatt Earp effect, and I'm not sure these games are conditional.
April 19th, 2011 at 6:50 pm
I wonder if the Donaghy book is what spurred the article on Crawford. I never heard of an article like that on a specific ref. I do recall a study being done on Earl Strom, aka the road ref, years ago. I believe the study showed that road teams won 29% of games during one NBA season but won 43% of the games Earl officiated. Interesting.
April 19th, 2011 at 6:53 pm
This is not exactly Wyatt Earp, but there is a selection bias. So I wonder, how many team-referee pairs are there with 15+ games? If the answer is of the order of a thousand (compare to the 0.05%), it's not convincing. If less than a hundred, it is. That is, it's convincing that there is a phenomenon here, not sure what phenomenon it is, of course.
April 19th, 2011 at 6:54 pm
I mean 15+ playoff games, of course, in #11.
April 19th, 2011 at 7:01 pm
No matter how you look at it when Danny Crawford refs our games we lose. Thats all there is to it. We have lost 6 in a row when he has refed in our games. Hopefully tonight that streak ends
April 19th, 2011 at 7:16 pm
Yariv is right, it's not exactly the Wyatt Earp effect. The general problem, though, doesn't have to do with the conditional aspect: it is about selecting a particular outcome (the 2-14 record) post hoc and then going back and calculating the odds of that occurring randomly. Basically, there are a ton of ref-by-team pairings out there and if you looked at all of them you find some bizarre outliers. Some of this could reflect real biases (intentional or unintentional), but much of it would just be random. Unless you had some reason a priori reason to suspect this specific pairing, you can't ignore all of those other possible pairings when you determine the significance of the event. This doesn't mean that there isn't something going on in Mavs' games that Crawford refs! It just means that you are over-estimating the rareness of this outcome. And lots of smart people have fallen prey to the same mistake (http://www.fivethirtyeight.com/2010/03/incredible-izzo-again-defies-odds.html). In the field I work in the problem is referred to as circularity because you select something on the basis of being unusual and then test how unusual it is (http://neurocritic.blogspot.com/2009/01/voodoo-correlations-in-social.html). It would be great if more people were aware of this issue and if the commentary on these kind of unusual stats took this into consideration.
April 19th, 2011 at 7:18 pm
Sorry, Neil, you were also right about it not being the Wyatt Earp effect in your last comment.
April 19th, 2011 at 7:28 pm
One analysis I'd really love to see: how do current Mavs' personal average foul differentials in Crawford-reffed games compare to their averages pre-Mavericks? And how do ex-Mavericks' Crawford foul differentials compare to when they were Mavs? This study would go a long way to indicate whether Crawford has some specific bias against the team or whether we have just accumulated players over the years that he finds to foul more often regardless of whether they are Mavs or not...
April 19th, 2011 at 10:24 pm
I bet this article affects crawford's calls toward dallas tonight, if anything...
April 19th, 2011 at 11:24 pm
Funny thing is, if Crawford is suspected to be biased against the Mavs, and the Mavs end up winning the next several Crawford-officiated games in a row, it will lead to more questions than answers. That is, would we just 'return to the mean', or would Crawford over-compensate to disprove the charge that he hates the Mavs?
April 20th, 2011 at 1:07 am
Relevant: The webcomic XKCD touches on what Ben mentions - the framing of supposed significance - http://xkcd.com/882/
Neil, might you agree with me that this is just a case of the Prosecutor's Fallacy?
April 20th, 2011 at 7:17 am
I find it interesting that their first win came against Utah of the Stockalone vintage -- a team that, according to some Utah players, was public enemy number 1 to, you guessed it, Danny Crawford in the 90's.
I wonder what that W/L stat looks like.
April 20th, 2011 at 7:42 am
But what lead to Tyson Chandler grabbing -3 defensive rebounds?
http://www.basketball-reference.com/boxscores/201104190DAL.html
April 20th, 2011 at 9:54 am
#19 - Yes, that sounds more like what this is a case of. But the solution is ... what? Find the historical probability of any referee having a 2-16 (now 3-16) stretch of games with a specific franchise? (Unfortunately we don't have ref data.)
And at the same time, nobody can deny that if I gave you a set of games with those win probabilities, and asked you to calculate the odds of a 2-16 record emerging from that set of games, you would come back with ~ 5 in 10,000.
April 20th, 2011 at 10:37 am
Vegas made some money last night!
April 20th, 2011 at 11:11 am
Neil, I would argue that what you DON'T do is put this stat out there without any correction, since it is does overstate the rareness of this outcome (see that brilliant XKCD comic!). And I'm not trying to pick on you personally, Neil, since I do think this mistake comes up frequently. I think one of the best solutions would be to go back and find all of that ref data and figure out all of the ref-by-team pairings. That's obviously a pretty big research project, since that information is not readily available, but I think it would be very illustrative. The other option that I think is reasonable would be to go back and find the first written mention you can of someone discussing the "Danny Crawford is biased against the Mavs idea." I feel like I heard this mentioned after the Finals appearance against the Heat, but I'm not entirely positive (the refs were definitely discussed, but I'm not positive the focus was on Crawford). If you can find a mention then, I would argue that you could construe that as an a priori prediction about how Crawford-Mavs games would turn out in the future. Then you could go test the games since then, since you have a legitimate prediction. Obviously, though, the 0-4 record since then won't be nearly as much of an outlier. I just don't think it's fair to put out this stat and not correct for the clear post hoc selection bias.
April 20th, 2011 at 11:48 am
Ben -
I understand your point, but it already has been tracked going forward, since 2005, when the Mavs' record was 1-8 in playoff games officiated by Dan Crawford..
April 20th, 2011 at 12:04 pm
That's great, M.W.! If you can find a solid source for that then I think it makes sense to generate calculate the probability of going 1-8 in the games after that. I'm really not against the particular argument being made here. I just hate that people notice something unusual, then run out and calculate the probability of that thing occurring even though they already knew it was unusual! In many of these cases they could be just overreacting to noise!
April 20th, 2011 at 1:09 pm
I'll see if I can find a minute to locate something for you. Of course, it's not 1-8 any more since then, is it? It's now 2-8.
April 20th, 2011 at 2:32 pm
Do we even need to know the record of every team-ref pairing? The data here says it would happen 5 out of every 1000 occurences with an "occurence" being defined as 17 playoff games played by a particular team and refed by a particular ref. Now, every occurence will have a slightly different probabibility (based on the win probability of the team in question versus its opponent), but I think it is safe to say they will all fall roughly into the same area. As such, all we need to do is find 200 "occurences" and we have a pretty solid shot of it happening at least once and 1000 occurences to find it happening about 5 times.
April 20th, 2011 at 2:34 pm
That last statement may be just a TAD simplified. But the point still stands, I think. Given a large enough sample size, even events that are exceedingly rare are likely to happen. My hunch is that the sample size of team-ref pairing in the playoffs of >= 17 games is high enough that we could reasonably expect an outcome like this to happen. Then again, my sense of how many times a given ref has reffered a teams playoff games 17 times might be wildly off base.
April 20th, 2011 at 3:55 pm
This is great conversation.
History occurs in two ways. Conspiratorially or Accidentally. Your piece clearly shows that numerically this was obviously no accident. So of course there was conspiracy!
Numbers rarely lie. I've always brought up the Miami series when discussion of referee involvement arises. Wade became a superstar at the stripe on phantom calls that generally occurred away from the basket. The players who set records for FT attempts are guys that the opponent wants to put at the stripe at the end of games like Shaq or Howard, not players who are hitting 90% like Wade in that series.
But for me it isn't as simple as examining the individual referee, in this case Mr. Crawford. No conspiracy is as simple as placing the blame on one individual. It's like blaming the president for all the worlds problems when there are so many forces in play. Crawford is just one piece in what is probably a complicated process.
The NBA powers that be have to be considered the major players in this conspiracy as they keep putting refs in situations where they know it creates justified questioning. To believe that they are unaware of Crawford's, and every referees scenario, is short sighted. The NBA is nothing more than an entertainment business and they attempt to control as much of the production as possible.
As a fan of the game and the League, it is a bit tough to swallow to think players can be relegated to be little more than puppets in an elaborate play. However, it is one more reason why comparing championship rings when comparing individual players is irrelevant...
April 20th, 2011 at 4:43 pm
This seems apropos.
http://www.xkcd.com/882/
April 20th, 2011 at 7:32 pm
does anyone know if crawford refereed either game 5 or game 6 of the 2006 Finals? in game 5, wade shot 25 free throws himself, equal to the entire dallas team. miami shot 49 - 49! think about that,virtually twice as many as the mavs.
in game 6, wade shot 21 FTs and the mavs took 23 as a team compared to 37 by miami. hmm. i do recall that the decisive play, a foul at the end of game 5, was a phantom whistle on a wade drive that made his FTs the difference in a 101-100 win.
i think crawford also refereed game 3 when dallas blew the big lead late which allowed miami to get back in the series instead of being down 3-0.
April 20th, 2011 at 7:40 pm
ok i just read some of the ESPN article and saw crawford officiated game 6 as well. i think it was predictable that dallas won last night given all the belated attention given to crawford's history with the mavs.
April 21st, 2011 at 12:31 pm
I don't really get why the Wyatt Earp effect is such a big deal. Yes, I understand that given a purely random situation, seemingly improbable events have to occur (ie, given enough attempts, there will be a time when I flip a coin heads a hundred straight times). Nevertheless, Wyatt Earp is only a fallacy if somebody tries to use a statistic as incontrovertible proof. If instead you're asking how likely it is that a statistic is random versus caused, then there's no reason you can't do this sort of analysis.
Example: given a purely random distribution and a large enough sample, it is inevitable that one day a terrible basketball team will win all 82 games by more than 10 points a game. Should that prevent us from evaluating how good a team is based on their wins and win margin? No. Because it's more likely that there is a cause behind their success than it is that it was random.
April 23rd, 2011 at 1:48 pm
There was some BS against Dallas in that Miami/ Dallas Finals. To now know this crap about Crawford AFTER I already thought something was fishy----is just bad.
May 6th, 2011 at 12:12 pm
I'd love to have all of the stats on ref-team pairings. You could generate a distribution with mean and st dev based on all of Crawford's pairings and see the probability that 2-16 is from the same distribution. You could do the same for the Mavs with all of the other refs. And see the distribution for the entire league.
Short of that, a Fishers T-Test helps investigate a 2-way table like this. Given 48 wins and 41 losses in non-Crawford games and 2 wins and 16 loses in Crawford games, the p-value for the two-tail test is 0.0013. This means, if we assume that Crawford had no affect on those games (this was random), and we wanted to be 99% sure of our conclusion, we would reject the hypothesis. Crawford is a statistically significant variable when considering the Maverick's win-loss record.
It doesn't mean he causes losses, but I cannot conclude that Crawford is an independent variable. The correlation between Crawford and Mavericks losses is not a random occurrence.