I found an interesting documentary on Netflix, a segment from the book Freakonomics. In the segment, economists examine the widespread epidemic of cheating and corruption in everything from inner-city school systems, to professional sumo wrestling in Japan.
Cheating in the Public School System
Economists had a hunch that teachers were cheating for students on their tests. Trends in the data collected on exam responses revealed that there was almost assuredly foul play. Specifically, the data showed that students who were constantly marking incorrect answers early on in their exams began to show a trend of having almost no incorrect answers in the latter part of exams. I found this interesting because intuitively, one would assume the latter part of exams would be the source of tougher questions under tougher time constraints; as nearly every exam in schools nowadays have time constraints. What these economists ultimately found, was that students were running out of time and leaving the latter portion of the scantrons blank. These blanks provided an opportunity for teachers who were desperate to show an improvement in their teaching abilities, to fill in the latter portion of the exams for the students. Instead of risking going back through the exam and erasing incorrect answers, teachers would simply fill in correct answers toward the end.
It is rather amazing that the manipulation of big data would reveal such trends. The findings of corruption in the school systems led to policy reformation in an attempt to try and fix the underlying learning issues. Students from the University of Chicago carried out experimentation in local high schools by utilizing incentives to improve test scores, grades, and eventually high school graduation rates. To do so, the University provided a grant that would allow monetary compensation to high school students contingent upon if they brought their grades up to a 2.0 GPA. Every month, students that were successful in maintaining a 2.0 would receive $50 cash as well as entry into a raffle to win $500. The result: after nearly a year of experimentation, graduation rates increased at the local high school by nearly 7 percent.
Corruption in Professional Sumo
Big data was also used to expose corruption in other facets of life. Another example: professional sumo wrestling in Japan. Economists again went on a hunch and rumors that professional sumo wrestlers were throwing matches in major tournaments in exchange for cash and the promise of other matches being thrown. The incentive for doing so: ranking in sumo is used to determine classes as well as overall compensation. The typical sumo tournament would last 15 matches and at the end of the tournament, those wrestlers who had at least 8 wins would move up in class, and those who failed to achieve 8 wins would fall in class. The difference in one class is huge; it could be the difference in $5,000 a month in compensation. Wrestlers also share a very tight-knit bond. From a young age, the wrestlers convene at a gym and practice under the watch of a gym leader for years. Much like Spartan culture, the wrestlers would eat, sleep, and train together year-round.
Analysis of tournament outcomes revealed that wrestlers who had 7 wins going into their final match against an opponent who had already succeeded in locking in the necessary wins to advance in class, were 75 percent more likely to win that final match. Ironically enough, the next time those same opponents met, the victor of the previous match almost always lost. Sumo carries an almost sacred quality, therefore very few, if any wrestlers, have been willing to come forth and admit to corruption. The few who have however, have revealed that large sums of cash and future matches have been exchanged for thrown matches.
I believe that these findings have shown that big data can be instrumental in probing for, and finding corruption in companies, governments, and sports. I urge you to watch the segment for yourself, and checkout the book by Steven Levitt.