During my graduate studies (’93-97), I looked at the history of prizes in science. I learned that from ~1600-1800, prizes funded science lots, and much more than did grants. But ~1830, science elites controlling top scientific societies, in both Britain and France, defrauded donors to switch funding to grants, which were then selected by society insiders to be given mostly to insiders. Thereafter such societies insisted that donors must fund grants, not prizes, if they wanted their donations to gain their prestigious scientific society association.
Later, ~1900, tenure became common in academia. Then mid-20th century, peer review became common in grants and publications. Also about then, journalism switched from its usual mode of questioning and investigating claims made to it, to accepting whatever academics said and trying to “communicate” that to the public. In the 1980s, college rating systems became widely available to the US public, ratings which depended mainly how how elite academics rated those colleges.
All of these changes were ways in which academic elites wrested control of academia from outsiders who previously could impose some degree of incentives and accountability. The elites of most any profession would love to fully control it, grabbing resources to spend at their discretion, with little need to accommodate demands of customers or investors or regulators or anyone else. But academic managed to achieve this ideal far more than have most, due to their peak prestige. Via elite schools, academic control prestige in many other areas of life.
I review this history to make clear just what academic reformers are up against. It is far from sufficient to enumerate academic failures; you’ll have to develop concrete alternatives that can win prestige fights against the usual academics. History has been moving against you; you’ll have to somehow reverse that strong tide.
In my 40+ years of thinking about how we might reform academia, I’ve considered many different parties as potential allies in this venture. First I and other hypertext publishing fans hoped to use backlinks to make criticism of claims easy to find from those claims, thus recruiting critics and honest readers into our reform venture. But we’ve now achieved that ease of finding criticism, without much impact. Readers care far more about publication prestige than which criticisms are persuasive with those who read them with care.
Second, I saw the public as an ally willing to bet lots on science and related policy questions. However, we’ve seen that if academics choose to ignore such bets, the public isn’t much interested in them either. And laws continue to block such bets.
Third, I saw research patrons as allies. Surely they’d want to fund research in ways more likely to induce intellectual progress, if only they understood the better ways. Like prizes instead of grants. But then I learned about the history of academia that I summarize above. No, patrons used to use better methods, but caved when academic threatened to take away their prestige by association. Patrons care more about such prestige with academics than they do intellectual progress.
Fourth, I hoped journal editors might be allies. But when we showed that polls and prediction markets could predict which papers wouldn’t replicate, and tried to get journal editors to publicly declare they’d consider such predictions as part of their article approval process, they all refused. Journals are happy to publish sexy papers that don’t replicate.
Now my best hope is to recruit as allies future folk willing to give honest appraisals of their distant past. One key claim that elite academics are not willing to give up on is this:
The people that academics now most celebrate with jobs, funding, publication, and publicity are in fact the people today who future folks centuries later, carefully considering the question, are most likely to identify as the people today who should have been listened most for the purpose of speeding intellectual progress.
Just as we can sometimes we can get auditors, judges, juries, and even journalists to give honest independent appraisals of others’ acts and accomplishments, there’s a decent chance that we can find a ways to fund “historians” (who might not be professional credentialed as such) to look back at particular areas of research, and rank past researchers in terms of who should have been listened to more. And the first order of business in this reform effort is to actually fund such efforts to rank researchers from centuries ago, to show that we can in fact robustly enough rank them now. Once we find that diverse approaches give correlated answers, we can search for approaches whose expected results are the most correlated with others. (The best approach might be to randomize of many approaches.)
Once we have demonstrated such a capacity, we could create markets today on assets that pay off proportional to (some monotonic transform of) rankings of particular researchers today. (Such assets should be built out of assets that accumulate long term value, such as stock index funds.) And we could make markets in such assets conditional on such evaluation exercises being done centuries later on their areas of research. This approach would thus be robust to the fraction of such areas later evaluated. It might be most all areas, or just a few, depending on how much funding becomes available, and how far in the future evaluations are done.
With market estimates of future rankings of current researchers, we could then highlight any contrast between such rankings and the people who academic elites now choose to celebrate, via jobs, funding, publications, etc. Such elites would then have to either (A), dismiss such markets as ignorant, (B) change their choices to better align with market estimates, or (C) trade in these markets to move their estimates to better match with their non-market choices.
Results (B) or (C) would give academics stronger incentives for, and thus rates of, intellectual progress. Academic institutions could then use such market prices as outcomes for futarchy-governance methods to choose jobs, grants, publications, etc.
To prevent (A), I’d start this approach in limited research areas, where I’d pick shorter term evaluation periods, and subsidize the markets enough to induce informative prices, which we’d then show to be informative at the end of those evaluation periods. Once prices were taken seriously in such areas, we’d switch to longer term evaluation periods. And then researcher efforts to manipulate their own prices could provide sufficient subsidies, allowing extra subsidies to be moved to new areas, to repeat this process. And hopefully with success, we’d attract more sources of subsidies.
And that’s my current best vision for reforming academia. Academia is one of the hardest social spheres to reform, given its peak prestige, and none of the groups you might hope to make allies here can actually be counted on. So my best ally hope is future “historians” looking back to say who should have most been listened to for the intellectual progress that was actually achieved.
During my graduate studies (’93-97), I looked at the history of prizes in science.
By Robin Hanson