Researchers at the Wharton School have revealed that authors’ self-assessments of their artificial intelligence papers could serve as a valuable tool for identifying influential research often missed by traditional peer review processes. This finding, published on August 24 in Nature Computational Science, stems from a study analyzing how authors ranked their submissions to the 2023 International Conference on Machine Learning.
Self-Ranking Shows Potential for Identifying Impactful Research
The team, consisting of professors Weijie Su and Bingxin Zhao, along with doctoral student Buxin Su, sought this alternative metric in response to the rapid growth of AI research outpacing the availability of skilled peer reviewers. The researchers emphasized the importance of providing a comparative rank, as authors were restricted from ranking all their papers equally, forcing them to be objective in their evaluations.
In this study, a total of 1,342 authors provided rankings for 2,592 submissions shortly after the submission deadline, prior to the release of peer reviews. The research team subsequently analyzed data from 797 authors and 1,527 unique papers, pairing these rankings with citation data gathered in the following months.
Findings indicated that papers ranked highest by their authors garnered approximately twice the average citations compared to those ranked lowest within a 16-month period. Notably, of the 22 papers that achieved over 150 citations, 17, or roughly 77%, had received a top ranking from at least one of their authors. Moreover, the study demonstrated that author rankings could predict future citations more reliably than traditional peer-review scores.
While the study acknowledged that citation counts are not an infallible measure of quality—highlighting various factors that influence citation frequency—it proposed a novel approach for prioritizing papers needing additional scrutiny. The system involves comparing author rankings with reviewer scores to identify discrepancies. Papers exhibiting significant differences are flagged for area chairs responsible for making acceptance recommendations.
Area chairs are not privy to whether the author’s score is higher or lower than the review score but can assess the extent of the differences. The goal is to allow for a more careful review of potentially significant submissions without creating opportunities for authors to unduly manipulate their rankings. The International Conference on Machine Learning has integrated this self-ranking approach into its 2026 review process, conducting a randomized experiment to evaluate its effectiveness. Observations from the experiment noted a 71% increase in comment lengths by area chairs on flagged papers compared to those who did not have access to discrepancy information.
Su remarked on the broader implications of this system, suggesting that author self-ranking can enhance the efficiency and effectiveness of the review process, enabling conferences to allocate their limited resources more strategically toward papers worthy of closer examination.
Why It Matters
This study underscores the potential for innovative metrics in academic evaluation, particularly within rapidly evolving fields like artificial intelligence. By adopting self-ranking methods, conferences may improve the quality of scholarly reviews and ultimately enhance the visibility of significant research contributions.
