Prophy Predicts: The Future of Academic Assessment [Webinar Recap]

This is a recap of the Prophy Predicts webinar — "The Future of Academic Assessment" — hosted on March 26, 2026. The full recording is available on YouTube.

The journal impact factor was invented in 1955 to help librarians decide which journal subscriptions to renew. It was never designed to determine who gets hired, who gets funded, or whose research gets taken seriously.

And yet here we are.

That was the unspoken premise running through our second Prophy Predicts panel—a 60-minute conversation with four people who sit at different corners of the scholarly publishing ecosystem: a funder-facing database founder, a language expert embedded in editorial offices worldwide, a metascience researcher who's spent years pushing preprints into the mainstream, and a platform builder whose clients include the European Research Council.

We asked each panelist the same question ahead of the session:

How do you think academic assessment will have to change over the next 2–5 years—and what will that mean for scholarly publishing?

What came back was sharper, and more honest, than we anticipated.

About the Webinar

Prophy Predicts: Future of Academic Assessment brought together four panelists for a moderated discussion on where research evaluation is heading—and what it means for publishers, funders, and researchers navigating the transition.

Panelists:

Judy Mielke — Founder & CEO, Scientify Research

Avi Staiman — Founder & CEO, Academic Language Experts

Dr. Jonny Coates — Founder, Rippling Ideas

Paul Tuinenburg — Founder, Global Campus

Moderated by Gareth Dyke, Partnership Director at Prophy.

Judy Mielke: Real-Time Assessment, Faster Peer Review, and a Fairer System

Judy Mielke opened by grounding the conversation in first principles: academic assessment exists because there isn't enough money to go around. Funders have to choose, and they need good information to choose well. The question is whether the current system gives them that.

Her answer: it doesn't. Not because the incentives are wrong in theory, but because the mechanism is too slow, too opaque, and concentrated at the wrong moment.

"Research assessment is too slow," she said. "A Nature paper typically takes years of research, then even more time for peer review and publication. So there's a lot of delay. Assessment needs to be more real-time, more transparent."

Her prescription is broader than timeline. She called for recognition at every stage of the research process—not just the final publication, not just the commercial product, but every step of the way: data shared, peer reviews completed, collaborations built. She pointed to the 80/20 problem in peer review specifically: a small minority of researchers carry the review burden for the entire field, while others contribute nothing. Her suggestion—publish five papers, review ten—is deliberately provocative, but it names something real.

From her position running a funding database that works directly with funder organisations, Judy also observed a shift already underway: more and more emphasis on real-world impact. Whether a Nature paper is still the appropriate unit of assessment is a question funders are beginning to ask seriously. Publishers should probably get ahead of the answer.

Avi Staiman: The AI Disclosure Gap Is Splitting Academic Publishing in Two

Avi Staiman self-describes as an analyst, not an idealist. His role in this conversation: honest about where things are actually heading, even when it's uncomfortable.

His central prediction: the flood of AI-assisted submissions overwhelming editorial offices will force a bifurcation in academic publishing. Journals with strong internal editorial teams and the resources to do rigorous triage will manage. Everyone else will feel the strain, and serious researchers—facing a landscape full of doubt about legitimacy—will consolidate around the publishers with centuries of brand credibility behind them.

"Some of these brands have been built over the course of hundreds of years," he said. "There's something about the power of that kind of gravitas which I think is going to become even stronger in a world filled with doubt."

The foundation of this prediction is a single statistic that stopped the panel: 50% of researchers report using AI somewhere in their workflow. Three percent disclose it. The gap isn't dishonesty—it's a structural problem. Publishers and editorial offices have said: use AI, just tell us how. That sounds simple. As Avi explained, lurking behind that simplicity is a lot of confusion and complication.

Policy inconsistency is part of it. But the deeper issue is a trust penalty: research shows that disclosing AI use systematically reduces perceived manuscript quality. Reviewers—human and AI alike—rate disclosed-AI work lower than identical undisclosed work. Researchers are making a rational decision to stay quiet, and no amount of policy language changes that calculus until the penalty for honesty is removed.

The editorial office experience, as Avi describes it from direct conversations with journal editors, is not a future scenario. It's happening now. There's a deluge. Already strained editorial staffs are spending more and more time on the "middling section"—papers that read fluently, look legitimate, but carry something slightly off. Is it a non-native English speaker who chose an unusual phrase? Or AI? The cost of getting it wrong runs in both directions.

Dr. Jonny Coates: Building More Trust Signals for Academic Research

Jonny Coates comes to this question as someone who's spent years trying to change the culture of academic publishing from the bottom up—through preprints, through open science advocacy, through metascience research that examines the system itself.

His prediction sits somewhere between Judy's idealism and Avi's sobriety: "It really pains me to say this, but I think I kind of agree with Avi on how journals are responding to all of these things going on." Journals are doubling down on peer review as their primary response to AI uncertainty. Jonny thinks that's the wrong move.

"I think part of the way we're going to combat things like the rise of AI is actually through a much greater variety of trust signals," he said.

What does that mean in practice? Peer review is a trust signal—it tells you that at least two or three people have read the work and commented on it. But as a signal, it's weakening: reviewers are increasingly running submissions through AI rather than reading carefully. Journal prestige is a trust signal too, but not a reliable one. Jonny cited Nature fast-tracking high-profile AI research by bypassing their own peer review policies as evidence that even the most established publishers will compromise the signal when the incentive is strong enough.

His argument: build more signals, more diverse ones, that are harder to fake. Are datasets shared and reproducible? Is the code usable and available? Are open science practices documented across the full research lifecycle, not just at the point of publication? These are the markers that can't easily be gamed, and Jonny argues they'll become the primary basis for evaluating research credibility over the next five years. The word he kept coming back to: trust. Not metrics, not rankings—trust, built from evidence that can be independently verified.

He also made the case for preprints as one of the few tools that genuinely facilitate the cultural change the system needs. Mandates don't change things, he argued—they shift with whoever's running the funding body. What changes culture is slow, grassroots adoption of new practices. Preprints, by getting researchers outside their institutional bubble, are one of the few mechanisms that actually do that.

Paul Tuinenburg: Goodhart's Law Has Been Running in Academia for Decades

Paul Tuinenburg spent ten years running workshops at Dutch universities telling researchers something they didn't want to hear: the average academic paper is read by approximately three people. That experience, watching the gap between research output and societal impact up close, is what led him to build Global Campus—a researcher-matching platform now used by the European Research Council and publishers and universities across Europe.

His contribution to the panel was the structural critique that framed everything else: Goodhart's Law.

"When you know what is being measured, you're going to game it," Paul said. "That's a very human thing to do."

Citations. Impact factors. Grant income banked for the university. All of these became targets rather than signals. The result is a system that selects for the wrong things—researchers who are skilled at producing the metric, rather than researchers who are skilled at doing the science. Gareth Dyke added from personal experience: at a Russell Group university, his academic assessment came down to impact factor of publications and pounds of grant income banked. That was it.

Paul then shifted to what's actually changing—cautiously, and slower than reformers want, but changing. In the Netherlands, PhD funding tied to completion is prompting real conversations about whether mandatory publication counts make sense as a qualification threshold. In the UK, the Research Excellence Framework limits researchers to submitting five publications for assessment. Small design choice, meaningful effect: quality over volume, by structural constraint.

His closing argument was the one that cut through most cleanly: stop comparing across disciplines. "It doesn't make sense to compare an engineer and an immunologist and a humanities scholar with each other," he said. "So let's stop pretending a single number can do that." Let researchers define what quality looks like in their own context and hold them accountable to that explanation. It makes comparison harder. But the comparison was already meaningless.

The Problem Every Panelist Agreed On: Academic Culture Moves Slower Than Policy

Every panelist, regardless of how optimistic or pragmatic their individual prediction, circled back to the same obstacle: culture.

Institutions can update their written promotion criteria. They can sign DORA. They can endorse CoARA commitments. None of it reliably changes the intuitions of the people sitting on hiring and promotion panels. Judy put the paradox plainly: "The inertia in the whole system is actually in the individuals. Researchers are asking to be measured more broadly, but when they judge their colleagues, they still say—where did you publish your paper?"

A 2024 study examining UK REF data across more than 100 institutions found exactly this: despite widespread institutional DORA endorsements, the correlation between journal rankings and expert assessment scores was unchanged. Declarations don't move culture.

Jonny offered the most actionable version of this insight: "We keep talking about things being systems, and I think we need to shift that to people. All the systems and signs are run, created, and maintained by people. So we need to change people, not systems." That means graduate training that actually teaches the norms and practices of open, rigorous, collaborative science—not just technical skills, not just whatever the PI happens to model. We still train researchers like it's the medieval period, he said. An apprenticeship model, luck of the draw on your supervisor, no standardisation across the system.

What This Means for Publishers

If you work in peer review operations, research integrity, or editorial strategy, the practical takeaway from this session is not a list of new tools to acquire. It's a shift in how you think about where assessment risk enters your workflow—and what role publishers play in the broader ecosystem.

The panelists converged on a few specific directions worth acting on now.

Treat AI disclosure as a structural problem, not a policy problem. The 3% disclosure rate won't improve through stronger language in author guidelines. It will improve when the trust penalty for disclosing AI use is removed—and when publishers actively signal that transparency doesn't disadvantage submissions. This requires editorial guidance, consistent reviewer training, and visible commitment that goes beyond a boilerplate policy statement.

Invest in the reviewer infrastructure that better assessment requires. Every reform model raised in this session—whether it's field-specific quality criteria, open science practices as trust signals, or team-based assessment—depends on reviewers who have genuine expertise in what they're evaluating. Matching submissions to the right reviewers isn't just an efficiency question. It's the foundation of the credibility the whole system is trying to rebuild.

Build citation pattern analysis into editorial thinking. The bifurcation Avi predicts—established brands consolidating, everyone else struggling—isn't inevitable. Publishers who invest in detecting structural anomalies across submissions, building cross-stakeholder intelligence, and surfacing integrity signals earlier in the workflow are the ones who will hold their position in a more skeptical research environment.

Start the conversation about modular research outputs. The constellation model Jonny describes—linked preprints, datasets, methods notes assessed together rather than waiting for a complete journal article—is being actively piloted by bioRxiv and medRxiv. Publishers who engage with this now rather than later will have more influence over how the infrastructure develops.

No single publisher, tool, or policy solves the assessment problem alone. But the direction the panel pointed is clear: more trust signals, more diverse ones, applied earlier, with researchers rather than at them.

About Prophy

Prophy Predicts is an ongoing webinar series bringing together researchers, publishers, and industry experts to discuss the future of scholarly publishing and academic assessment. The next event will focus on the future of research funding—how grant review processes are changing and what that means for publishers and researchers navigating the shift.

Prophy helps publishers and funding agencies evaluate research and researchers at scale—through reviewer discovery, conflict of interest detection, and semantic analysis across 190+ million articles and 95+ million researcher profiles.

If you work in peer review operations or research integrity and want to discuss how your current workflow holds up against the pressures raised in this session, get in touch at prophy.ai.

This recap was produced by the Prophy team from the full session recording. Quotes have been lightly edited for clarity.