Below you can read the discussion with Dr. Robin Boylorn about the Third Reviewer Model, and the impact it has had on her journal so far.
What is the Third Reviewer Model, and why did you decide to use it with your journal?
In the standard peer review process, the editor will find and contact a minimum of two reviewers. The reviewers are experts in the related research field, and they will be invited to review each article. We implemented the Third Reviewer Model for our journal, to allow graduate students and early career researchers the opportunity to gain reviewer experience, and better understand the peer review process.
The Third Reviewer Model helps clarify the peer review process. It allows new and upcoming researchers to get an ‘insider’ view of how research papers are evaluated. We also use the Third Reviewer Model as a strategy towards diversity, because it helps improve the reviewer pool by bringing more women, more researchers of color, and more junior and early career researchers to the peer review process.
How do you find researchers to act as third reviewers, and find faculty to sponsor or mentor them? Could you share some guidelines or processes for implementing the Third Reviewer Model?
I post ‘calls for participation’ every year to relevant divisions within the sponsoring organization, listservs, on social media, and by word of mouth. I also allow self-nomination or faculty nomination of students.
Third reviewers must be at least ABD (All but Dissertation). If they are ABD, they must have a faculty sponsor who agrees to serve as a liaison for them (this is usually their academic or dissertation advisor). The liaison will answer their questions and oversee the process.
Third reviewers can also be recent PhDs or early career researchers, who are interested in gaining experience and exposure to the journal review process. Liaisons and mentors are only required for those who are ABD or very recent PhDs. There isn’t a formal application process for third reviewers. I make it open to anyone eligible (ABD, or PhD, or with research expertise, or experience in the areas relevant and related to the aims and scope of the journal) and interested (willing to review 1-3 manuscripts a year, as assigned) to be added to the third reviewer list. This process makes sure it is accessible and erases any possibility of hierarchy or exclusion.
How do you weigh the third reviewer’s opinions differently?
Third reviewer publication recommendations are not considered equal to expert or editorial board member recommendations. So they are not ‘tie breakers’, but their qualitative feedback is provided to authors to help improve revisions. And, when possible, I offer feedback to the third reviewers with suggestions for future assessment.
Third reviewers also have access to other anonymous reviewer feedback after decisions are made. This can offer insight into what they may have missed, or what they may have helped highlight for revision.
What is the impact of the Third Reviewer Model on your journal?
The new model has brought a range of positive impacts to our journal:
It gives new and early career researchers an opportunity to learn how to give a constructive review, and how they themselves are evaluated.
It gives authors additional feedback to consider in revision (even if third reviewer recommendations are not given equal weight).
It creates a network that opens up the possibility for more people to gain peer review experience sooner and helps them to progress in their career.
Journal reviewer experience positions them for editorial board memberships in the future.
It helps to have fresh eyes and fresh perspectives. Sometimes early and new scholars are more aware of what is emergent in the field.
But there are some downsides too. For example, it requires extra help. In most circumstances, third reviewers are truly a third review – which means you have to be mindful that two reviewers for the same manuscript are already secured before assignment.
It also requires extra time resource. Third reviewers may require guidance and correction – and while this will ideally come from a sponsor, it sometimes requires my intervention as editor. I would recommend that anyone considering this model, makes sure that ABD reviewers have a sponsor or mentor who can consistently and reliably oversee their efforts.
It is also worth noting that ScholarOne doesn’t support labeling third reviewers. So, you have to trace these reviewers independently, adding more to your workload.
Can you give some details on how you keep track of the status of reviewers, such as how often you use them, their availability, and their spheres of interest?
I have a reviewer Excel spreadsheet that my editorial assistant updates regularly. We refer to the spreadsheet when making assignments – it includes the date of the last review assignment. The spreadsheet also includes the reviewers’ areas of interest, expertise, and topics they would prefer to opt out from.
We also use the reviewer database provided within ScholarOne. This helps identify reviewers based on their previous and relevant publications. You can also use the ScholarOne system to determine how recently a reviewer submitted a review. I encourage editorial board members to update their status within the ScholarOne system, for example, if they are unavailable or on leave. This means that we won’t reach out to them while they are away. And, it saves us time, as we don’t send review invites, to those we know can’t accept.
Guide to becoming a peer reviewer
Get an overview of what’s involved in becoming a peer reviewer for a Taylor & Francis journal.
Expert peer review training
Develop your peer review knowledge with our short form modules any time that suits you. You can also book on to our live workshops.