11 Managing peer review

Understanding peer review

The reviewer’s task is to scrutinise the quality of a manuscript before publication to ensure it meets the standards of the scholarly community. Scholarly experts in the relevant discipline normally undertake peer review.

According to Kelly et al. (2014), the purpose of peer review is twofold:

• It acts as a filter – excluding poor-quality research.
• It improves the quality of work that is deemed suitable to publish.

The peer review process is widely accepted as a quality control process. It is criticised for being slow, biased and ineffective in screening against plagiarism and other forms of academic misconduct (Kelly et al., 2014). However, there are no alternatives that reliably address the flaws in the current system.

Peer review and book publishing

Commercial publishers have a wide range of peer review practices. Some publishers require authors to submit a detailed book proposal that is peer-reviewed, for example, Cambridge University Press [PDF, 1.2 MB]. Other publishers peer review the book proposal and full manuscript, for example, MIT Press.

Currently, Monash University Library (MUL) organises peer review of the entire manuscript, using a single or double-blind peer review model.  One of the key reasons for undertaking this kind of review is the authors’ concern about publishing independently verified, quality titles.  Another reason is to receive constructive feedback that helps authors strengthen their manuscripts.  Other forms of review for the MUL program are being investigated, e.g. student review of activities and exercises.

Different models of peer review

There are several approaches to peer review in academic publishing.

• Blind or single-blind. In a blind review, the authors are unaware of the identity of the peer reviewers. However, the reviewer may know the name of the author.
• Double-blind. In a double-blind peer review, neither the authors nor the peer reviewers know each other’s identities.
• Open peer review. In an open peer review, the author and peer reviewers know each other’s identities. This may include making the review itself public.

Table 11.1: Blind and double-blind peer review

Table 11.2: Open peer review

Open peer review

Many open textbooks utilise a form of open peer review. For example, the BC Campus Self-Publishing Guide suggests that authors ask a colleague to act as a Content Matter Expert (CME) for relevant parts of the text (Aesoph, 2018). The Open Textbook Library invites faculty members to post rubric-guided reviews on the website as a form of post-publication peer review (Open Education Network, n.d.). Open peer review is a common practice in open textbook publishing.

If you are thinking of utilising open peer review, there are several things to consider:

• Will the authors or Library team manage the review?
  • Itemise the tasks involved and who will be responsible for them.
• Will you use internal or external reviewers?
  • External reviewers may be one way to reduce the potential for bias.
• How will the reviewers be sourced? Options include:
  • Authors, to ask colleagues.
  • Source content experts and request a review.
  • Put out an expression of interest for reviewers.
• Will remuneration be offered and how will it be determined?
• How extensive will the review be: chapter, section, or book level?
• Will the review be pre or post-publication?
• Will the review be published?

As with blind reviews, open reviews need review criteria. Review organisers may wish to use spreadsheets to track information about the process.

Regardless of the method or requirements for peer review, the focus should be finding someone who is knowledgeable and prepared to provide a robust review which will improve the quality of the work.

Student review

Some open textbooks utilise student review instead of, or in addition to, peer review. Peer review involves assessing the work’s accuracy and relevance from the perspective of a fellow expert. This usually happens before publication.

Student review, on the other hand, may focus on assessing clarity, comprehension, and relevance from the perspective of the learner. This will usually happen post-publication so that edits can be made based on student feedback.

Student review is based on open pedagogy, which aims to involve students in using and developing open learning resources (Elder, 2019), giving them autonomy and ownership over their learning. This can be done in several ways; having students fill in a survey, using an annotation tool such as Hypothes.is to comment on the open resource, or designing an assessment around the evaluation of the learning resource.

Choosing a peer review model

A number of factors will determine the peer review process selected for your project:

• funding
• staff resources
• time
• intended goals of  the review
• the kind of book, e.g. activity-based workbook vs research-based textbook vs instructional guide.

If you are considering a blind review you may also wish to think about the following questions:

• Does the work contain any easily identifiable information? For example, videos of the authors or multiple self-citations?
• What is the specialty of the subject? Is blind review possible or desirable?
• Are resources available to arrange blind peer review?

Ultimately, the method of peer review will depend on the goals the authors have for peer review and the kind of feedback they consider most relevant to their work. Do they want two subject experts or one subject expert and one educational expert, a helicopter view over the whole text, or a specialist view for each chapter? Project managers should work closely with the authors to decide what kind of feedback they are looking for and which method, or combination of methods, would be most suitable.

Peer review statement

Regardless of the type of review undertaken, ensure that the review process is detailed in the book’s Peer Review Statement so that readers are fully informed and the review process is transparent.

The Statement should let readers know:

• the type of review  – that is,  blind, double-blind, open
• the number of reviewers
• the review criteria followed
• whether the review was conducted internally or externally
• the names of the reviewers (in the case of open review).

This is an example of the Monash approach to peer review statements. The Rebus Guide also has a comprehensive template that can be adapted as needed.

Monash OER peer review

An overview of the process

At Monash University, the Library publishing team manages peer review.

The project manager discusses the peer review processes with the authors so there is a shared understanding of what kind of review and reviewer is needed.

We utilised blind and double-blind peer review in our first four open textbooks. It is our preferred form of peer review for the quality control and reputational benefits outlined earlier in this chapter.

Each chapter was reviewed by two peer reviewers. For some of our works the same two peer reviewers reviewed the entire book. This was done in sections, with the reviewers being sent roughly a third of the book at a time.  Our suggested timeline for review was set at 3 to 4 weeks depending on the word count, however we asked peer reviewers to suggest alternative dates if this didn’t work for their schedules.

For some projects it was necessary to have a range of reviewers. This was  because:

• the book was very long and finding two reviewers to review all of it was too difficult
• there was a range of subject areas which required subject experts
• there were Indigenous authors or content that required Indigenous reviewers.

MUL works were peer-reviewed before being sent to the copyeditor and then published. Although some open textbooks are peer-reviewed after publication, our experience demonstrated that a high level of editing by the authors was needed after both peer review and copyediting. We decided that doing this post-publication would be unnecessarily complicated. However, if the work is needed quickly, to be used in teaching for example, then post-publication peer review may be more suitable for your project.

Finding reviewers

Finding peer reviewers is a time-consuming process.  Start searching for reviewers as soon as a firm date for the chapter draft is known and be prepared to contact many academics before an affirmative answer is given.

We use the following strategies to find reviewers:

• asking authors for the names of academics who can provide referrals to peer reviewers
• searching citation databases, such as Scopus and Web of Science
• browsing ‘Find an expert’ pages on university websites.

After locating suitable reviewers, send them an email with the details of the review task.  Include contextual information about the book, the title/subject of the chapter, approximate word count, any special features (for example, interactive content), and the due date for the review. Provide sufficient information so reviewers can assess their suitability and availability.  As you will need to contact many potential reviewers, save time by using an email template that can be easily adjusted for subsequent messages.

It is important to keep track of the reviewers that have been contacted and their responses.  In this way, you can avoid contacting reviewers who have responded negatively, and also keep a record of positive responses, including reviewers that flag their availability for a later date.  You can use this Peer Review Requests template to keep track of reviewers.

Providing instructions to reviewers

Before sending the manuscript to reviewers, remove identifying information about the authors from the manuscript and the file’s metadata.

When an open textbook includes multimedia, ensure reviewers can access the files. Sometimes all that is needed are links to the files from the manuscript. However, some files (for example, H5Ps) will have to be uploaded to a site that supports the file type, such as a Learning Management System (LMS).  In one of our projects, we set up an LMS site for review purposes. For another project with many audio, video, and H5P files, we provided reviewers with access to the front end of Pressbooks and a downloaded version of the textbook with links to the multimedia files. In this way, they could comment directly on the manuscript and easily access the multimedia files without searching for them on Pressbooks. Another possible method is to ask reviewers to comment directly on Pressbooks using the Hypothesis browser extension.

Provide reviewers with specific instructions so they know what criteria to use to assess the manuscript and how to provide feedback. At MUL, we use a template that has been expanded to include more context about the book and the authors’ worldview or theoretical approach, based on past reviewer and author feedback.

The criteria we ask reviewers to consider are:

• subject matter
• accuracy
• diverse perspectives
• longevity
• clarity
• structure
• consistency.

We also ask reviewers for their overall assessment (for example, accept, minor revisions, major revisions, reject) and to comment on their overall impressions and suggestions for improvement.

Managing feedback

Getting timely feedback from reviewers is important for the publication schedule to remain on track. Ensure you record when manuscripts are sent to reviewers and the projected return date in the publishing timeline.   When a reviewer is late, contact them for an estimated return date.

If a reviewer cannot complete the review, contact alternative reviewers. In this situation, the spreadsheet of alternative reviewers is useful, as you will have already identified several reviewers. Reviewer delays extend the publishing timeline. When a new reviewer is confirmed, recalculate the publishing timeline and confirm the dates with the authors and the copy editor.

When a review is returned, check the feedback to ensure it is anonymised before forwarding it to the authors.  Also check the content of the review to assess its quality and to estimate how much additional work is required of the authors, as this may affect the publishing timeline.  A third reviewer may be needed if the review is written cursorily or if the two reviews provide contradictory assessments. On rare occasions, reviewers provide disrespectful feedback. When this occurs, the project manager should alert the authors before sending them the review and be prepared to have a discussion with them about the best way forward.

Remuneration and record-keeping

MUL provides an honorarium to reviewers who have reviewed a substantial portion of a book. The honorarium is funded internally or covered by a grant program (for example, CAUL Collective).  The payment does not fully compensate the academic for their time and effort,  rather it is an acknowledgement and thank you for their service.

There are no set rates for reviewers, so a decision needs to be made internally about how much reviewers are paid. The amount is usually determined by the budget for peer review and the organisation’s payment system. To explain further, a textbook grant may provide $500 for peer review. If a book has 15 chapters and two reviewers for each chapter, the amount a reviewer can be paid for a chapter is about $16.  An ideal situation is to have two reviewers for the entire work that can be paid $250 each. The reality is that a substantial textbook may have up to 20 different reviewers as it is difficult to find academics who can review an entire textbook. It is impractical to pay reviewers for a single chapter as the cost of invoicing and payment far outweighs the actual remuneration.

To manage the review process smoothly, keep records of each step,  particularly when many reviewers are involved. As mentioned above, MUL keeps records of the reviewers initially contacted, the date of contact and their response. We also record who reviewed a chapter,  including the dates the manuscript was sent and returned. If payment was made, we record the details of the amount paid to the specific reviewer.

Indigenous peer review

Note: This section refers to the Aboriginal and Torres Strait Islander peoples of Australia, however, international indigenous content should also have a peer reviewer from the Nations or Peoples in question.

When a chapter has Indigenous content it should be reviewed by Indigenous peer reviewers. The project manager should be alert about the following issues when engaging Indigenous peer reviewers.

Availability

There may be a limited number of Indigenous academics in the subject area. If this is the case they may be overworked and already receiving many requests for peer review. Therefore, you will need to allow more time to engage a peer reviewer, may need to give them more time to undertake the review, and may need to allocate more funds from your budget.

Budget

Due to the scarcity of experts as well as the socio-historical context of exploitation and appropriation of Indigenous knowledge and peoples, Indigenous peer review may require more funding than non-Indigenous peer review. The rate depends on several factors, such as length, complexity, depth, and detail of review, and subject area. This may have a significant impact on your budget, so planning ahead is essential.

At MUL our approach has been to provide potential reviewers with an outline of the content needing review, the timeline and our budget. We ask if they are interested and available to review the work and have a conversation with them about their rates and our budget.

It is important to note that rates vary considerably. As an example, we have received chapter quotes that vary from $400 to $1600 (taking into account different manuscript lengths and levels of complexity).

Finding Indigenous peer reviewers

Finding Indigenous peer reviewers is similar to finding non-Indigenous peer reviewers. Look at academic profiles on university websites and try to identify universities that focus on the research area in question, eg. Indigenous health. You may also ask for referrals from Indigenous contacts within your university.  The author of the chapter may also have recommendations and requirements about who should be peer-reviewing their work.

Blind review or open review

The type of review most appropriate for Indigenous writing depends on the content and perspective of the author.  If the content is fairly general then blind peer review may be appropriate. However, if it involves a niche research area with few experts, an open review may be more practical. Similarly, if the content is about specific local practices then a peer reviewer from that Nation would be necessary and project managers would need to consult with the author to arrange this.

Ultimately, the best practice is to discuss with the author of the chapter how peer review should be approached.

Downloadable templates

• Peer Review Request Email [.docx, 16 KB]
• Peer Review Requests Tracker [.xlsx, 37 KB]
• Peer Review Criteria and Instructions [.docx, 484 KB]
• Peer Reviewer Payment Information [.xlsx, 21 KB]

References

Abbey, A. K. (2019). The OER starter kit. Iowa State University Digital Press. https://doi.org/10.31274/isudp

Aesoph, L. M. (2018). Self-publishing guide. BC Campus. https://opentextbc.ca/selfpublishguide

Kelly, J., Sadeghieh, T., & Adeli, K. (2014). Peer review in scientific publications: Benefits, critiques, & a survival guide. Journal of the International Federation of Clinical Chemistry and Laboratory Medicine, 25(3), 227-243. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4975196/

Open Education Network. (n.d.). Frequently asked questions. Open Textbook Library. https://open.umn.edu/opentextbooks/faq