last updated:
06/28/2005

Reviewing Manuscripts: Advice to New Relationships Scholars

by
Sandra Metts
Illinois State University

The first time I was asked to review a manuscript I was still a graduate student. Unsure of what to do, I asked my advisor. He suggested that we do independent reviews and then compare. I spent the weekend agonizing and writing. (He spent an hour and a half jotting a few notes.) When we compared our reviews, we were similar on many points, but differed in the final recommendation. I said, “Revise and Resubmit.” He said, “Reject.” I was distressed. Inspired by my concern for the author’s loss of face, I said, “That’s not very kind.” My advisor’s response was telling. “A review is not about kindness; it is about contribution to cumulative knowledge.”

Certainly my advisor was not suggesting that a review should be unkind, or mean-spirited. He was merely reminding me to separate my concern for the author from my concern for the field. He wanted me to evaluate the manuscript as a product being recommended to a society of consumers who trust the product has been inspected by experts and meets criteria for the use to which it will be put. My admittedly wholesale adoption of this metaphor will be reflected in the comments that follow. By way of framing my comments, I need to make explicit that I have in mind a reviewer’s role when evaluating manuscripts submitted to relationship journals such as Journal of Social and Personal Relationships and Personal Relationships or to journals with similar mission statements within particular disciplines.

Knowledge? If No, exit; If Yes, continue

The first question to answer is whether the manuscript holds publication potential (revise and resubmit) or not (reject). Occasionally a manuscript is simply bad--no nice way to say it. The design is poorly conceived and the data are virtually meaningless. The gap between conceptualization and measurement is just too great to salvage. Occasionally, a researcher may return to a data set collected previously with much keener insights because important new findings have recently appeared. This often works well, but in some cases, the data are simply no longer trustworthy. For example, a study of sexual practices among college students using data collected prior to say 1985—when AIDS awareness became more common—would be suspect. Sometimes researchers conduct a study unaware that an almost identical study has already been done. This is perhaps endemic to interdisciplinary work. Replications are not a bad thing, but they cannot be done by happenstance; a good replication requires as much justification as any other piece of research. So generally, for the fatally flawed and the redundant study, a recommendation against publication is fairly easy to determine. Other than pointing out the problem(s), the reviewer does not need to struggle with the review; the decision is generally straightforward.

Fortunately for our field, these manuscripts are rare. More common, and more problematic, is the manuscript that does nothing wrong in a technical sense, but leaves the reviewer with the vague sense that something is missing. Such a study might, for example, extend a previous finding on some attitudinal or interactional variable into the domain of sex differences with no stronger rationale than, “No one has compared men and women so I will.” The “it hasn’t been done yet” argument is not a very compelling justification. These manuscripts, to various degrees, force the reviewer to consider criteria other than technical competency, validity, and redundancy.

The most important criterion guiding a reviewer in such cases is the question of whether a manuscript contributes (or holds the potential to contribute) knowledge to concerns of interest in the field. If a reviewer finishes reading a manuscript and finds him or herself asking the “so what” question, then the chances are good that this manuscript has not established its legitimacy as knowledge. For whatever reason, the expectation of knowledge as understanding, insight, or explanation has not been met. The product of such a manuscript might more aptly be described as information because the data, details, descriptions, or facts are undigested; they might be organized, but they are not assembled into a conceptual claim. A reviewer will sometimes suggest how the information might be assembled, but technically, that is not the reviewer's job. In fact, it might suggest that the research was not guided by a controlling theory or coherent set of issues. To overlay a structure after the fact can stretch or even misrepresent the data. If the research is exploratory and/or interpretative, as some types of qualitative approaches are, the author(s) still must meet the expectation that some conclusion or claim about relationships will be offered, no matter how tentative. Thus, in most cases, a reviewer who cannot see how the information in a manuscript contributes to cumulative knowledge, unless substantially reformulated, would be advised to recommend against publication.

Although it is easy to say that a manuscript needs to provide something more to its consumers than information, in practice this becomes a more difficult decision when the data are inherently interesting or exceptionally rich. For example, it is not easy to recommend against publication of a study when researchers have collected longitudinal data, data from a random sample of adults in the community, or data from special populations such as blended families or interethnic married couples. Similarly, when the data could potentially inform topics with important social implications (e.g., adults or children in abusive relationships or substance addicted relationships), it is tempting to want this information to be available to scholars and practitioners. If suggestions can be made to ground the data in previous research on problematic relationships or relevant theory, the data may be more useful to consumers. Alternatively, the reviewer may decide to reject the manuscript but suggest an alternative outlet specializing in "data report" formats. However, if the research agenda is unfocused or theoretical/conceptual issues are misrepresented, then even inherently interesting data cannot be published because it will mislead rather than enlighten a consumer.

If Yes, What to Look For

Assuming that a manuscript demonstrates potential contribution as knowledge, the reviewer then must decide how best to improve the product. In function, if not in form, a review is a conversation, subject to the same constraints and considerations. When talking to a friend, we assume that he or she is attempting to be informative and cooperative unless there is evidence to the contrary. Thus, when confused, we ask for clarification rather than assuming stubbornness or stupidity (although I suppose that might depend on the friend). For example, when confused by something a friend has said, we might ask for a rephrasing (What did you just say?), for clarification of intent (Did you mean to say that ...?), for a consistency check (Didn't you say yesterday that ...?), or for the logic behind a statement (Why do you believe that?). When not convinced that a friend is right, we might ask about the basis of his or her claim (Did you read that somewhere or just make it up?), offer counter examples (But what about ...?), or identify problems in the logic (If that's the case then wouldn't it also be true that ...?). Although these are not necessarily the questions a reviewer asks of an author, they do reflect the spirit of the best reviews.

More specifically, when recommending a revise and resubmit, it is typical that the reviewer be specific, detailed, and cooperative in his or her comments. Space does not permit a listing of all potential problems in a manuscript. However, it is useful to consider briefly the types of concerns that reviewers frequently express. These tend to be one or more of the following: I need clarification; I need explanation; I need justification. Put another way, the reviewer has found a problem in the what, the why, or the overall coherence/logic of the premises, decisions, analyses, or conclusions. These concerns can be illustrated in the four sections of the traditional research paper (but are relevant to all designs):

Rationale: The purpose of the rationale is to build an argument. In the classic form, especially for theory-driven research, the implicit argument structure is, "Given X, then Y should follow." X represents the premises derived from previous findings; Y represents the logical conclusions that have not been tested but should be. In quantitative research, the hypotheses represent a concrete, testable (operationalized) statement of the Ys. In qualitative or exploratory research, the argument might be "Given X, then Y is necessary." In this case, X is a condition or circumstance, or lack of knowledge; Y is the type of data necessary to understand the condition or circumstance or fill in gaps in our knowledge. Regardless of the overall design, some rationale for additional research and for this particular study must be provided.

The rationale in any manuscript under review may need attention because the "what" is not clear or is incomplete—what did you say? what did you mean? For example, the presentation of other research is not clear or the conclusions drawn from it are vague or not made explicit, the purpose statement for the proposed study is confusing, the hypotheses or research questions contain terms not clear, or the causal directions are confusing. Sometimes the "why" is unclear or not developed sufficiently—why do you think that? why do you claim that? For example, although concepts may be clearly defined, the reason they were chosen as appropriate for this investigation is not fully explained, or the reasons for claiming other research to be inadequate are not well grounded, or the claim that more research is needed is not supported. Finally, the hardest concern to address is that the justification is not compelling or the rationale is incoherent. When asking an author to make an argument more compelling, the reviewer is essentially saying, "I understand what you have said, and I understand why you said it, but I am not fully convinced that I buy it—I see other possibilities that would work just as well, or I don't see how knowing X will help us understand Y.” Often this level of concern arises because the author, like all of us, operates on taken-for-granted assumptions and these are the missing premises that would make the argument in the rationale more compelling or coherent.

Method: The purpose of the Method section is to make clear to readers who the sample was, how they were selected, what happened to them, how a concept or variable was operationalized and measured, how and why manipulations were done, and so forth. The "what" question here is usually about missing information (description of sample is not complete, sample items for scales are not provided, etc.). The "why" questions are sometimes more critical. Why a particular scale was used may need elaboration; good reliability does not necessarily mean it is a valid measure of the construct under investigation. Similarly, the "why" question might be addressed to the sample. If a college sample was used the reader should know why. I realize that sounds like the "well, duuhh" request, but just because "everyone does it," or because it is "convenient" doesn't mean that the why question has been answered. Most researchers use college student samples because they are, in fact, the best population to draw samples from given the purpose of the research (e.g., initial attraction, dating norms, premarital breakups, etc.), but studies of phenomena such as relationship maintenance, romance in the workplace, extra dyadic affairs, or love and sexual satisfaction may need to more fully explain why college students are the best population from which to draw a sample (or justify the limitations if it is not the ideal sample). Assuming that the “what” and the “why” are clear, the information may still be insufficient to offer a compelling justification. A reviewer might understand, for example, why a scale was reduced from a continuous measure to a categorical measure but may also see a way to test the hypotheses without doing so.

Results: The Results section presents the statistical evidence that supports (or fails to support) the hypotheses, that answers the research questions, or that, for qualitative work, exemplifies (provides suitable "evidence" for) a "finding." The "what" questions might be prompted by confusion in how an analysis was done or interpreted. Perhaps effect sizes were not included. Perhaps causal wording slipped in for correlational data. The "why" questions might be prompted by insufficient reasons why a particular test was done, why male responses were compared to female responses if not anticipated by the rationale, etc. The justification question might be prompted by concerns that statistical findings are not clearly responsive to the hypotheses. Decisions to offer information about trends, for example when significance levels are at .10, might be appropriate, but the reviewer may need to be convinced of the utility of reporting trends if conventional significance levels fail.

Discussion: The Discussion section is the point at which the knowledge generated by the study is made explicit. Findings are summarized, their meaning and importance are developed and situated within the area of concern, limitations are noted, and directions for additional research motivated by the study are suggested. A reviewer will often see his or her questions from previous sections answered here. But other questions might arise. The "what" questions might be focused on the clarity of the claims (What exactly are you concluding? Do you really mean to imply that . . .?). The "why" questions might deal with bases for interpretation (Why would you propose this larger claim about relationship behavior based on your findings? The link is not clear.). The justification questions may focus on the "nonexclusivity" problem where other equally plausible interpretations are possible, or on inconsistencies between the argument in the rationale and the interpretation of the findings. Questions of justification are common when directions for future research do not seem to be linked to the current study or when these directions for future research are formulaic rather than systematically derived. For example, the call for a larger or more diverse sample should be made because something in the current study warrants it, not because it’s so typically done.
It's Revised: Is It Knowledge Yet? If Yes, Exit; If No, How Close?

When the revised manuscript and the letter from the author(s) eventually find their way back to the reviewers, the decision process begins again. If the author addressed the reviewers' concerns and the knowledge is ready for consumers, then the manuscript can be accepted (or accepted with minor revisions). If more work is necessary because the changes are substantial and new issues are raised, then the review process recycles. If the letter from the author(s) indicates that a particular change cannot or will not be made, then the reviewer's letter to the editor must indicate whether or not this response is acceptable. The editor will consider this in light of his or her reading of the manuscript and other reviewers' comments.

More General Suggestions (Do's & Don'ts):

1. There are few absolutes in reviewing, but one injunction holds true across contexts: A reviewer should not assume the responsibility of reviewing a manuscript that is beyond the scope of his or her expertise. A good habit is to skim a manuscript as soon as possible after it is received to determine one's eligibility as a reviewer. If the topic and/or the method are not within one’s comfort level, then it should be returned in a timely fashion so the editor or associate editor can forward it to a more appropriate reviewer. It is discourteous to the editor to return the manuscript within days of the deadline; it is unprofessional to provide inappropriate or uninformed feedback.

2. Many journals, especially relationship journals, are interdisciplinary. Although we believe that our own discipline has much to offer scholars in other areas, we cannot take offense if work in our area has been overlooked. This is a perfect opportunity to familiarize others with work that might inform their efforts. Assume ignorance, not arrogance, on the part of the author(s). By the same token, if a concept, construct, or line of research is presented in ways inconsistent with treatments in our discipline, ask for clarification. Perhaps the author(s) made a mistake; more likely, they are working from the assumptions of a different discipline. A good reviewer says, "In psychology, Q is used to mean xxxx. Can you explain how your notion of Q is similar to or different from this conceptualization?"

3. Be specific by pointing to places in text where confusion arises. To say that the rationale "needs work" or the discussion "needs focus" states the problem but offers no direction to guide revision. It takes time to provide detailed feedback, but that is the obligation that we accept when we agree to review a manuscript.

4. Reviewers are not obligated to help authors become better writers. However, feedback on writing style can benefit the author who undertakes a revision. If there is no topic sentence or it is too deeply embedded within a paragraph to effectively organize it, say that. If preview statements, internal summaries, shorter sentences would make the manuscript more reader-friendly, make those suggestions. The goal is to introduce the best possible product. We get no individual credit (and lots more manuscripts to review) when we make helpful suggestions, but the field benefits. Ultimately, we also benefit as authors when we receive helpful reviews ourselves. We have to trust that what goes around will come around (although this assumption may need to be tested).

5. When appropriate, and a reviewer has the expertise, it is imperative to help authors with statistical analyses and interpretation. If there is a more parsimonious model or a more elegant multivariate test, then taking time to explain this is a useful contribution to the manuscript under review as well as for further research by the author(s).

6. Try to be consistent in comments to the author(s) and comments to the editor or associate editor. Reviewers' comments made separately to the editor or associate editor are not made available to the author(s). As a result, a recommendation by the editor or associate editor based on reviewers' feedback sheets may appear to the authors to be inconsistent with the reviewer comments. A clear statement at the beginning or end of the review specifying the reviewer's recommendation can be helpful in avoiding potential confusion for authors.

7. Finally, two suggestions for new scholars who are still graduate students. First, ask an advisor or mentor to provide feedback on your first few professional reviews. Second, ask that your graduate seminars include manuscript reviews as part of the course assignment. Manuscripts might be those authored by scholars within the department or related departments, or research papers produced by students in the class. Whatever the source, experience doing the reviews, particularly when they are compared to others’ reviews is a good way to develop competence as a manuscript reviewer.