1. The Importance of Reviewing Communities

It is unlikely that there are any subfields of sociology whose members don’t believe that they are getting fewer than their deserved pages in the AJS. As far as we can tell, it is a common and predictable form of cognitive asymmetry to see not only one’s own field as more meritorious than do others, but to imagine that there must be a nefarious reason to explain the lack of proportion between one’s self-estimates of the importance of one’s field and the appearance of articles in the AJS. This may be related to issues discussed below in (6), but we think it is the case that some of differences in these proportions can be due to something other than average quality, whatever that might mean.

The great explanatory factor is the presence of strong reviewing communities. The stronger a reviewing community in a subfield, the more the editorial board defers to the opinion of reviewers. What constitutes a strong reviewing community?

1. Senior figures regularly review papers, including papers by people they do not know, and they give them real reviews (see 4-7);
2. Those asked to review do not refuse when they suspect they will need to recommend a rejection;
3. The reviews are informative; they not only pronounce judgment, but they explain why the reviewer has come to this judgment;
4. They evaluate manuscripts more in terms of what the paper is trying to do and less in terms of whether the reviewer likes the goal, or the topic, or the framework, or the choice of terminology. Of course, these issues can be real ones, but in a strong reviewing community, there is either decent consensus about these, or a recognition that there is healthy disagreement about some such issues;
5. They do not indulge in the “questionable reviewing practices” (QRPs) described below in (7), especially that of using the heuristic of friend/enemy;
6. Reviews are turned in in a timely fashion and do not need to be edited for reasons of (6).

A subfield that develops such a reviewing community in effect chooses which of its articles deserve to be in a general journal like the AJS. This may seem strange, but it occurs all the time in practice: strong subfield reviews tell us where a paper stands in terms of its home subfield—is this a pretty standard contribution, even if well done, or is it one of the few articles from this subfield that others should know about?

2. Ethics and Practices

It is increasingly common for journals to require that reviewers assent to a pledge of ethical behavior. A reviewer is an expert advisor, and it is not, we feel, a promising way of beginning a relationship with expert advisors to assume that they are seduced by the devil and need to pledge their faith not to listen to such beckoning temptations. Such pledges may have seemed necessary because, with the globalization of many reviewing communities, the increased reliance on early career scholars to review papers, and some papers bringing reviews from reviewers in different disciplines, we have reviews coming from people who have not been socialized in the particular reviewing practices of the community in question. For this reason, we here lay out the sorts of reviews that have made the AJS an important outlet for sociological research—and, as these are the sorts of reviews that are weighty in the decisions of the editorial board, these are the sorts you want to write.

3. What Is Developmental Review?

The AJS tends to what is sometimes called “developmental review.” It is not that you have an obligation to fix others’ problems for them. But to the extent that reviewing can involve something that is like mentoring, it leads to a wonderful journal, and the spread of good reviewing practices. As an example, I am here sharing the reviews I received from the AJS for my first AJS paper (and my response memos): A Good Review Process.  This work came from my dissertation, and it took quite some time for a paper to be revised and accepted. It was enormously important to me not simply because I was an assistant professor coming up for tenure at Rutgers, but because the reviewers modeled amazing behavior for me, and I have tried to pass that on in my reviews of others, and in my past work on the editorial board. These many (!) reviewers did this not because I was tied to them, or on their “side,” and I do not know who they were.

Indeed, every paper that I submitted to the AJS was improved by the review process—even those that were rejected. In contrast, I have papers submitted to other journals that threatened to turn into Frankenstein monsters in the attempt to answer different, specific objections of only moderate relevance, coupled with perhaps well-intended, but often contradictory, demands for revisions. Often I was able, because of thoughtful editors, to navigate this in a way that did not turn the paper into the lowest common denominator of my three reviewers, but it wasn’t always the best use of any of our time. This was not development, but kibitzing. How does one tell the difference?

The key to developmental review is, first, a separation between evaluation and recommendation. First, one attempts to determine whether the paper has succeeded in doing what it claims that it is doing, and if not, why not. Some journals, especially in the open access world (Socius, PLOS One) attempt to have reviewers only evaluate and not make recommendations for improvement (but sociologists find this very difficult!). That is a fine review model, but developmental review involves a second portion as well: making recommendations for improvement. And by improvement, we usually mean improvement in attaining the original stated goal. Of course, sometimes this goal is impossible, and reviewers have to say so. But by keeping these two issues separate, reviewers are not tempted to recommend rejection because they disagree or dislike a claim that is being made. Let’s look at these two aspects of the review process separately.

4. Evaluation

Evaluation means an informed and informative evaluation. It is not uncommon for there to be parts of a paper that you do not understand—part of the literature it is in dialogue with may be foreign to you; you may not know the case, or the data; you may not be familiar with the methods. That is fine: you should proceed with the assumption that another reviewer will handle this part. Give your evaluation of the parts you do understand. And do not simply give the evaluation: explain why. If you tell us, “this is the worst paper I have ever read,” but say nothing else about it, you will be assumed to be using a QRP (see 7). What are the questions that editors want reviewers to answer? Well, they are…

1. Is there a mismatch between the claims and the analyses made?
2. Are the data appropriate to answer the question?
3. Are the analyses sound?
4. Are there other explanations that are probable (not remotely possible) for the patterns being investigated that have not been explored? Could these be explored? How?
5. Do the theoretical arguments make sense—not, do you agree, but are they internally consistent and coherent? Or do they assume what is in question?
6. Is the paper novel? Novelty on its own is not to be prized, but a sound paper that simply repeats what is widely accepted to be the case is not necessarily something that should end up in the AJS.
7. Does the paper claim preregistration? If so, does the preregistration document match what is reported here?  Does it add credibility to the claims by making specific hypotheses, or is it more or less just announcing a line of work?

5. Recommendations

Given your evaluative analysis, if the paper can be improved so that it is appropriate for the AJS, please give your best sketch of what would be required for acceptance. You do not have to do the work of the author(s), nor of the author(s)’ advisor. It is wonderful if you really hand them the answer on a plate. But what the editors need is a sense of what is missing that can plausibly be added.

It is in no way impermissible to address issues of how the author discusses certain issues, down to and including what words they use. Nor is it impermissible to consider potential effects of the publication of an incorrect finding. Different subfields have different ideas about the importance of such issues. But if these come as recommendations separate from the evaluative part (not necessarily organizationally separate—you need not use any particular organizational form!), your review will not be misread as part of a conflict within a subfield.

6. Tone

You do not need to be excessively enthusiastic. It has become common for reviewers to have a great difference in tone between the comments returned to authors and the confidential comments to the editors. If the former begin, “Thank you for giving the opportunity to read this amazing paper!! I absolutely loved it, and think it will be a big contribution when it is published!! I do have….”, and the latter take the form of “Kill the bearer of this message”, the editorial board is put in a very bad position.

On the other hand, even when a paper is written in an irritatingly dismissive manner, it is not appropriate to make ad hominem remarks, or to mock the author. If the author has said foolish things, you can point this out without calling the author a fool, or even using the word “foolish.” Just point to the problem. Overly biting reviews are generally assumed to stem from personal animosity.

7. Questionable Reviewing Practices (QRPs)

There are some easily recognized QRPs that you should avoid relying on. You may be right that the manuscript is unsalvageable, but you will do everyone a boon if you communicate that with calm sincerity and not a poison pen. The key is that these QRPs are all intensely obvious to editors. You may think you’ve been very smooth, but that’s not how it will read.

a) Bluffing
The reviewer can be tempted to hide behind the anonymity of the review process to claim to know that an article is fatally flawed, but, for whatever reason, to be disinclined to share the reason with the author and editor. One of the first papers I published got such a review from someone who claimed to have read the paper very carefully, more than once, and had been able to conclude that it was impossibly riddled with errors, but the reviewer could not take the time to mention a single one. But why would anyone spend so much time reading a terrible paper? Would a restaurant critic say, “I ordered a second helping of the trout, and this is a terrible restaurant, and it isn’t worth saying why”? Not surprisingly, this review was ignored by the editor, and the paper was published. If you took the time to read the paper, take a bit more time to explain the problems. If the paper was so irritating you couldn’t read it, let us know. But bluffing (“I could…”) is painfully obvious to editors.

b) Death by a Thousand Cuts
This is when a reviewer raises many, many small issues, none of which are in themselves serious, but seems to act as though the sheer number that have been listed justifies a rejection. This is often done in good faith by early-career reviewers, who tend to read closely but sometimes do not have a good basis of comparison (how this manuscript compares to others submitted to similar journals). Authors and editors are always grateful to reviewers who take the time to point to small problems. But unless they indicate that sort of negligence which approaches the criminal, it is important that an evaluation of a manuscript focuses on the most important aspects of the argument and evidence.

c) The Real Issue Is …
Sometimes reviewers will simply use their position of authority to assert that the great problem with the manuscript is that it is what it is, and not something else. This is sometimes put as “the real question is…” with the reviewer substituting his or her own preferred research questions. It may indeed be the case that the question asked by a paper is a bad question, because it is…

• Not of interests to sociologists (which sometimes happens when non-sociologists submit);
• Already answered;
• In principle, unanswerable;
• Constructed on the basis of doubtful assumptions.

Where this occurs, reviewers tend to concur. When only one reviewer insists that the real question is something else, this is a misuse both of the position of reviewer and of the word real.

d) Questions, questions, questions
Sometimes a reviewer will pose dozens of challenging questions asking about minor analytic choices made by the author, adding up to casting a cloud of suspicion that is not warranted. If after every statement, the review has a challenge, it becomes clear that the reviewer is less seeing a problem than wanting to bog down the author in such a way that s/he could not, under any circumstances, write a coherent paper. For example, a paper based on 85 (why 85? Why not 92?) in-depth interviews with adults over 25 (why 25? Why not include 18-24?) living in flood-prone areas in the South Atlantic (why only the South Atlantic?), etc.

e) “He’s Going for the Stairs!”
Here I draw upon a wonderful analysis of the bureaucratic personality by David Thorne. He retells the story of the experimental instillation of norms among laboratory monkeys, placed together in an enclosure, with one set of stairs leading to a loft. Should any monkey tread on the stairs, all the monkeys were given a long, hard spray of ice-cold water. The fast learners made sure to warn away their slower conspecifics via cuffs, and when new monkeys, who had not yet experienced the cause-effect nexus, were introduced, should they come near the stairs, the older monkeys would attack them viciously. Eventually, there was complete turnover, the spray had been disconnected, and yet each generation of newcomers was initiated into obedience of the limits through simian savagery. “Lookout! He’s going for the stairs!” was Thorne’s take on a classic pathology of the bureaucratic personality.

We sometimes see reviews that seem animated by a profound resentment that the author is attempting to do something that (it seems) the reviewer was unfairly prevented from doing or punished for doing. “Who are you, to get away with the thing that I had to renounce?” is the implicit tone. Of course, many times authors are indeed punching above their weight, and it is only a matter of time before they get knocked down. But reviewers should not pass on the abuse they may have suffered as early-career scholars, and attempt to hold others back simply because they were held back. We do not want incorrect, wild, silly, or incompetent papers, but we do want innovative ones.

f) Friends and Enemies
Some reviewers think that they are doing the right thing when they keep out bad work, and bad work is work that makes a false claim, and false claims are those that contradict what the reviewers in question believe. The logic sounds reasonable … until you realize that this is actually replacing the reviewing process with the heuristic “help your friend, hinder your enemy.” Reviewers should be able to recognize when there is a strong (if not perfect) paper that they disagree with, and when there is a weak one they happen to agree with. You may have been asked to read a paper precisely because we know that it is critiquing your own approach. We want to know not whether you agree, but whether there are strengths and weaknesses in the author’s argument. You can evaluate the paper fairly without having to change your own position!

g) Bargaining
Sometimes reviewers will—perhaps without realizing what a misuse of their position this is—in effect attempt to bargain with authors. The reviewer believes that A is X. The author is arguing that A is Y, and has marshaled evidence to support this claim. The reviewer complements the author for a job well done, but sadly notes that the paper is “one-sided” or “extreme,” and that it would be more “helpful” were the author to say that the evidence suggests that A is both X and Y, or somewhere between the two—not because the evidence does suggest this, but because the reviewer can implicitly make this concession a condition of acceptance. The review process is not supposed to degenerate into blackmail.

8. Why Should You Review?

Unfortunately, some people believe that the whole reason to agree to review is to blackmail others. If you cannot use the review as a position from which to exert power, why should you do it? That’s a good question! The answer is similar to that to other questions like, “why shouldn’t you just chuck your bottles on the neighbor’s lawn?”, “Why shouldn’t you dump used engine oil in the storm drain?”, “Why not take that little kid’s funny Pokemon hat to decorate your office?”, “Why shouldn’t you divide your standard errors by 10 and claim that your results are significant?”, “Why should you pay taxes?” If you haven’t yet been fortunate enough to profit from the contributions made by reviewers who are doing their part to contribute to their discipline, in the same way they might contribute to their neighborhood, or their family, or a friendship group, this may strike you as odd. But if you continue to submit to the AJS, it should be only a matter of time before you come across those people. Those of us who have the burden of gratitude to other reviewers will feel obliged to pass it forward.

9. Some Technicalities

• Please maintain anonymity. Even if you don’t mind being identified, other reviewers may, and knowing your identity may mislead the author(s) into having greater convictions about the secret identities of others than if you had not identified yourself.
• Please do not decline simply because you know who the author is, or because in the past you have been critical of the author (or vice versa). While you should not review someone if you have a conflict of interest such that you cannot trust yourself to give the same evaluation you would give if the author were a stranger, merely disagreeing strongly with someone does not constitute a conflict of interest. If you have been asked to review this paper before, you are free to re-use an earlier review if the manuscript has not changed: while authors are frustrated and depressed to find they have not managed to escape a severe critic by fleeing to another journal, this should not prevent you from giving appropriate feedback that the author needs to hear. We appreciate your spelling out any connections between you and the author in comments to the editor, so that we can best position your review.
• Please do not put in explicit or implicit recommendations in your response to authors. The “line” at which different reviewers decide on a revise-and-resubmit decision varies greatly, and if we have two reviewers who say R&R, but actually are doubtful that the paper can be revised to be an AJS contribution, and another that calls for rejection, the author who sees this explicitly noted (especially only by the more positive reviewers) may be misled into thinking there was more enthusiasm for the paper than there actually was.
• Related to this: reviewers do not help an author by “encouraging” them via an R&R if they do not actually think that the authors can pull off a successful revision. The authors will waste a great deal of time that could be better spent elsewhere. Without being hurtful, please be honest. Those authors who really can’t take criticism will need to exit the research world pretty soon anyway.
• Please do not change your mind when you see that a paper is worse than you thought, and, after accepting a review, now decline it. The fact that you cannot be an advocate for the paper does not mean that you cannot be of immense help to the author. Your honest evaluation and recommendations can help that author get past one bad paper. If you decline, and other reviewers are not as centrally located in the author’s field as you, the review process is likely to be less useful!

10. Our Appreciation…

cannot be overstated. The AJS is managed by Susan Allan, and the editorial process is curated and directed by the editorial board, but the AJS is controlled by the field, by the outstanding reviewers across all types of institutions, all types of subfields, all types of methods, interests, and so on. Keeping this community in dialogue is essential for preserving the AJS as a place where rigorous and transformative work can continue to appear. If you join in this community, we will be in your debt.