July 8

Cite Check

Citations figure prominently in academic promotion and peer review. Theoretically, scholarly references serve a dual purpose: They indicate an author’s familiarity with established literature and assign credit to previous work, while from the other direction many would argue they signal a paper’s relevance and standing within a discipline.

Looking for a job?

See all 208 new postings

Browse all job listings:

Faculty: 3,375
Administrative: 1,315
Executive: 170
FEATURED EMPLOYERS

That’s, of course, in theory. The reality may surprise many academics who might not stop to think about the system they rely on for the production of knowledge, or who studiously ignore those little superscript numbers that indicate (again, in theory), Read the referenced paper to learn why the preceding assertion is correct.

What if it isn’t correct? What if the authors didn’t even read half of the papers they cited?

Like any self-enclosed, loosely policed network, citations are far from perfect. It’s well documented, for example, that researchers tend to cite papers that support their conclusions and downplay or ignore work that calls them into question. Scholars also have ambitions and reputations, so it’s not surprising to hear that they might weave in a few citations to articles written by colleagues they’re trying to impress — or fail to cite work by competitors. Maybe they overlook research written in other languages, or aren’t familiar with relevant work in a related but different field, or spelled an author’s name wrong, or listed the wrong journal.

All of these shortcomings are reviewed and discussed in an article published this year¹ in the management science journal Interfaces along with the critical responses to it.²

As it turns out, scholars have already done some work quantifying problem citations, divided into two categories, “incorrect references” and “quotation errors.” The authors of the paper, J. Scott Armstrong of the University of Pennsylvania’s Wharton School and Malcolm Wright of the Ehrenberg-Bass Institute at the University of South Australia, Adelaide, write of the former type, “This problem has been extensively studied in the health literature ... 31 percent of the references in public health journals contained errors, and three percent of these were so severe that the referenced material could not be located.”

More serious than such botched references are articles that incorrectly quote a cited paper or, as the authors put it, “misreport findings.” For example, in the same study of health literature³, they write, “authors’ descriptions of previous studies in public health journals differed from the original copy in 30 percent of references; half of these descriptions were unrelated to the quoting authors’ contentions.”

It wasn’t until Wright noticed that a paper Armstrong co-authored in 1977 had been inaccurately cited that they realized the extent of the problem, Armstrong said in an interview. It was Wright who suggested investigating the problem further in a more systematic way. So they focused on that specific article, which outlines a precise method for estimating the extent to which non-responses to mail surveys bias the results. Since the article has been heavily cited and the method it describes can be identified, they were able to trace how well articles that reference it represent the original material.

Using a combination of Google Scholar searches and the ISI Citation Index, Armstrong and Wright found results that are disconcerting even when acknowledging that Google doesn’t crawl every academic paper: Among academic studies using mail surveys, only 6 percent mentioned the non-response problem at all, and of those, 2.1 percent (339 articles) cited the 1977 paper. The authors found 36 variations of the paper’s citation among those that referenced it, with an “overall error rate” of 7.7 percent.

By analyzing a sample of 50 papers (out of 1,184) that cite the 1977 article (including the 30 most frequently cited of the bunch), the authors also found significant inaccuracies. By their standards, those papers didn’t fare especially well either: “In short, although there were over 100 authors and more than 100 reviewers, all the papers failed to adhere to the [1977 paper’s] procedures for estimating nonresponse bias. Only 12 percent of the papers mentioned extrapolation, which is the key element of [the paper’s] method for correcting nonresponse bias.”

The paper concludes: “Given the understandability of the recommendations and the fact that no one contacted Armstrong or [his 1977 co-author] for clarification, one might question whether the citing authors read the ... paper. To present their studies in a more favorable light, some authors may have wanted to dispel concerns about nonresponse bias; thus, they cited [it] for support for their own procedures. Interestingly, one of our colleagues said that it is common knowledge that authors add references that they have not read in order to gain favor with reviewers. One wonders: If it is possible to write a paper without reading the references, why should the authors expect readers to read the references?”

If the problem is as widespread as Armstrong and Wright suggest — and Armstrong said he believes the findings generalize to other scientific fields — then a more systemic fix might be warranted. They provide several common-sense remedies intended to address what the peer review system currently, it appears, is unable to counteract ("My experience is most peer reviewers don’t seem to be competent to do the job,” Armstrong says). “When an author uses prior research that is relevant to a finding, that author should make an attempt to contact the original authors to ensure that the citation is properly used,” they write.

“As I point out in the paper, I’ve been doing this for years, and it doesn’t really require that much work,” Armstrong said. “Generally, I found it to be easy to do. I do it as an author; I hardly get anybody asking me — they just go ahead and quote me incorrectly.”

The paper also argues that researchers should have to verify to journal editors that they’ve tried to contact the relevant authors, and that they’ve read the papers they cited. Furthermore, they suggest, there could be a solution waiting on the Web — one that sounds a lot like a cross between the Wikipedia model and Amazon.com reviews: “Journals should open Web sites (free to nonsubscribers) that allow people to post key papers that have been overlooked, along with a brief explanation of how the findings relate to the published study.”

Already, Interfaces and a journal Armstrong co-founded, the International Journal of Forecasting, are planning to introduce those suggestions into their editing processes. Rob J. Hyndman, the forecasting journal’s editor-in-chief, said in an e-mail that within two weeks, the Web submission system will include a check box with this text: “Confirm that the list of references has been checked carefully for accuracy and that each of the references has been read by at least one of the authors.”

And the 2008 paper? For the record, Armstrong said he and Wright followed their own advice in publishing their research: “Oh yeah, we talked about that. We had to make sure that each one of us had read every one of the papers.”

¹Armstrong, J. S., M. Wright. 2008. The Ombudsman: Verification of Citations: Fawlty Towers of Knowledge? Interfaces 38(2) 125-139.

²This paper is available to download, in PDF format, by clicking here.

³See previous paragraph.

— Andy Guess

Want it on paper? Print this page.
Know someone who’d be interested? Forward this story.
Want to stay informed? Sign up for free daily news e-mail.

Comments

Not a new phenomenon, and as the paper cited below (which I have read :-) ) shows, it can lead to some interesting problems as bad cites are, well, cited:

Treadway, M. & McCloskey, M. (1987). Cite unseen: Distortions of the Allport and Postmann rumor study in the eyewittness literature. Law and Human Behavior, 11(1), 19-25.

D. Johnson, at 7:35 am EDT on July 8, 2008

Be serious. Cite correctly, but do these authors really think anyone has time to contact all the authors they cite?

TBD, at 7:35 am EDT on July 8, 2008

How is this meant to work with often-cited authors such as Bruno Latour or Richard Dawkins? These guys would be dealing with numerous requests every day!

Oh, and I’ll be sure to contact Jacques Derrida next time I cite him!

Rich, at 8:10 am EDT on July 8, 2008

Late to the Party

Every PhD student who had a component research methods instructor knows (1) how to calculate correctly non-response bias and (2) Armstrong and Overton’s (1977, Journal of Marketing Research). article is always incorrectly cited when discussing non-response bias.

Armstrong and Overton is the most widely cited paper in marketing specifically and business in general. For Scott to take an interest in incorrectly cited papers reminds me of the scene in Casablanca when Captain Renault discovers there is gambling going on.

Where was Scott on this issue in 1982, five years after his paper appeared and scholars were starting to cite it incorrectly?

bevo, at 8:15 am EDT on July 8, 2008

citations and referees

The fault does not always lie with referees. I read a translation for a U Press and chased down some citations that didn’t seem right (and which in fact were not), corrected them and gave the appropriate listings, but they were not corrected in the publication. It’s not the first time it’s happened either. It’s very frustrating when asked to review an article or book and this happens.LM

LM, at 8:30 am EDT on July 8, 2008

Shocking? Or just amusing?

” .. Where was ... on this issue in 1982, five years after his paper appeared ..”

Uh .. in the citation room with this fella?

http://www.colorado.edu/news/reports/churchill/

Sorry. Just could not let this go by, without that citation.

Shocking, just shocking.

Bart, at 8:50 am EDT on July 8, 2008

faulty citations

No kidding!?! I am so shocked!?! (not)And those same people whine about diploma mills.

Dr. Betty, at 9:40 am EDT on July 8, 2008

Footnote to the above

For a related account of professorial dereliction, see

http://www.timeshighereducation.c...code=26&storycode=402598&c=2

a British report on self-plariargism — the resubmitting of one’s own articles in repackaged form in order to puff one’s CV with “new” articles. It’s a Wild West out there in the classroom — all kinds of stuff going on on both sides of the lectern.

Abbott Katz, at 9:40 am EDT on July 8, 2008

Questions

Regarding citations: Has anyone tested how published articles impact a faculty members promotion and tenure or a consultant’s marketability?

It is rumored that review committees check the citation index to measure the relevance of an article. Thus, by citing a key article, the article, itself, gets cited in a search, raising its stature and rankings. The entire citation index enterprises are based on this strategy, including which journals are included in the index and thus the need to publish in the “right” journals.

The need to publish forces one to use minimal information in order to increase the number of articles, thus the need to use supporting citations rather than taking longer to do more detailed research and publishing less.

Citing only supporting articles or select literature, again, depends on similar factors and is, in part, why English is the default language of choice with some areas being the exception.

In other words, a publication in a journal has intrinsic value just by its very existence.

Yes, citations and accuracy in writing, including citations are important. The question that is not answered is whether, in the internet age, are there not more efficient ways to get critical new ideas into the intellectual marketplace, particularly in ways which become self correcting over time.

Is the idea of a journal, in its current embodiment, in a web 2.0+ era, relevant other than for its meta value?

cynic, at 10:20 am EDT on July 8, 2008

Citations

Librarians’ job security grows with citation inaccuracy, for their expertise, tools, and experience-informed hunches can yield results when people need to catch and correct errors, enhance impoverished citations, or identify and locate misquoted, badly cited, and/or elusive sources. At the same time, a librarian’s keen frustration is the inaccurate citation that impedes direct retrieval or interlibrary loan. Usually the citation itself is the problem, not an error by the person seeking help.

James Pakala, Library Director at Covenant Theological Seminary, at 10:40 am EDT on July 8, 2008

Law reviews v. peer reviewed

Strangely enough, people condemn law reviews for not being peer-reviewed, yet law review ARE cite-checked (usually meticulously). When I was in law school I had to verify all 400 cites in an article, and if I had a doubt I had to ask the author for clarification (or source materials). The articles I wrote post-law school were subject to the same level of scrutiny.

The “contacting the author” requirement is a bit difficult. First of all, from my perspective, anything that is published should speak for itself. Allowing authors to “selectively clarify” their research allows the original authors to 1) decide who is “worthy” of using their research; and 2) decide whether they like the ultimate political result of such research. If authors are concerned about being misquoted, they should just be careful about how they write, and this goes for papers that they are co-authors on.

Larry, at 10:45 am EDT on July 8, 2008

Excellent work

I’m going to cite this article as evidence to show how careful we academics are about citations.

What?

Assistant Professor of Physics, at 11:00 am EDT on July 8, 2008

As you find out when one of the peer reviewers of your manuscript is someone you have cited, the author of the cited work rarely agrees with what you say about it, especially if you are expressing a divergent view in your own paper or were critical of that cited work. Sometimes this problem is so bad that you must quote instead of paraphrasing to work around this objection. The peer reviewer always blames you for misquoting or misusing their work. Does it mean you really have? Not necessarily. What someone intends to communicate and what is received are two different messages. I think assuming that people who cite are not reading may be wrong. We all filter what we read through our own capacity for understanding. I don’t see any way around this, since I don’t believe that authors should be able to veto whatever future use is made of their work. I think the audience should be going to and reading the cited references, a remedy that is always available to them if the correct source information is available.

Perry, at 11:35 am EDT on July 8, 2008

Cite Check

This study confirms my own observation over 35 years as an academic librarian that citation accuracy follows a Bell curve. Freshmen are terrible, by the time they’re seniors many are pretty good, grad students generally sweat bullets over their citations and when they get their “terminal degree” that accuracy starts to die. They assume they remember the titles of journals and articles and the names of colleagues, but are increasingly incorrect over time. I’m reminded of the old joke that Full Professors are so designated because they are full of it.

Jim Dwyer, Bibliographic Services Librarian at CSU, Chico, at 12:05 pm EDT on July 8, 2008

Larry makes an excellent point: the importance of being careful what one writes as both an author and co-author in order to avoid being misquoted (or misinterpreted) Excellent scholarly writing is not an easy task.

Carla, at 12:20 pm EDT on July 8, 2008

law reviews vs. peer review

I wish that I could say that I am surprised by this. After the Bellesiles scandal—where historians at first closed ranks around one of their own engaged in not just sloppiness, but intentional, politically-motivated fraud—I am not surprised. See “Why Footnotes Matter: Checking Arming America’s Claims, Plagiary 1(11):1-31 [2006] http://www.plagiary.org/why-footnotes-matter.pdf for a few examples of really gross fraud.

I’ve seen the criticism that law reviews are too focused on citation checking, and sometimes miss the forest in their desire to verify that every tree is in place. But if history journals even did a random check of 10% of the citations in the papers that they published—and did so before the peer review process—it would have caught the Bellesiles fraud. I’ve been through the painstaking process of law review citation check a number of times now, and while it is sometimes exasperating to find what are sometimes the legal equivalent of showing an authority that the sky, on most days, is blue, the alternative is this faith-based system upon which history journal peer review relies.

Maybe Bellesiles and Churchill are very atypical; maybe the academic profession is 99.9% squeaky clean. But seeing articles like this makes me inclined to think otherwise—especially because of how much effort it took to get historians to look at a gross fraud like Bellesiles, who seems almost to have been trying to get caught, because the fraud was obvious—and almost got away with it.

Clayton E. Cramer, at 12:25 pm EDT on July 8, 2008

Editing someone else’s work

I am working on a site report and have discovered that almost every reference and direct quote needed to be corrected. It has taken more time than correcting and supplementing the text itself. The ILL folks love my requests for 18th century texts and obscure site reports.

Sara Orel, Professor of Art History at Truman State University, at 1:00 pm EDT on July 8, 2008

So this is a big revelation?

Bob A., Huh?, at 1:35 pm EDT on July 8, 2008

Peer-reviewed vs. final published article

Anyone who has spent any time copyediting scholarly manuscripts, as I did for several years early in my career, would not be surprised at all by this information about the high rate of incorrect citation. At first, as a beginning editor, I was appalled to find so many mistakes in the footnotes of senior scholars. (I particularly remember an expert on Martin Luther whose chapter in an edited volume contained multiple errors in the citations to the authoritative edition of Luther’s works, which I systematically checked in the Princeton library.) Who knows how many scholars have been spared from embarrassment by their copyeditors working quietly behind the scenes to repair their flawed writings? It is just because I know many scholars to be so sloppy with their apparatus that I worry about the trend to “open access” that will place uncopyedited manuscripts in institutional repositories with all their warts showing. Harvard faculty should not be so eager to display their flawed products to the world. (That Luther scholar I mentioned was on the Harvard faculty, by the way.)

Sandy Thatcher, Director, Penn State University Press, at 1:55 pm EDT on July 8, 2008

Cynic II

” .. Has anyone tested how published articles impact a faculty members promotion and tenure or a consultant’s marketability?”

Excuse me — is the Pope Catholic?

Want to increase your cite count? Just question a popular belief (e.g., multiple intelligences, “the academy is not biased") — and watch your work get cited in response.

Next: speaking invitations. Promotion. Tenure. Travel. Conferences based on your critique. NSF/NIH funding. Get named a president’s professor — no silly departmental meetings.

BTW: Lar, ol’ bud — welcome back.

BTW: Prof. Cramer — your work, and that of others, have put fear into the hearts of large numbers of potential academic frauds. That their obvious biases, based on academic fraud, could be discovered and lead to termination — gives them pause.

That such frauds could not be detected by “peer reviewers” and administrators brings into serious question, the competence, professionalism, and judgment of entire groups of alleged academics and administrators.

Per previous: contrary to many believers — academic freedom does not permit academic fraud. Fraud is fraud. Bush/Cheney/Iraq are not mitigating factors.

Bart, at 2:45 pm EDT on July 8, 2008

Sloppy references

I believe that sloppy references are associated with sloppy research in general. Show me a paper with multiple cited articles placed in the wrong journal or wrong decade or with an incorrect paper title, and I believe the data set will be garbage also.

Gillian, at 5:10 pm EDT on July 8, 2008

“BTW: Prof. Cramer — your work, and that of others, have put fear into the hearts of large numbers of potential academic frauds.”

Thanks for the kind words, but it isn’t “Prof.” Cramer. Since my part in the Bellesiles scandal, I have definitely become persona non grata in the history field.

Clayton E. Cramer, at 5:10 pm EDT on July 8, 2008

Prof check

” .. Thanks for the kind words, but it isn’t “Prof.” Cramer ..”

Once the professor — always the professor, IMHO.

I’m reminded of that transitional period in the Ivys when full professors weren’t always PhDs, while some of the instructors were PhDs. As if those young bucks were as knowledgeable as their elders.

Bart, at 6:10 pm EDT on July 8, 2008

Incorrect citations by scholars

I teach a required course called Urban Research Writing in an M.A. program in Urban Studies at Queens College, N.Y. City. I spend a lot of time checking references in student papers to see if they are citing them correctly. to facilitate this, students are required to submit photocopies or printouts of the pages on which the material cited appears, with the referenced material highlighted or indicated by marks in the margins. Incorrect citations are frequent but no worse, I now see, than those of supposedly reputable scholars! This is a sad state of affairs.

It also seems to indicate that few publications do adequate “fact checking” of citatations in articles they publish.

Jacqueline Skiles, Adj. Asst. Prof. at CUNY/Queens College, at 8:15 pm EDT on July 11, 2008

Order of author names

Isn’t there an error in the citation of the Interfaces article by the author of “Cite Check"? When I checked the citation, I found the order of author names reversed: Wright is the first author. Maybe Inside Higher Ed uses a different style manual than I do (APA and AMA, primarily), but I thought that authorship order remains the same across style manuals.

Nit-picker, at 2:05 pm EDT on July 12, 2008

SELECTIVE CITATION

I am a long-time science writer, and my husband is the author of multiple books. Over the years, we have found that numerous academics have clearly read our work, borrowing sources and summarizing or paraphrasing material unique to our stories or books; yet, often, we go uncited in notes and/or bibliography. We long since concluded, as we’ve also worked in the academy and have had occasion to explore this phenomenon from the inside, that scholars often don’t consider it necessary to bother citing mere journalists—perhaps since those on their tenure committees also consider only “legitimate” scholars to be worthy of recognition. Interestingly, articles we’ve published in the New York Times are cited with great frequency, presumably because publication there warrants a stamp of approval.

Gina Maranto, Director, English Composition at University of Miami, at 7:05 pm EDT on July 13, 2008

the politics of citation

I’ve long been interested in the politics of citation because it illustrates Karl Mannheim’s notion of “competition in the sphere of intellectual production". Unfortunately, it’s widespread and gives the lie to claims of objectivity and “value neutrality” in academic writing.Wolf Heydebrand, July 28, 08

Wolf Heydebrand, Prof.emeritus of sociology at NYU, at 4:40 pm EDT on July 28, 2008