“Am I Famous Yet?” Judging Scholarly Merit in Psychological Science

Departmental judgments. I have suggested four criteria that especially should matter in departmental judgments of scientific merit: quality, productivity, visibility, and impact (Sternberg, 2016). But evaluations of these criteria are highly subjective and can be influenced greatly by the particular sampling of professors in the institution. It is for this reason that most departments seek outside evaluations and also more quantitative indices of performance.

Letters from distinguished referees. Letters provide a useful qualitative evaluation of candidates. They go beyond mere numbers, such as the nature of the individual’s scientific contribution, why it is (or is not) important, and the referee’s assessment of future potential. They also can provide detailed and reflective qualitative feedback. But letters come inevitably with the biases of the letter writers. People tend to feel positively, and be more favorably disposed, toward others they view as similar to themselves (Sternberg, 1987, 1998), and letter writers are no exception. They may favor people they perceive to have similar research interests or approaches, comparable educational backgrounds, or similar professional values. And they may prefer candidates not because they are thought leaders (Antonakis, Cianciolo, & Sternberg, 2004) but rather because they are nonthreatening to the referees’ views or professional standing (Sternberg & Lubart, 1992). Moreover, candidates who are working in faddish areas and who have larger “networks”—perhaps because they went to a more prestigious graduate school, had a more prestigious postdoc, or had a more well-known adviser—are at an advantage. Letter writers are also susceptible to all the fads, biases, and personal idiosyncrasies any one of us can possess. Given the relatively small number of letters, there also is the possibility of serious error of measurement. One’s hiring, tenure, or promotion can rise or fall on the basis of one or two letter writers with a particular axe to grind, positive or negative. At the same time, referees’ letters are probably the single best qualitative source of evaluation. It is often said that democracy may be the worst form of government except for all others, and so it may be that external letters of evaluation from prominent and relevant referees are the worst form of qualitative evaluation except for all others.

Quantity of publications. Quantity of publications is a somewhat reasonable measure of productivity, but how really does one compare books to articles, edited books to written books, refereed articles to book chapters, and so on? Moreover, most articles can get published somewhere if one does not worry too much about the prestige of the outlet. Quantity in and of itself tells one relatively little about quality or impact.

Quantity of publications controlling for impact factors of journals. Impact factors of journals tell us roughly how much, on average, articles in particular journals are cited. One problem is the “on average.” Some articles in high-impact journals are rarely cited, whereas others are highly cited. Another problem is that prestigious journals sometimes are more conservative in what they are willing to publish. There are some prestigious journals that acquire an informal reputation for publishing seriously boring articles that have nothing in particular wrong with them but not much right with them either.

Number of citations. This measure has the advantage that it applies to the individual over a career and not to, say, a journal for which most of the impact data come from people other than the candidate. Number of citations gives a gross but useful measure of an individual’s visibility and impact. But investigators who work in hot areas, whose work is particularly controversial, or who write about topics that appeal to very broad audiences have an advantage, perhaps sometimes an undue advantage, in number of citations.

h. The h index is the number of publications cited at least h times. If Professor Grinder has an h index of 38, then he has 38 publications that have been cited at least 38 times. The h index is probably one of the best quantitative indices of impact, because it takes into account how both quality and quantity affect impact. It tends not to overvalue scientists who have written fewer papers that collectively have been cited a lot of times, although Dean Simonton (2002, 2004) has shown that such scientists are actually quite rare. Simonton’s data indicate a high correlation between quantity and quality, presumably because the scientists with the most hits also have the most misses. Thus, h is probably among the best indices currently available.

i10. The i10 index is the number of publications that have been cited at least 10 times. If Professor Grinder has an i10 index of 25, then he has 25 publications that have been cited at least 10 times. The advantage of this index is that it recognizes work that gets an appreciable but not great number of citations. The index especially values scientists who have written large numbers of papers—none of which have been cited a lot of times.

Grants and contracts. Through grants and contracts, scholars show that they have a systematic and valued proposed program of research. Moreover, after the first grant, funding shows a successful track record of performance on past grants (Sternberg, 2013). But because getting grants is so competitive these days, there may be some very fine scholars who do not get funded, perhaps because their work is not quite good enough but also perhaps because they are working in areas that attract little or no funding.

Editorships. Editorships (including full editorships, associate editorships, and consulting editorships) show that a scholar’s work has been recognized in his or her field. That said, some distinguished scholars just do not enjoy doing editorial work, whereas others who are perhaps not quite as distinguished may decide that their contribution will lie in their editorial work.

Service on major grant panels. Invited service on a grant panel, such as of the National Science Foundation or the National Institutes of Health, is another recognition of success in one’s professional endeavors. Serving on panels is often a tremendous amount of work, and some scholars simply prefer not to do it.

Awards. Awards are a useful measure of recognition by peers. Unlike citations, they measure, at least in theory, quality of work rather than just citation to work, which may be positive or negative. Awards tend more to reward scientists who contribute to well-established areas, who are from institutions with greater name recognition, and who have networks of colleagues and friends on awards committees.

Honorary doctorates. Most scientific awards are given by members of a given field. In contrast, honorary doctorates are recognitions by broader academic audiences—typically the entirety of a college or university—of the merit of a scholar’s work. Honorary doctorates tend to favor scientists who do work that is highly visible outside narrow fields of interest.