Shadowes

Roberto Casati (CNRS-EHESS-ENS, Paris, France)

What follows is a series of back-of-postcard remarks about the thorny issue of research quality and how to assess it. Some of it is hypothetical. Some is based on published research, reports, editorials, or interviews. Definitely more empirical research on various aspects described here is needed. I was involved in a three-year project of the European Commission (LiquidPublications) and some of the ideas here are part of a in progress Green Paper we jointly wrote for the Commission. Some are going to be used in a report I am writing for the French Ministry of Research. Opinions expressed here are, of course, quite personal. I will be glad for any comment and references to existing research. Opinions expressed here are mine and should not be ascribed to my employer.

Rationale: A lot of research is funded and produced nowadays. Evaluation is of the essence: the output is measured in many different ways, projects are assessed before being funded, researchers have reputations, nations improve their wealth. What concerns me here is the question of potentially risky feedback loops of our necessary but ever more pervasive evaluation practices. I take it for granted that we need to evaluate research, and warn about possible difficulties.

Loop 1 – Biomedical sciences bias research evaluation

Fact: Biomedical issues are perceived as fundamental in our society.

A chain of consequences:

1. Biomedical research has the lion’s share in research fund allocation.

2. A large community of researchers is involved in biomedical research.

3. A large number of articles is published in the biomedical sciences

This number is further increased by the fact that biomedical companies fund both journals and articles¹ , (unethical behavior is well documented²)

4. The large number of articles supports citation-based evaluation.

5. Citation-based evaluation induces strategic behaviors (well beyond the medical sciences: cf. Kapeller for economics; ³ cf. Malle for psychology⁴; see Loop 2).

6. Citation-induced strategic behavior increases the number of articles and reinforces the prestige of some winner-take-all journals (ref. Marder, Kettenmann, Grillner).

Finally, comparatively high values of scientometric indicators for biomedical sciences lead to further requests of funds, and the loop is closed. More funds, therefore more article, therefore more funds.

Loop 2 – Citation-induced strategic behavior affects the quality of research.

In the above context, it has been observed that

1. Writing time increasingly dominates reading time. ⁵

1.1. Thus only abstracts tend to be read.

2. Time is spent in aiming for publication in top journals, less prestigious outlets are neglected, and PhD time and energy is in fact wasted if publication does not follow.⁶

3. Negative results are not published. This leads to unnecessary duplication of efforts. Ethical issues have been raised about unnecessary animal suffering (See EU recommendation)⁷.

Sub-Loop

More technical sub loops directly affect research output.

Fact: Referees eveluate papers by potential competitors.

Consequences:

1. Referees try to dump citations of teir own work in papers by competitors

2. And artificial and biased inflation of citations follows

3. Uncontroversial papers are preferred over controversial papers and innovation is stifled.

Loop 3 – The hidden costs of shifting research funds to project-based research

Fact: Research funding is increasingly project-based. This means that more and more scientific jobs are on short-term contracts.

Consequences:

1. Short-term researchers are labor-intensive but have lower commitment (they need to spend a sizable amount of their time in looking for the next job).

2. Senior researchers waste their time in training an intrinsically volatile workforce.

Thus:

3. Lab research memory is lost.

4. Senior researchers use an ever increasing share of their time in administrative work: project writing, project administration; reporting.

5. Senior researchers delegate to contracted junior researchers the administrative part, and thus

6. Junior researchers have less time for doing research in a critical period of their life.

Loop 4 – Costs of political choices to innovation

Fact: Research funding is project-based and research lines are centrally (politically) steered.

Consequences:

1. Strategic behavior is generated: groups with the ability to write projects according to specifications are privileged over less strategic groups.

2. Strategically minded groups obtain more funding, that they can invest in even more project writing.

3. Corruption of research: projects are risk-aversive and novelty-aversive, their outcome is probably already available.

4. Furthermore, blue-sky research is inhibited, and potential for innovation is lost.

Loop 5- Humanities do not fit scientometrics well

Compliance with scientometrics derived from biomedical model, combined with small, fragmented communities in the humanities, combined with publication of books and book chapters rather than research articles, combined with importance of reading (determining lesser writing time, hence lesser productions in general), induces lower bibliometric indices for humanities.

Consequences:

1. This puts humanities at a double disadvantage in competition for funds: they receive less funds because they are perceived as less important than, say, cancer research; but in consequence less articles are produced, impact factors are lower, and a further competitive disadvantages follows (see Loop 1).

2. This ecology is favorable to corruption of humanities. (More generally, humanities are intrinsically study-intensive, not result-oriented, and thereby suffer from specific types of measurement.)

3. Neglect of research published in “minor” journals⁸

One cas imagine some lines of remediation here.

For instance, a vigourous re-assessment of the value of research in the humanities (to give but an example, even many issues that are perceived as mainly biomedical – aging, climate change, drugs, dyslexia, to mention but a few – are in fact much wider societal issues; not least the medicalization of society itself! Total spending on health care is 16% of GDP in US in 2007 and figures can increase to almost 49% in 2082⁹)

Another possible remediation consists in re-thinking evaluation in the humanities; so as to avoid its collapse on biomedically inspired metrics. (Peer evaluation appears to be the safer bet here. However, there is a set of less traveled paths to the measurement of general impact of research in humanities: numbers of visitors at curated exhibits¹⁰; factoring in of time devoted to reading¹¹; number of foreign editions of a book)]

Loop 5 – Humanities, continued; US example; costs of outsourcing evaluation

Fact: Publication metrics based on rules of thumb (“2 books required for tenure”)

Consequences;

1. Tenure is outsourced to University Presses’ editorial choices¹²

2. Corruption of the quality of publication: uninspiring books.

A Possible remediation: require actual reading of works by peer committees in universities, forbid reliance on general cv assessment, even less on indexes.

Loop 6 – Humanities, ctd.: costs of absence of agreed upon metrics

Fact: the humanities are in generally not evaluated

Consequences: Unnecessary and counterproductive absence of characterization of humanities¹³

Remediation: independent, evaluation-free tools for characterization (RIBAC)]

Loop 7 – Rhetorical insistence on “excellence”

Fact: Excellence in research is assumed to be a target in many policies.

Consequences:

1. Research that is around the median is neglected.

2. Difficulty for median research to improve

However median research has to exist ¹⁴ and should be supported, as no one can prejudge which research is excellent from the outset.

Loop 8 – Costs of Incentives (e.g. monetary incentives)

Fact: A number of incentives only work for those who are close to the top ten percent and are likely to receive them¹⁵

Consequences:

1. Frustration in median researcher

2. Feeling that a certain type of effort is not worth;

3. The median researcher moves towards the lower end of distribution

Loop 9 – Rhetorical insistence on indicators and their respective advantages

If we focus all of our attention on indicators, we may neglect some of the main issues; in particular

• deliberation about means is taken to be deliberation about ends of the research process.
• understanding and risks of strategic behavior is neglected

Remediation: in a limited extent, we may experiment with alternate models (on which more empirical research is needed). Here are two examples:

1. Lotteries for seed grants

Advantages:

• No project writing, no reporting
  No evaluation process (big savings)
  No personal bad feelings, and sometimes good gratification

2. Randomly elect researchers to pursue their own projects for a given time (one semester, one year)

3. One day per week of sandbox research, protected time for any researcher to devote to his/her own project

Loop 10 – Costs of reliance on automated agents: Google Scholar, ISIS

Fact: The use of some of the instruments (such as Google Scholar and Isis) to compute various indexes is more an more widespread

Risk: Opacity of the underlying databases

Possible Remediation: exploration of larges bases, including blogs; exclusion of commentaries by close researchers. Development of “homophily aware” citation search bots (such as the one developed in LiquidPublication).

Loop 11 – Costs of reliance on automated agents, continued: Google Books scenario

Fact: Erosion of effective reading time.

Consequences:

1. “snippet reading”: short quotes are read as snippets in Google Books¹⁶

2. main point of the book, its unity, is lost (L. Water).

3. books are going to be written for being read through Gbooks or automated agents, not by judging humans.

Loop 12 – The “Shangai effect” – costs for society of massive presence of private sector in higher education

Fact: ever inceasing focus on (opaque) indicators that put a premium on private universities which have vast sums at their disposal

Consequence:

“Rich parents would relish the opportunity to drive fees even higher, beyond the reach of less wealthy parents of more able children.”¹⁷

Loop 13 – Authors sign articles that they did not even read

“I am officially the author of over 296 peer-reviewed journal articles, as of May 26, 2006. Yes, that’s correct: 296 articles. Many of these I have not even read”: a very candid statement¹⁸ by physicist David.C.Williams. Notice that it is not uncommon to publish papers with hundreds of authors. See for instance “Measurement of the ZZ production cross section in pp? collisions at ?s=1.96 TeV”¹⁹, a paper with 421 authors, or “Initial Sequencing and Analysis of the Human Genome”, published by Nature and listing approximately 2900 authors²⁰. It is simply incredible that unstructured lists of authors be considered and, what is worse, counted in measurement without any significant weighting.

Remediation: Gloria Origgi, Judith Simon and myself have proposed to add a “production box” to papers resulting from collaborations, in which microcredits are attributed to functional portions of the paper (such as figures, datasets, and even summaries). The “production box”, hollywood-style, has a line for the Lab Director who did not write (or even read) the article. ²¹

Alternate models for research productivity

We need to think harder about what “productivity” really means: quality of research needs to be improved as well as quantity²².

Reference