Tackling the ‘credibility crisis’ in science

Zacharias Maniadis, Associate Professor of Economics, introduces new Southampton-headquartered initative, Meta-research & Evaluation of Scientific Scholarship & Institutions (MESSI), and reflects on the first Southampton Conference on the Credibility of Empirical Research (SCOCER 2015).

An important issue that has received increased attention in academic circles and the mass media concerns the difficulties in replicating a portion of scientific findings. Popular outlets such as The Guardian, The Economist, and The New Yorker have devoted coverage to the issue, thus raising doubts on the credibility of empirical results. The issue is being debated intensely among scientists in Psychology, Biomedicine, Marketing, and, as of late, Economics.

MESSI
The issue has motivated researchers at the Faculty of Social, Human, and Mathematical Sciences, University of Southampton, and at the School of Economics, University of Nottingham, to join forces in applying economic and behavioural insights for addressing the scientific credibility crisis. Meta-research and Evaluation of Scientific Scholarship and Institutions (MESSI) is a novel initiative with headquarters in Southampton and researchers-members from Economics, Psychology, and Social Statistics. MESSI aims to bring together scientists from all over the world as well as from disparate disciplines in order to assess and, if possible, defuse the crisis. MESSI has recently become affiliated with the Meta-Research Innovation Center (METRICS) at Stanford University, a centre “focused on transforming research practices to improve the quality of scientific studies in biomedicine and beyond.”

SCOCER 2015
One of the key objectives of MESSI is to help enhance the cross-fertilisation of ideas related to the credibility issue across disciplinary boundaries. Summoning researchers from various disciplines and inviting them to speak a common language presents its own challenges. We took the first step in that direction by organizing the first Southampton Conference on the Credibility of Empirical Research (SCOCER2015). We gratefully acknowledge financial support from Southampton’s Strategic Interdisciplinary Research Development Fund, Southampton’s Psychology Unit, Southampton’s Social Sciences, and the Royal Economic Society.

The organisers were MESSI founders Thomas Gall (Economics, Southampton), Constantine Sedikides (Psychology, Southampton), Fabio Tufano (Economics, Nottingham), and I.  Speakers included leading economists, psychologists, and biomedical scientists, along with several stakeholders. The conference was attended by over 100 scholars from prominent institutions around the globe.

The main goal of SCOCER2015 was to forge links between Economics and other disciplines. Keynote presentations were given by meta-researcher John Ioannidis (Stanford) and applied econometrician Charles Manski (Northwestern). Debate during the entire conference was lively and spirited. This was particularly so in the roundtable discussion, which featured the keynote speakers as well as demographer Jane Falkingham (Chair, Dean of the Faculty of Social, Human, and Mathematical Sciences, Southampton), behavioural economist Colin Camerer (Caltech), and psychologist Roger Giner-Sorolla (Kent).

The first day was devoted to contributions from psychological and biomedical sciences by researchers who have studied the credibility problem in depth and have developed advanced meta-research techniques. Meta-researcher Daniele Fanelli (Stanford) and biomedical researcher Malcom Macleod (Edinburgh) set the stage, presenting empirical evidence for the wider prevalence of the credibility crisis. The talks included revealing and humorous insights into the misadventures of scientific practice. As an example of a false positive, Macleod mentioned the exploits of a dead North Atlantic salmon, whose brain reacted systematically to various visual stimuli, despite the fish’s demise. Such an inference could be drawn from statistical methods that were based on MRI data.

The second session started with talks from psychologist Zoltan Dienes (Sussex) and biomedical researcher Despina Contopoulos-Ioannidis (Stanford), who illustrated new techniques in Bayesian data analysis and in meta-research, respectively. Next, psychologists Klaus Fiedler (Heidelberg), Roger Giner-Sorolla (Kent), and Simone Schnall (Cambridge) discussed implications of the replicability crisis for their discipline. One such implication raised the question of whether the relentless pursuit of replicability risks turning into a ‘witch hunt’ for researchers, thus obstructing creativity and imagination and being, in the end, counterproductive.

John Ioannidis and Charles Manski

METRICS director John Ioannidis summarised the first day of the conference and offered his own view on the credibility issue. He defined what meta-research is and exemplified it with a series of applications from an array of disciplines. He also discussed the incentives and mechanics of the knowledge system while assessing possible solutions for the credibility crisis.

The second day was devoted to Economics. Presenters gauged the credibility of relevant findings and considered the role of economic theory in the way institutional changes (for example, in the peer-review publication system) purport to tackle the credibility problem. Philosopher of Science Francesco Guala (Milan) pointed out that Economics has its own crisis in public confidence, which is unrelated to the problem of false positives. This crisis concerns the shattered public belief in economic science’s predictive power and quality of policy advice.

Economic theorists Marco Ottaviani (Bocconi), Elizabeth Schulte (Marburg), and Alex Tetenov (Carlo Alberto) showed how game-theoretic models can contribute to understanding of the role of incentives and predict the effects of institutional reforms. Analysing the complex interactions between agents operating under incentive systems (e.g., publication process), they offered powerful examples of how Economics insights can improve the evaluations of proposals for institutional reform, as well as generate new ones and inform relevant experimental research.

Manski approached the credibility problem looking at the role of assumptions in model-building. He emphasised that policy work often does not acknowledge adequately the degree of uncertainty in two kinds of expectations: which economic models represent a most accurate depiction of reality and what the magnitude of relevant parameters is. As a result, policy research can be misleading, thus undermining the credibility of Economics.

A lively roundtable discussion closed the conference. The topics included the role of incentives in ‘scientific production,’ and in particular the key role of the system of rewards and punishments. Have changes of the reward system been implemented in some disciplines as a result of the crisis? What can be learned from such changes? How can disparate scientific disciplines communicate in the face of crisis? How can rigorous social science methodology be implemented to assess proposed reforms?

Roundtable participants

MESSI AT SOUTHAMPTON
The Faculty of Social, Human, and Mathematical Sciences at University Southampton has the potential to become a key player in researching science as a social institution, utilising economic, psychological, and sociological insights. A positive public view of science cannot be taken for granted, but active research and initiatives across disciplines can strengthen the traditional view of science as a self-correcting process and as an arbiter of truth. MESSI aspires to carry out this vision.