What is ‘results management’?

Accountants and auditors are often concerned with various kinds of alteration of figures, a practice euphemistically called ‘earnings management’. For example in “An Assessment of the Change in the Incidence of Earnings Management around the Enron-Andersen Episode” – Mark Nigrini

In 2001 Enron filed amended financial statements setting off a chain of events starting with its bankruptcy filing and including the conviction of Arthur Andersen for obstruction of justice. Earnings reports released in 2001 and 2002 were analyzed. The results showed that revenue numbers were subject to upwards management. Enron’s reported numbers are reviewed and these show a strong tendency towards making financial thresholds.

Benford’s law which is a conjecture concerning the expected frequency of digits in unmanaged data, is useful for detecting fraud and other forms of results management. I have posted on some results applied to time series data here, and here. An R module used for analysing these time-series data is available from this site here.

However, study of digit frequency only captures a part of what could be termed ‘results management’. As the goal in science, as in accounting, is an objective statements of results, all forms of ‘results management’ are to be deplored. Here are some practices that I have seen, or would consider to be ‘results management’.

• Rounding of results, up or down.
• Discarding of outliers.
• ‘Cherry picking’ between samples.
• Using biased statistical measures.
• Stating only positive or negative effects and not net effects (e.g. these species will have increased probability of extinction, but not mentioning those with decreased probability of extinction).
• Inflating significance to correlations (e.g. the test is not significant but we know from the physics that it must be real).
• Fabricating data points (e.g. filling in missing values).
• Inflating sample numbers (e.g. adding in more subjects).
• Statements so vague they can’t be proved wrong.
• Boilerplate implications in conclusions (e.g. this is important to study because of global warming).

I am sure there are many more — I’ll add them as I think of them. The term ‘results management is a useful concept as covers a spectrum from tacitly accepted to outright fraudulent activities. If I were to attempt a definition it would be something like the definition of earnings management: (investopedia.com)

Earnings management is a strategy used by the management of a company to deliberately manipulate the company’s earnings so the figures match a pre-determined target. Abusive earnings management is deemed by the Securities & Exchange Commission to be “a material and intentional misrepresentation of results”.

While some forms of earnings management are generally accepted, such as smoothing earnings to keep the figures relatively stable by adding and removing cash from reserve accounts (known colloquially as “cookie jar” accounts) the driving force behind managing earnings is to meet a pre-specified target (often an analysts’ consensus on earnings). As the great investor Warren Buffett once said, “Managers that always promise to ‘make the numbers’ will at some point be tempted to make up the numbers”.

Results management in science is very similar in form and motivation. The driving force is to meet expectations, usually for significant publishable results. This entails meeting the expectations of the peer reviewers, and the science community in general. Because of this it is difficult to see how science can be self-correcting within a group of like-thinking peers.

Business has found independent auditing a worthwhile investment through the confidence it creates in investors. Perhaps science funding agencies would also find value in directing some funds to independent audit of their funded projects. The audit would largely consist of statistical specialists, preferably from a different field. They would examine a fraction of projects an determine if obligations had been met, methodologies were appropriate, and conclusions justified by results.

Among a call for proposals there are many that ‘should’ be funded, because they tug on the ‘politically correct’ consensus, but deliver results that are usually foregone conclusions, and contribute little breakthrough research. This may over time correct such mal-investments. Publications arising from projects awarded the statistician’s stamp of approval would gain stature, guiding research towards greater accountability.