how food companies can sneak bias into scientific research

This is the first in our occasional series of longer reads titled, Essays on health. Enjoy!

Should we eat breakfast every day? How much dairy should we have? Should we use artificial sweeteners to replace sugar? If we had the answers to these questions, we could address some of today’s biggest public health problems such as heart disease, cancer, diabetes and obesity.

Consumer choice is often guided by recommendations about what we should eat, and these recommendations also play a role in the food that’s available for us. Recommendations take the form of dietary guidelines, food companies’ health claims, and clinical advice.

But there’s a problem. Recommendations are often conflicting and the source of advice not always transparent.

In September, a JAMA Internal Medicine study revealed that in the 1960s, the sugar industry paid scientists at Harvard University to minimise the link between sugar and heart disease. The historical papers the study was based on showed researchers were paid to shift the blame from sugar to fat as responsible for the heart disease epidemic.

The paper’s authors suggested many of today’s dietary recommendations may have been largely shaped by the sugar industry. And some experts have since questioned whether such misinformation can have led to today’s obesity crisis.

We’d like to think industry influence of this scale won’t happen again. We’d like to have enough systems in place to shine a spotlight on any potential bias, or risk of it, as soon as it happens. But the reason it took so long to expose the sugar industry’s tactics is bias can be well hidden. To avoid the potentially huge ramifications, we need much better systems in place when it comes to nutrition research.

How are national guidelines put together?

Governments issue national dietary guidelines to inform people’s food choices and the nation’s food policies. To be credible and scientifically sound, they should obviously be built on rigorous evidence.

Best practice for creating guidelines includes beginning the process with a systematic review, which is a study that identifies all the available evidence on a particular research question. This ensures studies favourable to a particular party can’t be cherry-picked. But systematic reviews are only as valid as the studies out there.

An important part of any systematic review is to evaluate the biases in the studies included. Public health dietary guidelines and policies are influenced by political, economic and social factors. That’s inescapable. But if the evidence on which these decisions are based is flawed, the entire foundation for systematic reviews, guidelines and policy, crumbles.

So identifying and minimising bias in each part of the research process – from the researcher’s decision on which question to answer in the study, to the publication of the results – is essential to having a strong evidence base.

Bias in research is the systematic error or deviation from true results or inferences of a study. Pharmaceutical, tobacco or chemical industry funding of research biases human studies towards outcomes favourable to the sponsor.

Even when studies use similar rigorous methods – such as keeping study information away from participants (blinding) or removing selection bias between groups of patients (randomisation) – studies sponsored by a drug’s manufacturer are more likely to find the drug is more effective or less harmful than a placebo or other drugs.

This bias in pharmaceutical industry sponsored studies is just like the sugar industry sponsored studies that downplayed sugar’s link to heart disease while putting the blame on fat.

Financial conflicts of interest between researchers and industry have also been associated with research outcomes that favour companies researchers are affiliated with.

So how does this happen? How can industry-funded studies use methods similar to non-industry funded studies but have different results? Because bias can be introduced in several ways, such as in the research agenda itself, the way research questions are asked, how the studies are conducted behind the scenes, and the publication of the studies.

Industry influences on these other sources of bias in research often remains hidden for decades.

Types of hidden bias

It took over 40 years to show how the tobacco industry undermined the research agenda on the health effects of secondhand smoke.

It did this by funding “distracting” research through The Center for Indoor Air Research, which three tobacco companies created and funded. Throughout the 1990s, this centre funded dozens of research projects that suggested components of indoor air, such as carpet off-gases or dirty air filters, were more harmful than tobacco. The centre did not fund research on secondhand smoke.

There is a high risk of bias when the methodology of the study (how the study is designed) leads to an error when assessing the magnitude or direction of results. Clinical trials with a high risk of methodological bias (such as those lacking randomisation or blinding) are more likely to exaggerate the efficacy of drugs and underestimate their harms.

A 2007 paper that compared over 500 studies found those funded by pharmaceutical companies were half as likely to report negative effects of corticosteroid drugs (used to treat allergies and asthma) as those not funded by pharmaceutical companies.

Many industry-sponsored studies of drugs are conducted for regulatory approval and the regulators require certain methodological standards. So often, the design of industry-sponsored studies is pretty good and the bias is elsewhere. It can be in how the questions are framed or another common form: publication bias.

Publication bias occurs when entire research studies are not published, or only selected results from the studies are published. It is a common myth publication bias comes about because scientific journal editors reject studies that don’t support the hypothesis or question the studies were asking. These are called negative or statistically non-significant studies. But negative research is as likely to get published as positive research. So it’s not that.

Analysis of internal pharmaceutical industry documents from 1994 to 1998 shows the pharmaceutical industry had a deliberate strategy to suppress publication of sponsored research unfavourable to its products. Industry-funded investigators were not allowed to publish negative research that did not support the efficacy or safety of the drugs being tested.

This has contributed to a clinical literature dominated by studies demonstrating the efficacy or safety of drugs. The tobacco industry also has a history of stopping the publication of research it funded if the findings didn’t lean in favour of tobacco products.

Previous research on bias in tobacco, pharmaceutical, and other industry-sponsored research is relevant here because the biases that affect research outcomes are the same, regardless of the exposure or intervention being studied. When it comes to nutrition research, we actually know little about how corporate sponsorship or conflicts of interest might bias the research agenda, design, outcomes and reporting.

Industry influence on nutrition research

The credibility of nutrition research has come under attack because the funding source is often not transparent and industry-funded research affects food policy. But we actually know very little about how sponsorship biases nutrition research.

Our systematic review, published this week in JAMA Internal Medicine, identified and evaluated all studies that assessed the association between food industry sponsorship and published outcomes of nutrition studies.

We were surprised to find few studies examining the effects of industry sponsorship on the actual, numerical findings of the studies. Only two of 12 studies assessed the association between food-industry sponsorship and the statistical significance of research results, and neither found a link.

Only one paper found studies sponsored by the food industry reported significantly smaller harmful effects of consuming soft drinks than those without industry sponsorship. Overall, our review showed we know very little about the association between industry sponsorship or authors’ conflicts of interest and the actual results of nutrition research.

More studies assessed the association of industry sponsorship with authors’ conclusions or interpretations of their findings (not the results). Eight reports, when taken together, found industry sponsored studies had a 31% increase in risk, compared to non-industry sponsored studies, of having a conclusion favouring the sponsor’s product.

So what we know is that food industry sponsorship is associated with researchers interpreting their findings to favour the sponsor’s products. Conclusions don’t always agree with results but can be spun to make readers’ interpretations more favourable.

For example, a study might find that a particular diet leads to weight loss and an increase in heart disease but the harmful effects of heart disease are omitted from the conclusion. Only the weight loss is mentioned. This spin on conclusions is a tactic in other industries and can influence how research is interpreted.

But it is the results (the research data) that really matters. From the standpoint of developing systematic reviews and evidence-based recommendations, the results are more important than conclusions because only the data, and not a researchers interpretation of them, are included in the reviews.

We need more rigorous investigation of the effects of industry sponsorship on the results of both primary nutrition studies and reviews. For example, our recent study examined 31 reviews of the effects of artificial sweeteners on weight loss. We found reviews funded by artificial sweetener companies were about 17 times as likely to have statistically significant results showing artificial sweeteners use is associated with weight loss, compared to reviews with other sponsors.

Nutrition research agenda

Our studies mentioned above didn’t identify any differences in the quality of industry-sponsored and non-industry sponsored nutrition research. But, similar to research sponsored by the pharmaceutical or tobacco industries, sponsors could affect outcomes by setting the research agenda, framing the questions or influencing publication.

A research agenda focused on single ingredients (such as sugar) or foods (such as nuts) rather than their interactions or dietary patterns may favour food-industry interests. This is because it may provide a platform to market a certain type of food or processed foods containing or lacking specific ingredients, such as sugar-free drinks.

Most data sources used to study publication bias in other research areas are not available for nutrition research, which make it more difficult to detect.

Researchers have identified publication bias in pharmaceutical and tobacco research by comparing the full reports of drug studies submitted to regulatory agencies with publications in the scientific literature. Researchers have also compared data released in legal settlements with published research articles. There are no similar regulatory databases for foods or dietary products.

It is possible to use statistical methods to estimate publication bias in large samples of nutrition research, as in other research areas. Interviewing industry-funded researchers could be another way to identify publication bias.

Another obstacle to rigorously assessing bias in nutrition research is the lack of transparency about funding sources and conflicts of interest. Our review of artificial-sweetener studies found authors of 42% of them had conflicts of interest not disclosed in the published article.

Also, about one third of the reviews didn’t disclose their funding sources. Although disclosure in journals is improving over time, not all journals enforce disclosure guidelines for author conflicts of interest and research funding sources.

Reducing bias in nutrition research

Studies on research bias related to pharmaceutical and tobacco industry sponsorship and conflicts of interest has led to international reforms. These have been in the area of government requirements for research transparency and data accessibility, stricter journal and university standards for managing conflicts of interest, and methodological standards for critiquing and reporting evidence (and conducting systematic reviews). Similar reforms are needed in nutrition research.

Further studies will determine which mechanisms to reduce bias should be urgently implemented for nutrition research. Options include:

• refined methods for evaluating studies used in systematic reviews

• enforced policies for disclosing, managing or eliminating financial conflicts of interest across all nutrition-related journals and professional associations

• mechanisms to reduce publication bias, such as study registries that describe the methods of ongoing studies, or providing open access data

• revised research agendas to address neglected topics and to produce studies relevant to population health, without corporate sponsors driving the agenda

• independent sources of funding for nutrition research, or, at a minimum, industry sources pooling their funding with research funds administered by an independent party.

In the current economic climate, in which industry funding is encouraged by universities, studying bias is important and contentious research.

Research institutions should implement strategies that reduce the risk of bias when industry sponsors research. They could do this by a risk-benefit assessment for accepting industry sponsorship of research. This would evaluate the sponsor’s control of the design, conduct and publication of the research, as well as any risk to the institution’s reputation.

The full effects of industry sponsorship and financial conflicts of interest on nutrition research remain hidden. An evidence base as rigorous and extensive as the the one on bias in pharmaceutical and tobacco research is needed to illuminate how nutrition research is at risk of bias.