alek0

Since so many have been mentioning publication in peer reviewed journal as "proof" that something works or not, has adverse effects or not, I would like to ask several questions to those who accept research published in peer reviewed journals without questioning:

1) Are you familiar with peer review process? Have you published any articles in peer reviewed journals and have you reviewed any articles?

2) Your opinion on objectivity and source of funding. There is lots of evidence that source of funding causes bias, and here are some examples:

TITLE: The association between funding by commercial interests and study outcome in randomized controlled drug trials.
AUTHOR: Yaphe,-J; Edman,-R; Knishkowy,-B; Herman,-J
SOURCE: Fam-Pract. 2001 Dec; 18 6: 565-8.
MAIN ABSTRACT: BACKGROUND: Previous studies limited to specific drugs or journal types have shown an association between the source of funding of research and the published results. OBJECTIVE: The aim of the present study was to determine the association between source of support of research and published outcomes of randomized controlled drug trials in general interest medical journals. METHODS: Randomized controlled drug trials n = 314 published in five general interest medical journals over a 2-year period were reviewed. Study outcome was classified as positive or negative. Support was classified as pharmaceutical industry or non-industry. Association between source of support and outcome was tested with the chi-squared statistic. RESULTS: Positive findings were found in 77% of studies, negative findings in 20% and an uncertain outcome in 3%. Support from commercial sources was found in 68% of trials. Negative findings were found in 13% of industry-supported studies and in 35% of non-industry-supported studies chi-squared = 18.36, P < 0.0001, odds ratio = 3.54, 95% confidence interval 1.90-6.62. CONCLUSIONS: An association was found between the source of study support and the published outcome. Though the reason for this association cannot be determined from the data collected, future studies may clarify the importance of this finding for readers concerned with the relationship of funding bodies to the publication of research outcomes.

TITLE: Are pharmaceuticals cost-effective? A review of the evidence.
AUTHOR: Neumann,-P-J; Sandberg,-E-A; Bell,-C-M; Stone,-P-W; Chapman,-R-H
SOURCE: Health-Aff-Millwood. 2000 Mar-Apr; 19 2: 92-109.
ABSTRACT: The argument that prescription drugs are cost-effective has been made both by the pharmaceutical industry to support rising drug prices and expenditures, and by advocates of expanded drug coverage for elderly and low-income persons. A new database of 228 published cost-utility analyses sheds light on the issue. According to published data, some drugs do save money or are cost-effective, but the issue depends critically on the context in which the drug is used and the intervention with which it is compared. Cost-utility analyses funded by the drug industry tend to report more favorable results than do those funded by nonindustry sources. Cost-effectiveness analysis can help policymakers to determine whether drugs and other interventions offer value for money.

TITLE: The uncertainty principle and industry-sponsored research.
AUTHOR: Djulbegovic,-B; Lacevic,-M; Cantor,-A; Fields,-K-K; Bennett,-C-L; Adams,-J-R; Kuderer,-N-M; Lyman,-G-H
SOURCE: Lancet. 2000 Aug 19; 356 9230
: 635-8.
ABSTRACT: BACKGROUND: Reporting of pharmaceutical-industry-sponsored randomised clinical trials often result in biased findings, either due to selective reporting of studies with non-equivalent arms or publication of low-quality papers, wherein unfavourable results are incompletely described. A randomised trial should be conducted only if there is substantial uncertainty about the relative value of one treatment versus another. Studies in which intervention and control are thought to be non-equivalent violates the uncertainty principle. METHODS: We examined the quality of 136 published randomised trials that focused on one disease category multiple myeloma and adherence to the uncertainty principle. To evaluate whether the uncertainty principle was upheld, we compared the number of studies favouring experimental treatments over standard ones. We analysed data according to the source of funding. FINDINGS: Trials funded solely or in part by 35 profit-making organisations had a trend toward higher quality scores mean 2.94 SD 1.3; median 3 than randomised trials supported by 95 governmental or other non-profit organisations 2.4 0.8; 2; p=0.06. Overall, the uncertainty principle was upheld, with 44% of randomised trials favouring standard treatments and 56% innovative treatments p=0.17; mean and median preference evaluation scores were 3.7 1.0 and 4. However, when the analysis was done according to the source of funding, studies funded by non-profit organisations maintained equipoise favouring new therapies over standard ones 47% vs 53%; p=0.608 to a greater extent than randomised trials supported solely or in part by profit-making organisations 74% vs 26%; p=0.004. INTERPRETATION: The reported bias in research sponsored by the pharmaceutical industry may be a consequence of violations of the uncertainty principle. Sponsors of clinical trials should be encouraged to report all results and to choose appropriate comparative controls.

TITLE: Raising questions about antidepressants.
AUTHOR: Antonuccio,-D-O; Danton,-W-G; DeNelsky,-G-Y; Greenberg,-R-P; Gordon,-J-S
SOURCE: Psychother-Psychosom. 1999; 68 1: 3-14.
ABSTRACT: Antidepressant medication has apparently become the most popular treatment for depression in the USA. Several beliefs about the efficacy of antidepressant medications prevail among mental health professionals and the public. This paper explores relevant research data and raises questions about these beliefs. Many of the common beliefs about these medications are not adequately supported by scientific data. The following issues are raised: 1 industry-funded research studies which result in negative findings sometimes do not get published; 2 placebo washout procedures may bias results in some studies; 3 there are serious questions about the integrity of the double-blind procedure; 4 the 'true' antidepressant drug effect in adults appears to be relatively small; 5 there is minimal evidence of antidepressant efficacy in children; 6 side effects are fairly common even with the newer antidepressants; 7 combining medications raises the risk for more serious complications; 8 all antidepressants can cause withdrawal symptoms; 9 genetic influences on unipolar depression appear to be weaker than environmental influences; 10 biochemical theories of depression are as yet unproven; 11 biological markers specific for depression have been elusive; 12 dosage and plasma levels of antidepressants have been minimally related to treatment outcome; 13 preliminary evidence suggests that patients who improve with cognitive-behavioral psychotherapy show similar biological changes as those who respond to medication, and 14 the evidence suggests that psychological interventions are at least as effective as pharmacotherapy in treating depression, even if severe, especially when patient-rated measures are used and long-term follow-up is considered.

TITLE: Evaluation of conflict of interest in economic analyses of new drugs used in oncology.
AUTHOR: Friedberg,-M; Saffran,-B; Stinson,-T-J; Nelson,-W; Bennett,-C-L
SOURCE: JAMA. 1999 Oct 20; 282 15: 1453-7.
ABSTRACT: CONTEXT: Recent studies have found that when investigators have financial relationships with pharmaceutical or product manufacturers, they are less likely to criticize the safety or efficacy of these agents. The effects of health economics research on pharmaceutical company revenue make drug investigations potentially vulnerable to this bias. OBJECTIVE: To determine whether there is an association between pharmaceutical industry sponsorship and economic assessment of oncology drugs. DESIGN: MEDLINE and HealthSTAR databases 1988-1998 were searched for original English-language research articles of cost or cost-effectiveness analyses of 6 oncology drugs in 3 new drug categories hematopoietic colony-stimulating factors, serotonin antagonist antiemetics, and taxanes, yielding 44 eligible articles. Two investigators independently abstracted each article based on specific criteria. MAIN OUTCOME MEASURE: Relationships between funding source and 1 qualitative cost assessment favorable, neutral, or unfavorable and 2 qualitative conclusions that overstated quantitative results. RESULTS: Pharmaceutical company-sponsored studies were less likely than nonprofit-sponsored studies to report unfavorable qualitative conclusions 1/20 5% vs 9/24 38%; P = .04, whereas overstatements of quantitative results were not significantly different in pharmaceutical company-sponsored 6/20 30% vs nonprofit-sponsored 3/24 13% studies P = .26. CONCLUSIONS: Although we did not identify bias in individual studies, these findings indicate that pharmaceutical company sponsorship of economic analyses is associated with reduced likelihood of reporting unfavorable results.

TITLE: Why review articles on the health effects of passive smoking reach different conclusions.
AUTHOR: Barnes,-D-E; Bero,-L-A
SOURCE: JAMA. 1998 May 20; 279 19: 1566-70.
ABSTRACT: OBJECTIVE: To determine whether the conclusions of review articles on the health effects of passive smoking are associated with article quality, the affiliations of their authors, or other article characteristics. DATA SOURCES: Review articles published from 1980 to 1995 were identified through electronic searches of MEDLINE and EMBASE and from a database of symposium proceedings on passive smoking. ARTICLE SELECTION: An article was included if its stated or implied purpose was to review the scientific evidence that passive smoking is associated with 1 or more health outcomes. Articles were excluded if they did not focus specifically on the health effects of passive smoking or if they were not written in English. DATA EXTRACTION: Review article quality was evaluated by 2 independent assessors who were trained, followed a written protocol, had no disclosed conflicts of interest, and were blinded to all study hypotheses and identifying characteristics of articles. Article conclusions were categorized by the 2 assessors and by one of the authors. Author affiliation was classified as either tobacco industry affiliated or not, based on whether the authors were known to have received funding from or participated in activities sponsored by the tobacco industry. Other article characteristics were classified by one of the authors using predefined criteria. DATA SYNTHESIS: A total of 106 reviews were identified. Overall, 37% 39/106 of reviews concluded that passive smoking is not harmful to health; 74% 29/39 of these were written by authors with tobacco industry affiliations. In multiple logistic regression analyses controlling for article quality, peer review status, article topic, and year of publication, the only factor associated with concluding that passive smoking is not harmful was whether an author was affiliated with the tobacco industry odds ratio, 88.4; 95% confidence interval, 16.4-476.5; P<.001. CONCLUSIONS: The conclusions of review articles are strongly associated with the affiliations of their authors. Authors of review articles should disclose potential financial conflicts of interest, and readers of review articles should consider authors' affiliations when deciding how to judge an article's conclusions.

TITLE: A reappraisal of economic evaluation of pharmaceuticals. Science or marketing?
AUTHOR: Drummond,-M-F
SOURCE: Pharmacoeconomics. 1998 Jul; 14 1: 1-9.
ABSTRACT: In pharmacoeconomic research sponsored by companies, there is an obvious tension between the desire to undertake studies to show a marketing advantage and the desire to adhere to good scientific principles. This tension was explored in an early issue of PharmacoEconomics and is now revisited 5 years on. Bias is still perceived to be a major problem in industry-sponsored studies, both by healthcare decision-makers and journal editors. However, the debate about bias has matured over the years. Also, actual evidence of bias in study methodology is sparse, although biases in study topic selection and in the use, in promotion, of study results have been found. A number of constructive suggestions have been made in the last 5 years for reducing potential bias. These include developing methodological guidelines and standards, improving the peer review process, clarifying contractual relationships between sponsors and analysts, and ensuring appropriate use of studies in promotional activities. However, further initiatives could be undertaken. These include additional guidelines for specific detailed areas of economic evaluation methodology, changes in the structure of funding for pharmacoeconomic research, more education of consumers for pharmacoeconomic data and more research partnerships between industry and its customers.

TITLE: Research into smoking or nicotine and human cognitive performance: does the source of funding make a difference?
AUTHOR: Turner,-C; Spilich,-G-J
SOURCE: Addiction. 1997 Nov; 92 11: 1423-6.
ABSTRACT: AIMS: To establish whether there is a relationship between tobacco industry support of basic research and the conclusions drawn by the authors of that research. DESIGN: A sample of 91 papers investigating the effects of tobacco or nicotine use upon cognitive performance was analyzed to see if the pattern of conclusions drawn by researchers acknowledging tobacco industry support differed from the pattern of conclusions drawn by researchers not acknowledging tobacco industry support. FINDINGS: Scientists acknowledging tobacco industry support reported typically that nicotine or smoking improved cognitive performance while researchers not reporting the financial support of the tobacco industry were more nearly split on their conclusions. CONCLUSIONS: While it is only possible to speculate on the possible reasons, the existence of a possible bias in the published literature according to funding source must be given serious consideration.

TITLE: Conflicts of interest: conceptual and normative issues.
AUTHOR: Pritchard,-M-S
SOURCE: Acad-Med. 1996 Dec; 71 12: 1305-13.
ABSTRACT: Growing university-industry ties, particularly in biomedical areas, naturally raise concerns about conflicts of interest. Such conflicts are essentially problems in business and professional ethics. As of the fall of 1995, all institutions seeking funding from either the Public Health Service or the National Science Foundation have been required to maintain and enforce a written policy on conflicts of interest. The PHS and the NSF also require the disclosure of "significant" financial interests that might affect the research. Although the PHS and NSF requirements may prove helpful, they are not sufficient for monitoring the full range of serious conflicts of interest that can arise in university-industry relations. The PHS and the NSF are basically concerned with potential bias in the design, conduct, and reporting of research. Their disclosure requirements are restricted to financial considerations of $10,000 or more. However, bias in research can result from conflicts of interest when much less is at stake financially. Furthermore, it can arise at both individual and institutional levels. This article attempts to provide a conceptual and normative analysis of conflicts of interest that better enables us to understand the subtleties that can be involved. This article is one of three in this issue of Academic Medicine that deal with issues of conflict of interest in university-industry research relationships. These articles are discussed in an overview that precedes them.

TITLE: Third generation oral contraceptives and risk of venous thrombosis: meta-analysis.
AUTHOR: Kemmeren,-J-M; Algra,-A; Grobbee,-D-E
SOURCE: BMJ. 2001 Jul 21; 323 7305: 131-4.
ABSTRACT: OBJECTIVE: To evaluate quantitatively articles that compared effects of second and third generation oral contraceptives on risk of venous thrombosis. DESIGN: Meta-analysis. STUDIES: Cohort and case-control studies assessing risk of venous thromboembolism among women using oral contraceptives before October 1995. MAIN OUTCOME MEASURES: Pooled adjusted odds ratios calculated by a general variance based random effects method. When possible, two by two tables were extracted and combined by the Mantel-Haenszel method. RESULTS: The overall adjusted odds ratio for third versus second generation oral contraceptives was 1.7 95% confidence interval 1.4 to 2.0; seven studies. Similar risks were found when oral contraceptives containing desogestrel or gestodene were compared with those containing levonorgestrel. Among first time users, the odds ratio for third versus second generation preparations was 3.1 2.0 to 4.6; four studies. The odds ratio was 2.5 1.6 to 4.1; five studies for short term users compared with 2.0 1.4 to 2.7; five studies for longer term users. The odds ratio was 1.3 1.0 to 1.7 in studies funded by the pharmaceutical industry and 2.3 1.7 to 3.2 in other studies. Differences in age and certainty of diagnosis of venous thrombosis did not affect the results. CONCLUSIONS: This meta-analysis supports the view that third generation oral contraceptives are associated with an increased risk of venous thrombosis compared with second generation oral contraceptives. The increase cannot be explained by several potential biases.

S2Focus

I don't think anyone here accepts research in peer-reviewed journals without question. It sounds like you are trying to set up a straw man here.

I won't disagree that funding sources can bias the research. But in the absence of peer-review, there would be no checks on the validity of that research.

And yes, although I'm not a scientist, I have been a co-author on one or two peer-reviewed articles. The review process was a major source of heartburn. Having to revise a paper that you put a lot of time and effort into is not much fun!

It it is interesting to note that the sources you have cited in order to make the case that articles in peer-reviewed journals are not infallible happen to be peer-reviewed journal articles themselves!

Before you try to cast too much doubt on the admittedly imperfect peer-review process, just remember the old saying, "You can't beat a horse with no horse.".

In the absence of meaningful peer review, you can easily end up with the kind of worthless creationist trash that gets served up without question in many churches. Peer review, as imperfect as it is (of course, that can be said about *any* human endeavor), is really the only effective way to impose any kind of meaningful "quality control" on published research.

(Edited to add: I just checked your profile, so I'll back off on implying that you are trying to set up a creationist-style "straw-man". Your points about the infallibility (or lack thereof) of peer-reviewed research are well-taken. But there is no better way to impose quality control on research than the admittedly imperfect peer-review process that is used today).

[ May 25, 2002: Message edited by: S2Focus ]</p>

seebs

Who paid you to write this? What's their agenda?

alek0

S2Focus,

I am not saying that peer review is useless. But it is very far from perfect. Often people get to review articles which are not within their area of expertise. I've received articles not in my field several times, and I have promptly returned them to the editor with suggestions for more appropriate reviewer. But not everybody does that, so bad works get accepted and good works get rejected. Did you know that Phys. Rev. Lett. has rejected von Klitzing's work on quantum Hall effect for which he later got Nobel Prize (this is true fact, he showed us rejection letter on last ICPS).

One big problem is that majority of papers will eventually get published somewhere, even if they have been previously rejected in several journals.

Another big problem is that peer review is anonymous only one way. Referee always knows who authors are. Hence, you can have bias. I've wrote a comment to a paper of one guy and he hates my guts since. He always gives me negative review, while I am trying to be objective about his work.
Not because I am particularly good person, because major revision is more torturous than outright reject Just kidding, I beleive it is my duty to be as objective as possible no matter who author is and whether I personally agree with the content of the paper or not. But I admit that I am very thorough, which makes it tougher than usual. First I check the literature to see whther references are adequate. In 80% of the cases, authors leave out the references which disagree with their result. Then I double check the results, once I found serious error and poor guys had to recalculate all of it. Not everybody does this, but peer review would have been much better if people took it more seriously. However, when you have 100+ exam papers to grade, new equipment to order for the lab, your own paper to revise etc. than of course doing quality review has lower priority.

Another problem - some journals send papers to only one reviewer. Which makes the whole process significantly less objective. I really love the review process for new IEEE trans. on nanotechnology though - there are three reviewers, you access the paper online and you submit report online which makes whole thing a lot faster. Quite nice.

Since I am very familiar with the process, both as author and reviewer, it irks me when people regard the act of publication in a peer review journal as "the proof" of the value of the work. In scientific community, officially of course people pay attention to number of publications, number of citations, and impact factors of the journals. Unofficially, common attitude is that whether you'll publish in a particular journal or not is a lottery.