Detached9

Although I have a fair understanding of the scientific method, I've never understood the details of the peer-review process. All I know is that experts in the field review the authors paper before it is published. That doesn't tell me much.

How are these experts designated? Are these up the authors choosing (that could lead to complications)?

What do these experts look for in the paper? Do they ensure the interpretation meets the data from the report? Do they encourage the authors to clarify some points?

On a similar note, are there any books that go into explicit detail with regard to the rigors of science you would recommend? I'm especially curious with regard to ensuring experimenters don't fabricate their results. Afterall, outside of the experimenters, who would know what the true results of the experiment was? Must experimenters give video-taped evidence of their results or something?

And what about the statisticians who calculate the amount of deaths, marriages, etc, for each gender, race, and so on. Couldn't the authors be friends or acquaintances and knowingly distort the figures?

Thanks for any and all help.

theyeti

It differs by journal, but in general, it's the editors of the journal who choose the reviewers. I have never heard of a case where the author chooses his own reviewers -- that would seem to defeat the purpose. The editors of a given journal are usually themselves experts in the area that the journal is concerned with, and I believe that they usually have final say in what gets published. They also act to filter out the real crap before they waste anyone else's time with it. But when it comes to legitimate articles, they usually defer to the judgement of the reviewers.

In general, they look for the stuff that makes for a good scientific paper:

A. Does the study address a relevant question?

B. Do the methods offered up properly address the question?

C. Were the experiments conducted properly as reported?

D. Do the conclusions follow from the results?

E. Are there any additional experiments that would strengthen the results that should be conducted?

F. Is the writing clear and concise, does it use the proper terminology, etc.

That's just a short list, but it's the kind of thing that reviewers look for.

There is little to safeguard experimenters from doing this, other than relying on their sense of integrity. The best (perhaps only) way of catching a cheater is to replicate the experiment. Very few scientists are willing to risk their reputations by fabricating evidence, when they know that anyone can check their results by repeating their methods.

theyeti

catalyst

Theyeti pretty much hit on all of the relevant points, but if you are interested in more detail, an example of the process courtesy of the NIH can be found here.

Detached9

THEYETI:
C. Were the experiments conducted properly as reported?

D. Do the conclusions follow from the results?

E. Are there any additional experiments that would strengthen the results that should be conducted?

This is where I've often found problems with peer-reviewed studies. I wonder how they even get published given some of the problems with C, D, and E.

The best (perhaps only) way of catching a cheater is to replicate the experiment. Very few scientists are willing to risk their reputations by fabricating evidence, when they know that anyone can check their results by repeating their methods.

theyeti

I completely forgot about the replication part. I should have remembered that.

Thanks for the help.

Roland98

The ones I've published in allow us to suggest reviewers, so that we have people familiar with the techniques we use. (One of the worst things is to get back a paper or grant and have it critiqued by people who obviously have no idea what the hell they're talking about--ugh!) So the suggested reviewers are meant to avoid that problem. But definitely, the editors have the final say, and it's generally recommended to avoid suggesting people you've collaborated with in the past, to ward off any accusations of bias or conflict of interest.

BioBeing

What problems, specifically, have you encountered, related to the peer review process in general? It is not the reviewers job verify the conclusion (or even support it), rather verify that the authors could probably have reached the conclusion that they did from the experiements as reported, and the results they state they obtained. Also, alternate conclusion should be stated in the paper.

The peer-review process is not the be-all and end-all in scientific publishing. There is room for abuse by a reviewer who doesn't agree with your conclusion, for example. One is able to bypass this to a certain extent by also suggesting people who should not review your paper.

However, peer-review is probably the best system we have. Rolling-peer review, by publishing online and allowing users to post comments for example, is an alternative, but could easily get unwieldy. Some journals (esp in physics) IIRC are trying such procedures.

Undercurrent

I had a boss once who worked on a lidar system and got a paper back with suggestions to change "observations mad with the lidar" to "observations made through the lidar". That (and some other things) led us to believe that the reviewer thought that a lidar was something like an optical telescope that you could look through to make observations.

BTW, Det9, just to completely destroy your faith in the peer-review process, I've been one of the "designated experts" on more than one occaision. Thinking that some part of the sicentific process occurred in my rat-hole apartment with "Teletubbies" playing in the background gives me the giggles every time.