dogwood

I don't know if this has been covered recently, but I've written a review of Schwartz's new book on "after death communication" and put it up at the JREF and the John Edward forum at tvtalkshows. I thought I might as well schlep it over here if there's interest and if it hasn't already been done to death.

It's kind of long and only half finished, but any interest?

dogwood

Bunda

I can't speak for everyone else, but I'm interested. Why not? We are always looking for a good read, possibly about a bad read.

dogwood

Well, 1 yes vote's better than nothing I guess.


A Review of “The Afterlife Experiments” by Gary Schwartz, Ph.D. with William L. Simon. Part I


Following a brief introduction, I will provide a rough description of the experiments and then a review of the book itself. In this review, I will primarily focus on the science involved in these tests and the experimental design of Dr. Schwartz et al. Part I will present overall impressions. If there is interest, I will follow-up with Part II, which will look at each experiment in greater detail. I will not attempt to critique Dr. Schwartz’s personal views or philosophies, as ideally, these should have no bearing on the science involved.
I also want to make something clear in advance. At no time will I make any accusations of fraud or dishonesty against Dr. Schwartz. After seeing him present his work in November of last year, I formed the opinion that Dr. Schwartz is an honest, sincere man with a real desire to see a proper scientific examination of claims of afterlife communication. That opinion has not changed.


In “The Afterlife Experiments”, Dr. Schwartz describes a series of four experimental trials designed to test the possibility of the survival of consciousness after death. The survival hypothesis suggests that communication with the disembodied intelligence’s of the deceased can be accomplished by the living. This communication is facilitated through the use of mediums who “read” sitters wishing to contact “the other side”.
The basic protocol designed by Dr. Schwartz to test this hypothesis is simple. A medium, who is physically separated from a sitter in such a way that the two cannot see each other, makes statements that are allegedly provided by the spirits of the deceased. In addition to statements, the mediums will occasionally, or frequently depending on their style, ask confirmational questions that are limited to a yes or no answer. Once the reading is over, the sitter scores the results for accuracy. The trials become progressively more rigorous, adding controls designed to reduce the possibility of cold-reading, fraud, and reporter bias. Mediums are not told the identity of the sitters nor are they allowed any contact in advance. In most experiments, prior to the readings the sitters provide the researchers with names and personal details of the deceased that they desire to contact for later independent confirmation. In all cases, the readings and the scorings are recorded on video and/or audiotape.

A basic run down of the experiments is as follows:

Trial One: The HBO Dream-Team Experiment.

Mediums: Suzane Northrop, John Edward, George Anderson, Anne Gehman, Laurie Campbell
Sitters: Patricia Price (provided by HBO), Ronnie Nathanson (Met and chosen by experimenters only a few days prior to the experiment)

A medium is escorted into the room by a researcher after a sitter has been seated behind a screen. Following a two minute meditation period, the medium begins a ten to twenty minute reading of the sitter consisting of statements and questions that may only be answered by a yes or no.


Trial Two: The Miraval Silent Sitter Experiment

Mediums: Suzane Northrop, John Edward, Anne Gehman, Laurie Campbell
Sitters: Patricia Price, Elayne Russek, Diane Goldner, Heather Rist, Christopher Barentsen, and five others. (Note: All sitters are personally known by the experimenters)

Similar to the above experiment. However the question period is preceded by a ten-minute “sitter-silent” period in which the medium provides information without asking any questions of the sitter. (Note: of the 40 readings done, only one sitter scored the results from two of the mediums.)


Trial Three: The Canyon Ranch Experiment

Mediums: Suzane Northrop, John Edward, Laurie Campbell
Sitters: Juliet Speisman, Sabrina Geoffrion, Janna Excel, Terri Raymond, Lynn Ferro. (Note: All sitters are personally known by the experimenters)
Magician consultant: Ross Horowitz
Added control: Each sitter scores all other sitters readings to reduce possibility of rater bias.

Medium and sitter are back to back separated by “two sets of doubled white sheets.” Following the ten-minute sitter-silent period, the mediums are permitted to ask questions but do not hear the voice of the sitter. Instead, the sitter indicates their response to the experimenter by a shake or nod of the head. The experimenter then voices the response to the medium.

Trial Four: The Campbell White Crow Readings

Medium: Laurie Campbell
Sitters: George Dalzell and two others. (Notes: Only Dalzells reading is described. Dalzell is a working medium)

Unlike all previous experiments, only the medium is present with the primary investigators. The sitters remain in their own homes. The reading begins with a 30 minute meditated reading of the sitter with no contact whatsoever. This is followed by a ten-minute sitter-silent period. For this section, a call is placed to the sitters’ home, but the mediums’ phone is placed on mute. The sitter cannot hear the medium, and the sitter remains silent. Following this, the mute is turned off and the medium reads the notes taken during the previous periods for confirmation by the sitter.


Before I take a look at the specific details of each experiment in part II, I would like to address two points that are common to all, or most of them. These issues represent serious obstacles to the over all experimental design and if not addressed, in my opinion, invalidate everything presented within them.


The scoring procedure:

Schwartz describes some very impressive scores for each of the trials. 83% average accuracy for the first trial, followed by 77-85% for the second, 40% for the third, and 90-100% for the last. (Note that accuracy ratings in general drop as the experiments become more stringent, with the obvious exception of the last one, an issue that I’ll examine in part II.) It’s very important therefore to examine closely how these numbers were determined and to make sure no errors were introduced
.
Following the readings. The experimenters transcribed the information provided by the mediums and reworded them into brief statements. Some examples:

sitter is female
sitter’s grandmother is in spirit

see mountains…
and a person walking in white tennis shoes
and there is sun on her face
sitter’s father is tall man, with dark hair
see father in a suit

hear David
David has dark hair
David like a soul mate
that was close
sitter’s broken heart related to him
but sitter’s heart is broken, needs healing
connection with David and a car

(Note: The information when provided initially by the medium is expressed in a much more elegant and comprehensive fashion. The examples given above are from the scoring sheet only and not representative of the manner in which the reading is given.)
The statements are further identified by six categories of information; Names, Initials, Descriptions, Historical Facts, Temperament, and Opinions. This is a curious division because very little is done with these categories after they’ve been so segregated. One might think that initials would score lower than names on a probability scale, or a degree of difficulty, but this is not the case. Dr. Schwartz addresses this on pg. 295,

“Note that ratings of accuracy do not discriminate between differences in degree of specificity. For example, accurate information such as ‘M’ (initial), ‘Michael’ (name), ‘committed suicide’ (historical fact), ‘thin’ (personal description), ‘playful sense of humor’ (temperament), and ‘does not blame you for his decision to kill himself’ (opinion) differ in their degree of specificity, but they were all correct and received accuracy ratings of +3 by a sitter.” (Originally published in the paper “Accuracy and Replicability of Anomalous After Death Communication Across Highly Skilled Mediums” Journal of the Society for Psychical Research, 2001, Vol. 65.1, No. 862, pages 1-25)

More on the actual scoring, from page 119

“For each item, the sitter was asked to assign a rating on a hit-or-miss scale, in the range of –3 to +3, with the minus numbers representing a complete miss (-3), a probable miss (-2), or a possible miss (-1), and the plus numbers representing a possible hit (+1) to a definite hit (+3). If the sitter did not know, she was instructed to assign no rating. Along with rating each statement, the sitter was required to justify her answer and to tell us if information other than Opinion could not be verified by another living person.”
Sitters were also reminded that whenever they were uncertain, they were to assign a lower rating.

This method is highly subjective and very easily influenced by reporter bias (The conscious or unconscious desire to affect the outcome of the experiment). If you’ve ever wondered why skeptics are constantly bellowing for binary data, this is why. By the time of the Canyon Ranch experiments (Trial Three) Schwartz attempts to address this. (Page. 182)

“Because we were deeply concerned about the possibility of rater bias, we decided to require that each reading would be scored not just by the sitter of that reading, but by the other four sitters as well.”

On page 199, Schwartz describes the results of this control,

“When the sitters rated one another’s readings as a control group, they detected an average of half as many +3 hits per reading (6 for the control group versus 12 for the mediums). In other words, as expected, the mediums’ actual readings rated significantly higher than the control readings, suggesting that the mediums results were not achieved by guesswork.”

There is a very serious, and I would think obvious problem with this “control”. The sitters heard everything the mediums said when present for their own reading, and therefore could easily recognize their own transcribed sitting from the others. This does nothing to eliminate rater bias and in fact increases the risk. If the sitters are eager to prove that their reading was genuine, and if they can tell their own from those of the others, they may give the other sitters readings a biased lower score, while they give their own a higher one. Making the bias gap even broader!
There is another telling bit of information from the statistics Schwartz provides above. He mentions that the control group scored an average of half as many +3 hits for the other sitters. This means that, on average, 50% of the +3 type hits provided by the mediums applied to ALL of the sitters. Either all of the sitters in this example share some amazingly identical personal details about their lives, or most of the information was so generic that it could apply to almost anyone. A fact apparently not noticed by Schwartz. This is why a specificity scale of some sort is necessary. It is also why the scoring system used by Schwartz is completely ineffective for these studies.

The Controls

The main problem with the controls in this study, is that there aren’t any. Controls in this context refers to actual experimental controls that are used at the time of the experiment and not the general conditional controls as described for scoring above. Without proper controls an experiment is not an experiment, it is at best data gathering. I cannot stress this strongly enough. There were no controls during any of the experiments mentioned in this book. An ideal experiment has a positive control and a negative control. That means, a sample that you know will give positive results and a sample that you know will give negative results, respectively. Most experiments are not ideal however, including those on paranormal phenomena. Positive controls are not always easy to come by, especially for a science that is in its infancy. After all, if the experimenters knew without a doubt that they had a legitimate medium, then the experiments wouldn’t be necessary. This doesn’t mean that they couldn’t have made use of a compelling negative control. (Note: At first look, the use of a “hot reader” might seem like a good positive control, someone who was allowed to “cheat” and dig up data on the sitter in advance. But the fact that someone in the experimental group had done so would open the door to accusations that the info had been shared.)
The primary question being asked here is, “Does the data reflect that the ‘mediums’ are actually talking to the dead, or are they merely cold-reading?” (Or hot reading or warm reading?) Schwartz attempts to present his accuracy ratings as evidence that the mediums could not have arrived at the information by guessing, however no one within the experiment IS guessing (or at least admitting to it). The only way to properly demonstrate the difference between mediums and cold readers in a controlled setting, is to include an admitted cold-reader in the experiment. This is not done even once. Dr. Schwartz does make an attempt to recruit a cold reader, as he mentions on page 213.

In the aftermath of a FOX television show in which a cold reader in a mall is compared to Schwartz’s controlled readings, Schwartz attempts to contact the cold reader.

“We were more certain than ever that someone with his abilities could be enormously helpful in our studies. Watching our raw videotapes, he could advise us whether the mediums were using any trickery he could identify. And we hoped he might be willing to serve as medium under our controlled conditions, to see whether he could do as well as the research mediums we had been working with.”

Schwartz attempted several times to contact the man through the FOX network, but never received a reply. They eventually, “gave up trying.” And that’s it. Schwartz had previously used the stage medium Ross Horowitz in the Canyon Ranch experiment as a “magician consultant”. Was he unreachable too? What about an ad in Variety, or a tour of the Vegas strip? Why didn’t he contact Michael Shermer or James Randi for a simple recommendation? Why didn’t he contact a Magician’s club or cruise the internet? I cannot believe that with just a little effort, Schwartz could not have come up with an experienced cold reader that would have JUMPED at the chance to participate in this. Perhaps he did and they were all afraid of being ridiculed. Perhaps. But an untrained amateur would have been better than nothing. In fact, that would have made an excellent secondary control.
If this seems too nit-picky, imagine if something as simple as the Pepsi challenge was run this way.

Announcer: We gave 20 people Pepsi in an unmarked bottle and they all liked Pepsi better than Coke!

Dubious person: Was the Coke in an unmarked bottle too?

Announcer: There was no Coke.

Dubious person: No Coke?

Announcer: No. Just Pepsi.

Dubious person: How do you know they liked it better?

Announcer: They said so.

Dubious person: But that doesn’t make sense!

Announcer: Go away son, you bother me.


Would you be convinced?


Since Dr. Schwartz is always inviting of suggestions on how to improve his experiments, here is my idea. The experiment should include at least two cold-readers. One that the sitter would come to know about, and one that they wouldn’t. The two cold-readers would be kept in the dark about any prior information concerning the sitter. Neither the sitter, the other mediums, nor the attending experimenters would know that they are fakes.
Our first fake, let’s call him Jim, would give a cold-reading to the best of his ability. After this reading, the information he provided would be given to the second fake to learn, let’s call him Bob. Bob would give the same basic information to the sitter that Jim did, but in a different order and using his own words. When the time came for scoring, the sitter would not be informed of Jim’s true nature and would be asked to score as normal. Prior to scoring Bob’s reading, the sitter would be told that Bob was a fake but be encouraged to score as honestly as possible.
This method would provide much greater control over several areas, and answer two very important questions. One, and the most important “Can an admitted cold reader in a controlled environment provide the same degree of accurate information that a medium can?” Two, “Will a sitter score a known fake (Bob) lower than an unknown fake (Jim) even though they provided the same information?"
There are several different variants on these suggested controls that could be employed over a series of trials, any of which would provide better fidelity of reporting than in “The Afterlife Experiments”. Until this very serious deficiency is addressed, Dr. Schwartz’s research, though fascinating and compelling, is not science.

In part II, if there is interest, I will examine the individual experiments in detail and provide a few opinions on Schwartz’s general approach to investigation and the scientific method.

dogwood

Dammit! I spent four hours on this! Somebody at least make the effort to tell me it sucks!

Bunda

It sucks.

No, seriously it looks like you did a good job. I could not give a good critique of your review before I read the book myself.

I do want to read the book after reading your review. That must mean something.

adamcz

I'll review your review, but only if you review my review of your review.

dogwood

Fair enough.