How do you evaluate evidence? How you ever really thought about it? Advertisers certainly do, they use the way people think to help them convince you to buy their latest product. Did you know that advertisers like tv shows with paranormal psychic or alternative medicine type content because the audiences for these shows tend to be gullible audiences and the ads are more effective if placed around such a show?
Here are a few general principles to help you to evaluate evidence - whether this is the latest anti-ageing cream, alien abductions or the latest “in thing” in corporate land to that unsolved mystery spot on the local news. The world is full of people trying to trick you into things here are some tools to help you keep out of their clutches.
It can be great fun unpicking these things - give it a try.
How to evaluate evidence
There are six simple rules to follow when considering any claim. We will look at these six rules and give a few examples from my pet subjects of pseudoscience, the paranormal and alternative medicine.
These six rules of evidential reasoning are a simplification of the scientific method. Good science is simply not trying to be tricked into the wrong answer by the evidence.
“Hang on a minute though when did science come into this?” I hear you thinking. Well, this is all science is, really - looking at the evidence - trying not to get tricked by it and coming up with a conclusion. All the math and complicated language comes in because the world is such a complicated place. So don’t let any preconceived ideas of what you think of when you think “science” stop you here - these are general rules to stop you being fooled in life.
The rules are;
Falsifiability or Contingency
Logic
Comprehensiveness
Honesty
Replicability
Sufficiency
Falsifiability or contingency
The first question to ask or rule to follow is whether or not evidence, logic and facts actually apply to the matter at hand.
Some claims are impossible to prove as false - they are immune to any possible evidence that might turn up in the future. e.g. Such a claim might be “There is an invisible old man who is always behind you and is completely undetectable by any sense or any kind of instrument or tool.” There is no way of spotting this chap at all so there is no way of proving he doesn’t exist the claim is not contingent upon and evidence. It is not falsifiable.
In other words the claim must be contingent upon some form of evidence or it is not a claim that evidence can ever shed any light upon.
This a general rule about any claim being made. It must be possible to conceive of evidence that would prove the claim false. (Karl Popper first suggested this in the 1930’s)
It may sound paradoxical, but in order for any claim to be true, it must be falsifiable. The rule of falsifiability is a guarantee that if the claim is false, the evidence will prove it false; and if the claim is true, the evidence will not disprove it (in which case the claim can be tentatively accepted as true until such time as evidence is brought forth that does disprove it). The rule of falsifiability, in short, says that the evidence must matter, and as such it is the first and most important and most fundamental rule of evidential reasoning.
This does not mean, however, that an unfalsifiable claim is true; instead it would mean that the claim is meaningless. This is so because it is impossible -- logically impossible -- for any claim to be true no matter what. For every true claim, you can always conceive of evidence that would make the claim untrue -- in other words, again, every true claim is falsifiable.
Unfalsifiable claims do communicate information, but what they describe is the claimant's value orientation. In the example above of the invisible man we would probably think anyone making such a claim was delusional. However claims of this type communicate nothing whatsoever of a factual nature, and hence are neither true nor false. They are instead making an emotive statement, a declaration of the way the claimant feels about the world.
There are two main types unfalsifiable or non-contingent claims.
The first is the undeclared claim: a statement that is so broad or vague that it lacks any propositional content. The undeclared claim is basically unintelligible and consequently meaningless.
Consider, for example, the claim that crystal therapists can use pieces of quartz to restore balance and harmony to a person's spiritual energy. What does it mean to have unbalanced spiritual energy? How is the condition recognised and diagnosed? What evidence would prove that someone's unbalanced spiritual energy had been - or had not been - balanced by the application of crystal therapy? Most New Age wonders, in fact, consist of similarly undeclared claims that dissolve completely when exposed to the solvent of rationality.
The undeclared claim has the advantage that virtually any evidence that could be produced could be interpreted as congruent with the claim, and for that reason it is especially popular among Astrological predictions.
Here is a classic exposure of Astrology by James Randi;
This demonstrates that astrologers use the way the human brain works to provide you with “prophecies” which are simultaneously very accurate for you whilst at the same time are perceived as very accurate by other individuals. In fact those shown in the video seem to be very accurate to the vast majority of people in the room.
The second variety of unfalsifiable statements, which is even more popular in the paranormal industry, involves the use of the “multiple out”, that is, an inexhaustible series of excuses intended to explain away the evidence that would seem to falsify the claim.
Creationists, for example, claim that the universe is no more than 10,000 years old. They do so despite the fact that we can observe stars that are billions of light-years from the earth, which means that the light must have left those stars billions of years ago, and which proves that the universe must be billions of years old. How then do the creationists respond to this falsification of their claim? By suggesting that God must have created the light already on the way from those distant star at the moment of creation 10,000 years ago. No conceivable piece of evidence, of course, could disprove that claim.
Additional examples of multiple outs abound in the realm of the paranormal. UFO proponents, faced with a lack of reliable physical or photographic evidence to buttress the claims, point to a secret "government conspiracy" that is allegedly preventing the release of evidence that would support their case.
Psychic healers say they can heal you if you have enough faith in their psychic powers.
Psycho-kinetics say they can bend spoons with their minds if they are not exposed to negative vibrations from sceptic observers.
Tarot readers can predict your fate if you're sincere in your desire for knowledge.
The multiple out means, in effect, "Heads I win, tails you lose."
Logic
Any argument offered as evidence in support of any claim must be sound or logically “valid”.
An argument is said to be "valid" if its conclusion follows unavoidably from its premises; it is "sound" if it is valid and if all the premises are true.
An invalid argument can be recognised by the simple method of counterexample: If you can conceive of a single imaginable instance whereby the conclusion would not necessarily follow from the premises even if the premises were true, then the argument is invalid.
Consider the following example: All dogs have fleas; Xavier has fleas; therefore Xavier is a dog. That argument is invalid because a single flea-ridden feline named Xavier would provide an effective counterexample.
If an argument is invalid, then it is, by definition, unsound. But not all valid arguments are sound.
Consider this example: All dogs have fleas; Xavier is a dog; therefore Xavier has fleas. That argument is unsound, even though it is valid, because the first premise is false: All dogs do not have fleas.
To determine whether a valid argument is sound is difficult; knowing whether a given premise is true or false often demands additional knowledge about the claim that may require empirical investigation. In other words you might have to look into it.
If the argument passes these two tests, however -- if it is both valid and sound -- then the conclusion can be embraced with certainty.
I have covered the most commonly seen logical fallacies on my “The Guide to Thinking Straight”.
The rule of logic is frequently violated by pseudo-scientists.
Erich von Däniken, who single-handedly popularised the ancient-astronaut mythology in the 1970s, wrote many books in which he offered invalid and unsound arguments with benumbing regularity. In Chariots of the Gods? he was not above making arguments that were both logically invalid and factually inaccurate -- in other words, arguments that were doubly unsound.
For example, von Däniken argues that the map of the world made by the sixteenth-century Turkish admiral Piri Re'is is so "astoundingly accurate" that it could only have been made from satellite photographs.
Not only is the argument invalid (any number of imaginable techniques other than satellite photography could result in an "astoundingly accurate" map), but the premise is simply wrong - the Piri Re'is map, in fact, contains many gross inaccuracies.
Comprehensiveness
The evidence offered in support of any claim must be exhaustive - that is all of the available evidence must be considered.
For obvious reasons, it is never reasonable to consider only the evidence that supports a theory and to discard the evidence that contradicts it. This rule is straightforward and self-apparent, and it requires little explication or justification. Nevertheless, it is a rule that is frequently broken by the paranormal industry and the alternative medicine industry.
For example, the proponents of biorhythm theory are fond of pointing to airplane crashes that occurred on days when the pilot, co-pilot, anchor navigator were experiencing critically low points in their intellectual, emotional, and/or physical cycles. The evidence considered by the biorhythm apologists, however, does not include the even larger number of airplane crashes that occurred when the crews were experiencing high or neutral points in their biorhythm cycles.
If you are willing to be selective in the evidence you consider, you could reasonably conclude that the earth is flat.
Honesty
The evidence offered in support of any claim must be evaluated without self-deception.
The rule of honesty goes hand in hand with the rule of comprehensiveness. When you have examined all of the evidence, it is essential that you be honest with yourself about the results of that examination. If the weight of the evidence contradicts the claim, then you are required to abandon belief in that claim.
The rule of honesty, like the rule of comprehensiveness, is frequently violated by the paranormal industry. Parapsychologists violate this rule when they conclude, after numerous subsequent experiments have failed to replicate initially positive psi results, that psi must be an elusive phenomenon. (Applying Occam's Razor, the more honest conclusion would be that the original positive result must have been a coincidence.)
Believers in the paranormal violate this rule when they conclude, after observing a "psychic" surreptitiously bend a spoon with his hands, that he only cheats sometimes.
The rule of honesty means that you must accept the obligation to come to a rational conclusion once you have examined all the evidence. If the overwhelming weight of all the evidence falsifies your belief, then you must conclude that the belief is false, and you must face the implications of that conclusion forthrightly.
In the face of overwhelmingly negative evidence, neutrality and agnosticism are no better than credulity and faith. Denial, avoidance, rationalisation, and all the other familiar mechanisms of self-deception would constitute violations of the rule of honesty.
Veteran psychic investigator Eric Dingwall summed up his extensive experience in parapsychological research with this observation: "After sixty years' experience and personal acquaintance with most of the leading parapsychologists of that period I do not think I could name a half dozen whom I could call objective student who honestly wished to discover the truth."
Replicability
If the evidence for any claim is based upon an experimental result, or if the evidence offered in support of any claim could logically be explained as coincidental, then it is necessary for the evidence to be repeated in subsequent experiments or trials.
If I correctly predict the next roll of the dice, you should demand that I duplicate the feat before granting that my prediction was anything but a coincidence.
The rule of replicability is regularly violated by parapsychologists, who are especially fond of misinterpreting coincidences. The famous "psychic sleuth” Gerard Croiset, for example, allegedly solved numerous baffling crimes and located hundreds of missing persons in a career that spanned five decades, from the 1940s until his death in 1980. The truth is that the overwhelming majority of Croiset's predictions were either vague and unfalsifiable or simply wrong. Given the fact that Croiset made thousands of predictions during his lifetime, it is hardly surprising that he enjoyed one or two chance "hits." The late Dutch parapsychologist Wilhelm Tenhaeff, however, seized upon those "very few prize cases" to argue that Croiset possessed demonstrated psi powers. That was a clear violation of the rule of replicability, and could not have been taken as evidence of Croiset's psi abilities even if the "few prize cases" had been true. (In fact, however, much of Tenhaeff's data was fraudulent)
Sufficiency
The evidence offered in support of any claim must be adequate to establish the truth of that claim, with these stipulations:
1.the burden of proof for any claim rests on the claimant,
2.extraordinary claims demand extraordinary evidence, and
3.evidence based upon authority and/or testimony is always inadequate for any paranormal claim
The burden of proof always rests with the claimant for the simple reason that the absence of disconfirming evidence is not the same as the presence of confirming evidence. This rule is frequently violated by proponents of paranormal claims, who argue that, because their claims have not been disproved, they have therefore been proved. (UFO buffs, for example, argue that because sceptics have not explained every UFO sighting, some UFO sightings must be extraterrestrial spacecraft.)
Consider the implications of that kind of reasoning: If I claim that Adolf Hitler is alive and well and living in Argentina, how could you disprove my claim? Since the claim is logically possible, the best you could do (in the absence of unambiguous forensic evidence) is to show that the claim is highly improbable -- but that would not disprove it. The fact that you cannot prove that Hitler is not living in Argentina, however, does not mean that I have proved that he is. It only means that I have proved that he could be -- but that would mean very little; logical possibility is not the same as established reality.
If the absence of disconfirming evidence were sufficient proof of a claim, then we could "prove" anything that we could imagine. Belief must be based not simply on the absence of disconfirming evidence but on the presence of confirming evidence. It is the claimant's obligation to furnish that confirming evidence.
Extraordinary claims demand extraordinary evidence for the obvious reason of balance. If I claim that it rained for ten minutes on my way to work last Tuesday, you would be justified in accepting that claim as true on the basis of my report. But if I claim that I was abducted by extraterrestrial aliens who whisked me to the far side of the moon and performed bizarre medical experiments on me, you would be justified in demanding more substantial evidence. The ordinary evidence of my testimony, while sufficient for ordinary claims, is not sufficient for extraordinary ones.
In fact, testimony or anecdotal evidence is always inadequate for any paranormal claim, whether it is offered by an authority or a layperson, for the simple reason that a human being can lie or make a mistake. No amount of expertise in any field is a guarantee against human fallibility, and expertise does not preclude the motivation to lie; therefore a person's credentials, knowledge and experience cannot, in themselves be taken as sufficient evidence to establish the truth of a claim.
Moreover, a person's sincerity lends nothing to the credibility of his or her testimony. Even if people are telling what they sincerely believe to be the truth, it is always possible that they could be mistaken. Perception is a selective act, dependent upon belief context, expectation, emotional and biochemical states, and a host of other variables. Memory is notoriously problematic, prone to a range of distortions, deletions, substitutions and amplifications. Therefore the testimony that people offer of what they remember seeing or hearing should always be regarded as only provisionally and approximately accurate; when people are speaking about the paranormal, their testimony should never be regarded as reliable evidence in and of itself. The possibility and even the likelihood of error are far too extensive.
Conclusion
The first three rules - falsifiability, logic, and comprehensiveness -- are all logically necessary rules of evidential reasoning. If we are to have confidence in the veracity of any claim whether normal or paranormal, the claim must be meaningful, and the evidence offered in support of the claim must be rational and exhaustive.
The last three rules - honesty, replicability, and sufficiency -- are all pragmatically necessary rules of evidential reasoning. Because human beings are often motivated to rationalise and to lie to themselves, because they are sometimes motivated to lie to others, because they can make mistakes, and because perception and memory are problematic, we must demand that the evidence for any factual claim be evaluated without self-deception, that it be carefully screened for error, fraud, and appropriateness, and that it be substantial and unequivocal.
Passing all six tests, of course, does not guarantee that the claim is true (just because you have examined all the evidence available today is no guarantee that there will not be new evidence available tomorrow), but it does guarantee that you have good reasons for believing the claim.
Being a responsible adult means accepting the fact that almost all knowledge is tentative, and accepting it cheerfully. You may be required to change your belief tomorrow, if the evidence warrants, and you should be willing and able to do so.
That, in essence, is what scepticism means: to believe if and only if the evidence warrants.
The material on this page has been stuck together from various sources but is based mainly upon the article “A Field Guide To Critical Thinking” by James Lett - I am grateful for his permission to use it.
He suggested the following sites as excellent further reading material and I can also recommend them;
Think Critical
