Forensic science was not developed by scientists. It was mostly created by cops, who were guided by little more than common sense. And as hundreds of criminal cases begin to unravel, many established forensic practices are coming under fire. PM takes an in-depth look at the shaky science that has put innocent people behind bars
.
Read more: CSI Myths: The Shaky Science Behind Forensics - Popular Mechanics
.
On Jan. 11, 1992, the jury in the murder trial of Roy Brown heard from a dentist named Edward Mofson. To establish his credentials, Dr. Mofson testified that he was certified in forensic odontology, belonged to six related professional organizations and did forensic consulting throughout New York state. He then explained that several months earlier he was called to the morgue in Cayuga County, New York, to analyze the body of 49-year-old Sabina Kulakowski.
Kulakowski's corpse was found by a volunteer firefighter on a dirt road some 300 yards from the farmhouse where she lived, which had burned to the ground in the night. She was severely beaten and stabbed, and there were multiple bite marks on her body. Brown was a natural suspect in the grisly murder. The week before the crime, the hard-drinking 31-year-old had been released from jail on charges of threatening to "wipe everybody out" at the social services office where Kulakowski worked; the agency had put his daughter into foster care. In addition to the motive, the district attorney at trial produced other circumstantial evidence, including testimony from Brown's two ex-wives that he had bitten them. But Mofson, now deceased, was the centerpiece of the prosecution.
Mofson testified that seven bite marks found on Kulakowski were "entirely consistent" with dental impressions taken from Brown. It was the only physical evidence tying Brown to the crime. Although a defense expert disputed Mofson's findings, the jury convicted Brown of second-degree murder. He was sentenced to 25 years to life in prison.
As the years ticked by, few listened as Brown proclaimed his innocence from his cell in the Elmira Correctional Facility. Then Brown got an unusual lucky break. His stepfather's house burned down, taking with it all of his records from the trial. To replace his documents, Brown submitted an open records request to the county. The sheriff who processed Brown's request mistakenly sent him the entire investigative file. It revealed another suspect: Barry Bench, the firefighter who discovered Kulakowski's body. Bench's brother had dated Kulakowski up until two months before the murder and Bench was reportedly upset that she continued to live in the family farmhouse. On the day before Christmas in 2003, Brown sent a letter to Bench letting him know he was seeking DNA testing. "Juries can make mistakes," he wrote. But, "DNA is God's creation, and God makes no mistakes." Soon after receiving the message, Bench committed suicide by jumping in front of an Amtrak train. DNA tests confirmed that Bench was guilty of Kulakowski's murder, and Brown was set free.
The faulty identification that sent Brown to prison for 15 years may seem like a rare glitch in the U.S. criminal justice system. It wasn't. As DNA testing has made it possible to re-examine biological evidence from past trials, more than 200 people have had their convictions overturned. In approximately 50 percent of those cases, bad forensic analysis contributed to their imprisonment.
On television and in the movies, forensic examiners unravel difficult cases with a combination of scientific acumen, cutting-edge technology and dogged persistence. The gee-whiz wonder of it all has spawned its own media-age legal phenomenon known as the "CSI effect." Jurors routinely afford confident scientific experts an almost mythic infallibility because they evoke the bold characters from crime dramas. The real world of forensic science, however, is far different. America's forensic labs are overburdened, understaffed and under intense pressure from prosecutors to produce results. According to a 2005 study by the Department of Justice, the average lab has a backlog of 401 requests for services. Plus, several state and city forensic departments have been racked by scandals involving mishandled evidence and outright fraud.
But criminal forensics has a deeper problem of basic validity. Bite marks, blood-splatter patterns, ballistics, and hair, fiber and handwriting analysis sound compelling in the courtroom, but much of the "science" behind forensic science rests on surprisingly shaky foundations. Many well-established forms of evidence are the product of highly subjective analysis by people with minimal credentials—according to the American Society of Crime Laboratory Directors, no advanced degree is required for a career in forensics. And even the most experienced and respected professionals can come to inaccurate conclusions, because the body of research behind the majority of the forensic sciences is incomplete, and the established methodologies are often inexact. "There is no scientific foundation for it," says Arizona State University law professor Michael Saks. "As you begin to unpack it you find it's a lot of loosey-goosey stuff."
Not surprisingly, a movement to reform the way forensics is done in the U.S. is gaining momentum. The call for change has been fueled by some embarrassing failures, even at the highest levels of law enforcement. After the 2004 train bombings in Madrid, Spain, the FBI arrested Oregon lawyer Brandon Mayfield and kept him in jail for two weeks. His incarceration was based on a purported fingerprint match to a print found on a bag of detonators discovered near the scene of the crime. As a later investigation by the Justice Department revealed, the FBI's fingerprint-analysis software never actually matched Mayfield to the suspect fingerprint, but produced him as an "unusually close nonmatch." Lacking any statistical context for how rare such similarities are, investigators quickly convinced themselves that Mayfield was the prime suspect.
The next year, 2005, Congress commissioned the National Academy of Sciences (NAS) to examine the state of forensics in U.S. law enforcement. The result was a blistering report that came out this February, noting "serious deficiencies" in the nation's forensic science system and advocating extensive reforms. It specifically noted that apart from DNA, there is not a single forensic discipline that has been proven "with a high degree of certainty" to be able to match a piece of evidence to a suspect. The obvious implication is the sobering possibility that more Roy Browns are currently locked up based on shoddy science. Then there's the flip side: A lot of bad guys who should be in prison still roam free. A study by the Innocence Project of the prisoners exonerated by DNA found that the real perpetrators were identified in 103 cases—roughly half. In all but one, the perpetrator committed at least one serious crime after the innocent person was jailed.
Kulakowski's corpse was found by a volunteer firefighter on a dirt road some 300 yards from the farmhouse where she lived, which had burned to the ground in the night. She was severely beaten and stabbed, and there were multiple bite marks on her body. Brown was a natural suspect in the grisly murder. The week before the crime, the hard-drinking 31-year-old had been released from jail on charges of threatening to "wipe everybody out" at the social services office where Kulakowski worked; the agency had put his daughter into foster care. In addition to the motive, the district attorney at trial produced other circumstantial evidence, including testimony from Brown's two ex-wives that he had bitten them. But Mofson, now deceased, was the centerpiece of the prosecution.
Mofson testified that seven bite marks found on Kulakowski were "entirely consistent" with dental impressions taken from Brown. It was the only physical evidence tying Brown to the crime. Although a defense expert disputed Mofson's findings, the jury convicted Brown of second-degree murder. He was sentenced to 25 years to life in prison.
As the years ticked by, few listened as Brown proclaimed his innocence from his cell in the Elmira Correctional Facility. Then Brown got an unusual lucky break. His stepfather's house burned down, taking with it all of his records from the trial. To replace his documents, Brown submitted an open records request to the county. The sheriff who processed Brown's request mistakenly sent him the entire investigative file. It revealed another suspect: Barry Bench, the firefighter who discovered Kulakowski's body. Bench's brother had dated Kulakowski up until two months before the murder and Bench was reportedly upset that she continued to live in the family farmhouse. On the day before Christmas in 2003, Brown sent a letter to Bench letting him know he was seeking DNA testing. "Juries can make mistakes," he wrote. But, "DNA is God's creation, and God makes no mistakes." Soon after receiving the message, Bench committed suicide by jumping in front of an Amtrak train. DNA tests confirmed that Bench was guilty of Kulakowski's murder, and Brown was set free.
The faulty identification that sent Brown to prison for 15 years may seem like a rare glitch in the U.S. criminal justice system. It wasn't. As DNA testing has made it possible to re-examine biological evidence from past trials, more than 200 people have had their convictions overturned. In approximately 50 percent of those cases, bad forensic analysis contributed to their imprisonment.
On television and in the movies, forensic examiners unravel difficult cases with a combination of scientific acumen, cutting-edge technology and dogged persistence. The gee-whiz wonder of it all has spawned its own media-age legal phenomenon known as the "CSI effect." Jurors routinely afford confident scientific experts an almost mythic infallibility because they evoke the bold characters from crime dramas. The real world of forensic science, however, is far different. America's forensic labs are overburdened, understaffed and under intense pressure from prosecutors to produce results. According to a 2005 study by the Department of Justice, the average lab has a backlog of 401 requests for services. Plus, several state and city forensic departments have been racked by scandals involving mishandled evidence and outright fraud.
But criminal forensics has a deeper problem of basic validity. Bite marks, blood-splatter patterns, ballistics, and hair, fiber and handwriting analysis sound compelling in the courtroom, but much of the "science" behind forensic science rests on surprisingly shaky foundations. Many well-established forms of evidence are the product of highly subjective analysis by people with minimal credentials—according to the American Society of Crime Laboratory Directors, no advanced degree is required for a career in forensics. And even the most experienced and respected professionals can come to inaccurate conclusions, because the body of research behind the majority of the forensic sciences is incomplete, and the established methodologies are often inexact. "There is no scientific foundation for it," says Arizona State University law professor Michael Saks. "As you begin to unpack it you find it's a lot of loosey-goosey stuff."
Not surprisingly, a movement to reform the way forensics is done in the U.S. is gaining momentum. The call for change has been fueled by some embarrassing failures, even at the highest levels of law enforcement. After the 2004 train bombings in Madrid, Spain, the FBI arrested Oregon lawyer Brandon Mayfield and kept him in jail for two weeks. His incarceration was based on a purported fingerprint match to a print found on a bag of detonators discovered near the scene of the crime. As a later investigation by the Justice Department revealed, the FBI's fingerprint-analysis software never actually matched Mayfield to the suspect fingerprint, but produced him as an "unusually close nonmatch." Lacking any statistical context for how rare such similarities are, investigators quickly convinced themselves that Mayfield was the prime suspect.
The next year, 2005, Congress commissioned the National Academy of Sciences (NAS) to examine the state of forensics in U.S. law enforcement. The result was a blistering report that came out this February, noting "serious deficiencies" in the nation's forensic science system and advocating extensive reforms. It specifically noted that apart from DNA, there is not a single forensic discipline that has been proven "with a high degree of certainty" to be able to match a piece of evidence to a suspect. The obvious implication is the sobering possibility that more Roy Browns are currently locked up based on shoddy science. Then there's the flip side: A lot of bad guys who should be in prison still roam free. A study by the Innocence Project of the prisoners exonerated by DNA found that the real perpetrators were identified in 103 cases—roughly half. In all but one, the perpetrator committed at least one serious crime after the innocent person was jailed.
(Photograph by Christopher Griffith; styling by Megan Caponetto)
The scientific method is instrumental to our understanding of the physical world. To scientists, the process is sacrosanct: Research your topic, generate a hypothesis, test the hypothesis, analyze your data and then publish the results for peer review. Forensic science, however, was not developed by scientists. It was created by cops—often guided by little more than common sense—looking for reliable ways to match patterns from clues with evidence tied to suspects. What research has been done understandably focuses on finding new techniques for putting criminals in jail.
In the academic community the legal sciences get a comparative trickle of federal funding. In 2007, the National Institute of Justice awarded 21 grants for forensic research (excluding DNA) totaling $6.6 million; the National Institutes of Health awarded 37,275 grants totaling $15 billion. And without a wealth of statistically defensible research to back up their evidence, forensic examiners generally rely upon their own intuition and the experience of their colleagues. "You can't take a few case studies and say, `Oh, it worked on these people; it must be reliable,'" says Karen Kafadar, an Indiana University statistics professor and a member of the NAS committee. "That is hardly a placebo-controlled, double-blind randomized trial."
The FBI's errors in the Madrid bombing case were particularly surprising because they called into question one of the gold standards of evidence—fingerprints. In recent years, legal experts have become deeply concerned about the accuracy of the "friction ridge analysis" central to fingerprint identification. Fingerprints are believed to be unique, but the process of matching prints has no statistically valid model. And forensic examiners are often working in an imperfect world, where prints taken in a police station on an ink pad are compared to prints left at a crime scene, which may be smudged or partially captured. Yet, as University of California-Los Angeles law professor Jennifer Mnookin has written, "fingerprint examiners typically testify in the language of absolute certainty."
A 2006 study by the University of Southampton in England asked six veteran fingerprint examiners to study prints taken from actual criminal cases. The experts were not told that they had previously examined the same prints. The researchers' goal was to determine if contextual information—for example, some prints included a notation that the suspect had already confessed—would affect the results. But the experiment revealed a far more serious problem: The analyses of fingerprint examiners were often inconsistent regardless of context. Only two of the six experts reached the same conclusions on second examination as they had on the first.
Ballistics has similar flaws. A subsection of tool-mark analysis, ballistics matching is predicated on the theory that when a bullet is fired, unique marks are left on the slug by the barrel of the gun. Consequently, two bullets fired from the same gun should bear the identical marks. Yet there are no accepted standards for what constitutes a match between bullets. Juries are left to trust expert witnesses. "`I know it when I see it' is often an acceptable response," says Adina Schwartz, a law professor and ballistics expert with the John Jay College of Criminal Justice.
In the academic community the legal sciences get a comparative trickle of federal funding. In 2007, the National Institute of Justice awarded 21 grants for forensic research (excluding DNA) totaling $6.6 million; the National Institutes of Health awarded 37,275 grants totaling $15 billion. And without a wealth of statistically defensible research to back up their evidence, forensic examiners generally rely upon their own intuition and the experience of their colleagues. "You can't take a few case studies and say, `Oh, it worked on these people; it must be reliable,'" says Karen Kafadar, an Indiana University statistics professor and a member of the NAS committee. "That is hardly a placebo-controlled, double-blind randomized trial."
The FBI's errors in the Madrid bombing case were particularly surprising because they called into question one of the gold standards of evidence—fingerprints. In recent years, legal experts have become deeply concerned about the accuracy of the "friction ridge analysis" central to fingerprint identification. Fingerprints are believed to be unique, but the process of matching prints has no statistically valid model. And forensic examiners are often working in an imperfect world, where prints taken in a police station on an ink pad are compared to prints left at a crime scene, which may be smudged or partially captured. Yet, as University of California-Los Angeles law professor Jennifer Mnookin has written, "fingerprint examiners typically testify in the language of absolute certainty."
A 2006 study by the University of Southampton in England asked six veteran fingerprint examiners to study prints taken from actual criminal cases. The experts were not told that they had previously examined the same prints. The researchers' goal was to determine if contextual information—for example, some prints included a notation that the suspect had already confessed—would affect the results. But the experiment revealed a far more serious problem: The analyses of fingerprint examiners were often inconsistent regardless of context. Only two of the six experts reached the same conclusions on second examination as they had on the first.
Ballistics has similar flaws. A subsection of tool-mark analysis, ballistics matching is predicated on the theory that when a bullet is fired, unique marks are left on the slug by the barrel of the gun. Consequently, two bullets fired from the same gun should bear the identical marks. Yet there are no accepted standards for what constitutes a match between bullets. Juries are left to trust expert witnesses. "`I know it when I see it' is often an acceptable response," says Adina Schwartz, a law professor and ballistics expert with the John Jay College of Criminal Justice.
Because of misleading forensic evidence, Roy Brown spent nearly a third of his life in jail for a murder he didn't commit. (Photograph by Tyler Hicks)
Not all forensic disciplines are in dispute.
Techniques that grew out of organic chemistry and microbiology have a strong scientific foundation. For example, chromatography, a method for separating complex mixtures, enables examiners to identify chemical substances in bodily fluids—evidence vital to many drug cases. The evolution of DNA analysis, in particular, has set a new scientific standard for forensic evidence. But it also demonstrates that good science takes time.
The double-helix structure of DNA was discovered in the 1950s, but it wasn't until 30 years later that sample analysis became sophisticated enough for positive ID. In 1987, a serial rapist by the name of Tommie Lee Andrews was the first person convicted in the U.S. using DNA. Nevertheless, for several years scientists continued to research and debate what constitutes a satisfactory match. The resulting process is broadly accepted and quantifiable (when using the most advanced analysis, there is a one in more than a quadrillion chance of a random match of two strangers' nuclear DNA).
But DNA constitutes less than 10 percent of the case load at U.S. crime labs. The goal going forward, everyone agrees, is to make the rest of forensics more rigorous and statistically grounded. Promising work is already being done: Sargur Srihari, a pattern-recognition expert with the State University of New York at Buffalo, is developing software to help quantify the certainty of fingerprint matches. And, Nicholas Petraco, a chemist and mathematician at John Jay, is working on a database of microscopic tool marks to give statistical significance to the identification of burglars' tools.
The NAS report recommends the establishment of an independent entity—a National Institute of Forensic Science—which would be the central authority responsible for funding research as well as creating and promulgating the standards of evidence and certification for experts. If such a system worked properly, juries would only hear from experts who are certified in their fields and examiners who work in accredited laboratories.
It's likely that the microscope of serious scientific scrutiny will turn disciplines such as fingerprint and ballistics analysis, which have long histories and large sample sizes, into stronger standards of evidence. But many other forensic disciplines may be classified as far less sound. Bite marks, footprints, tire tracks, handwriting, bloodstain patterns and other forms of analysis that suffer from multiple confounding variables could end up being used as exclusionary evidence or as qualified supporting evidence only. Some types of evidence may be completely discredited. That's what happened with voiceprint analysis and lead analysis of bullets, which were popular forensic techniques until studies showed significant error rates.
Within the forensic community, the reaction to the mounting criticism is mixed. Some are offended and blame the "propaganda" of defense attorneys and the snobbery of academics. Dean Gialamas, president of the American Society of Crime Laboratory Directors, says most techniques have "a strong foundation in science" even if they have not been subject to the type of applied research needed to satisfy critics. And he notes that his organization has long advocated more standardization and stronger ethics rules, so hired guns can't pollute courtrooms with biased testimony. At the end of the day, Gialamas and most other forensic experts say they are confident their methods will ultimately be validated by further research. Even critics of the current system say forensics should remain a critical part of law enforcement. "Let's just give it to people as completely and honestly as we possibly can," Saks says.
It will take years to fully reconcile the rigors of the scientific method with the needs and processes of the judicial system. But in the meantime, questionable forensic science will continue to tip the scales of justice. And when bad decisions are made in the courtroom, an innocent person's entire life can be swept right out from under him. It happened to Steven Barnes 20 years ago. Then 23 years old, he was brought to trial for the rape and murder of a 16-year-old girl. He had never been arrested before and was confident he'd be cleared. Yet he watched as forensics expert Elaine Pagliaro testified that two hairs found in Barnes's pickup were microscopically similar to the victim's. Pagliaro also noted that soil samples taken from the truck were consistent with dirt from the crime scene and even that a distinctive pattern from the victim's jeans was similar to an imprint left on the truck.
Due largely to her testimony, Barnes was sentenced to 25 years to life in prison. Last year, he was cleared by DNA and released. He'd never been on the Internet or used a cellular phone, and his girlfriend, who initially stuck by him after he went to prison, had long ago married another man. Barnes told Popular Mechanics that he works hard not to be overwhelmed by bitterness, even toward the jurors. "They must have thought, `[Pagliaro] knows what she is talking about.'"
Pagliaro, a veteran analyst with the Connecticut State Police, has recently co-authored a book called The Real World of a Forensic Scientist. "I think this scrutiny is actually good," she says. "It's important for the public to have a realistic expectation of what the science can do." As for the Barnes case, there is no suggestion of impropriety regarding her testimony, but none of the evidence she presented was based on statistically validated science. "You feel awful someone spent all that time in jail," she says. "All you can do is look back and say, `Was that the best we could do?'"
Techniques that grew out of organic chemistry and microbiology have a strong scientific foundation. For example, chromatography, a method for separating complex mixtures, enables examiners to identify chemical substances in bodily fluids—evidence vital to many drug cases. The evolution of DNA analysis, in particular, has set a new scientific standard for forensic evidence. But it also demonstrates that good science takes time.
The double-helix structure of DNA was discovered in the 1950s, but it wasn't until 30 years later that sample analysis became sophisticated enough for positive ID. In 1987, a serial rapist by the name of Tommie Lee Andrews was the first person convicted in the U.S. using DNA. Nevertheless, for several years scientists continued to research and debate what constitutes a satisfactory match. The resulting process is broadly accepted and quantifiable (when using the most advanced analysis, there is a one in more than a quadrillion chance of a random match of two strangers' nuclear DNA).
But DNA constitutes less than 10 percent of the case load at U.S. crime labs. The goal going forward, everyone agrees, is to make the rest of forensics more rigorous and statistically grounded. Promising work is already being done: Sargur Srihari, a pattern-recognition expert with the State University of New York at Buffalo, is developing software to help quantify the certainty of fingerprint matches. And, Nicholas Petraco, a chemist and mathematician at John Jay, is working on a database of microscopic tool marks to give statistical significance to the identification of burglars' tools.
The NAS report recommends the establishment of an independent entity—a National Institute of Forensic Science—which would be the central authority responsible for funding research as well as creating and promulgating the standards of evidence and certification for experts. If such a system worked properly, juries would only hear from experts who are certified in their fields and examiners who work in accredited laboratories.
It's likely that the microscope of serious scientific scrutiny will turn disciplines such as fingerprint and ballistics analysis, which have long histories and large sample sizes, into stronger standards of evidence. But many other forensic disciplines may be classified as far less sound. Bite marks, footprints, tire tracks, handwriting, bloodstain patterns and other forms of analysis that suffer from multiple confounding variables could end up being used as exclusionary evidence or as qualified supporting evidence only. Some types of evidence may be completely discredited. That's what happened with voiceprint analysis and lead analysis of bullets, which were popular forensic techniques until studies showed significant error rates.
Within the forensic community, the reaction to the mounting criticism is mixed. Some are offended and blame the "propaganda" of defense attorneys and the snobbery of academics. Dean Gialamas, president of the American Society of Crime Laboratory Directors, says most techniques have "a strong foundation in science" even if they have not been subject to the type of applied research needed to satisfy critics. And he notes that his organization has long advocated more standardization and stronger ethics rules, so hired guns can't pollute courtrooms with biased testimony. At the end of the day, Gialamas and most other forensic experts say they are confident their methods will ultimately be validated by further research. Even critics of the current system say forensics should remain a critical part of law enforcement. "Let's just give it to people as completely and honestly as we possibly can," Saks says.
It will take years to fully reconcile the rigors of the scientific method with the needs and processes of the judicial system. But in the meantime, questionable forensic science will continue to tip the scales of justice. And when bad decisions are made in the courtroom, an innocent person's entire life can be swept right out from under him. It happened to Steven Barnes 20 years ago. Then 23 years old, he was brought to trial for the rape and murder of a 16-year-old girl. He had never been arrested before and was confident he'd be cleared. Yet he watched as forensics expert Elaine Pagliaro testified that two hairs found in Barnes's pickup were microscopically similar to the victim's. Pagliaro also noted that soil samples taken from the truck were consistent with dirt from the crime scene and even that a distinctive pattern from the victim's jeans was similar to an imprint left on the truck.
Due largely to her testimony, Barnes was sentenced to 25 years to life in prison. Last year, he was cleared by DNA and released. He'd never been on the Internet or used a cellular phone, and his girlfriend, who initially stuck by him after he went to prison, had long ago married another man. Barnes told Popular Mechanics that he works hard not to be overwhelmed by bitterness, even toward the jurors. "They must have thought, `[Pagliaro] knows what she is talking about.'"
Pagliaro, a veteran analyst with the Connecticut State Police, has recently co-authored a book called The Real World of a Forensic Scientist. "I think this scrutiny is actually good," she says. "It's important for the public to have a realistic expectation of what the science can do." As for the Barnes case, there is no suggestion of impropriety regarding her testimony, but none of the evidence she presented was based on statistically validated science. "You feel awful someone spent all that time in jail," she says. "All you can do is look back and say, `Was that the best we could do?'"
Read more: CSI Myths: The Shaky Science Behind Forensics - Popular Mechanics
