It often seems that nearly everyone is on the search for a biomarker, whether it is for Alzheimer's disease, diabetes, certain types of cancer, or schizophrenia. The search, however, isn't always easy and can be fraught with challenges. At the end of this past May, Genome Technology invited Cornell University's Fabien Campagne and the University of Medicine and Dentistry of New Jersey's Scott Diehl to come to the GT office to speak with Ciara Curtin about the difficulties facing biomarker research. What follows are excerpts of that conversation.
Genome Technology: When you search in PubMed for 'biomarkers' you get 14,000 results for papers. That seems to be a lot of discovery. Where do you find the bottleneck?
Scott Diehl: What [do] we mean by the word 'biomarker'? I was thinking back to the 1990s when I was working at the NIH, it had a very different meaning than it does today. It's become a very vague term. My recollection, historically, was that biomarkers were considered proteins, non-DNA markers, in the terminology of the community I was working in.
Fabien Campagne: A biomarker is anything that you can assay in some biological material that can inform about a property — sometimes called the endpoint — which you are trying to predict, or should know about the patient, evolution of the disease, or response to treatment.
SD: That was the other part of it. Biological tissue can be used for anything, for diagnosis, for pre-symptomatic diagnosis of disease. At some point, I think the term becomes so broad that it loses its precision and becomes useless.
FC: It can be imaging as well. Some groups use PET scans to try and predict which patients will develop more aggressive forms of lung cancer. From my point of view as somebody who works in bioinformatics, a biomarker is an element of data that can be used to infer something about an individual patient.
An NIH working group described a biomarker in 2001, also in a very general way, as "a characteristic that is objectively measured and evaluated as an indicator of normal biological processes, pathogenic processes, or pharmacologic responses to a therapeutic intervention."
SD: I think that to try to talk in terms of potential use, the upside as well as the downside of using biomarkers, we need to be a little more precise. I'm not sure what your original question was.
GT: It was, where do you think the biggest problems and bottlenecks in biomarker research are?
FC: I would say validation because it really brings you down to the main problem in this field, which is lack of well-defined patient cohorts where you can actually validate. In a way, discovery is easier, because you just need one well-powered dataset. Adequate validation calls for independent samples. And I put the 's' on the end because you need multiple ones. This really is the bottleneck in the field at the moment because features as potential biomarkers, we can see an incredible number of them in any sample.
SD: I definitely agree. I guess I am going to narrow the focus of my answer to what I do, which is genetic studies, specifically genome-wide association studies. First of all, the NIH has taken a very controversial policy initiative to mandate data-sharing, so all NIH-funded studies now need to be — after some negotiated amount of time — deposited into the dbGAP database with the clinical phenotypes.
My point was that this was a good thing but it's also fraught with difficulty, controversy, for example, today's announcement of the president's initiatives on computer security — on the front page of the New York Times. Now some of those datasets are already out in 60 laboratories around the world and even though they don't have names attached with them, there are concerns about compromising subjects' privacy with these incredibly potentially very powerful sets of information.
I think that there may be a more fundamental biological limitation on biological markers. We're seeing that with the rather sobering recognition of the limitation of these large genome-wide studies. [When] I entered the field of human genetics, I was a postdoc in Francis Collins' lab, it was a very exciting era. There was that first wave of excitement when the low-hanging fruit were picked. Then [we] went after more difficult things, psychiatric disorders and more complex diseases, and the linkage studies failed. There was a sort of regrouping, 'What are we going to do?' And the next generation was taking off with the association approaches and now these are being complemented by biological markers, chemical markers, mass spec assays, all these multiple sources of data and a great excitement, some initial great successes at this point.
I certainly think some have been critical and have gone too far. The reality is you have these conditions, very common disorders — type 2 diabetes, cardiovascular disease — where huge studies with very high-quality clinical assessments and huge replicated samples [have been done]. We're getting these biomarkers, DNA biomarkers by and large, that have huge, absolutely certain validity in explaining 1 percent of the variation.
Even if we were to take, as a thought experiment, the next leap and say 'well, on the genetics front let's sequence everybody and get every nucleotide. Let's go and do every mass spec assay to get every secondary structure of every protein.' How much is predictable?''
FC: I think you're making a very important point, which is you will only be as successful as the breadth at which you capture information in these patients. If you are looking at DNA, current studies typically measure a certain sample of SNPs in each patient. Many more polymorphisms and mutations could have been assayed in the same samples. You limit your view of the world in order to make the problem tractable with current technology. However, the question is: did you capture enough in your assay in order to find something that's informative about the disease, because if you didn't measure what was informative, there's not much that you can do in terms of analysis but recognize there was no signal in the data.
SD: I love this technology. I'm confident that it has lots of potential for vast increases in our understanding of basic biological processes as well as understanding disease and developing therapies. There's no question about that. But I am also concerned about the overselling of it. Having lived through two cyles and now going into the third cycle of boom-and-bust, you start to say, 'Wait a minute. This looks kind of familiar. This looks kind of like when the linkage studies were taking off and all of human genetics would be solved in 10 years and people were worried [about] what were they going to be doing with the rest of their careers.'
I've also worked in Ireland many years ago on schizophrenia. Today, huge genome-wide linkage and association studies have come up with very little — not nothing, but very, very little — and now some recent studies of de novo copy number mutations are suggesting that a lot of the disease may be caused by novel mutation with many of these being copy number changes. The disease, it still has a heritability of 70 percent, but it's not five genes of modest effect, it's 500 genes popping up all over the place. Now, if that etiology underlies that disease — and I think probably will underlie other common diseases — even with huge datasets and then optimizing and improving our statistical programs, the ultimate ability to do something meaningful clinically will be a much, much greater challenge than at least if we have a few big player genes that we can discover among the causative variants.
FC: You say 500 genes, but it is possible that you have 50 in a tale of two pathways that are interrelated and involved in signaling cross-talks. I think a major challenge for bioinformatics and investigators who try to understand complex diseases at a mechanistic level is to try to use high-throughput data that you can measure in many patients to try to get at those pathways or at the biological mechanism that actually connects those mutations.
SD: I agree.
FC: We're just not very good at this at the moment, but this is one of the grand challenges, definitely.
SD: What do you think about patenting? I do some studies of susceptibility to different kinds of pain [and] I was doing some background searching the other day as I was trying to help my pain scientist colleagues explain what the field of genetics is all about and I ran across a patent application on one of the candidate genes for pain.
I was just shocked at the breadth of the claim. In my opinion, the fundamental, basic association with susceptibility to pain for this gene, for any variants in this gene, has not even definitively been demonstrated. I think there is suggestive evidence, but some new studies have come in negative and yet the patent office may have given this group, based on the initial publishable but only marginally publishable finding, may have given a lock on every conceivable diagnostic and therapeutic use of the gene. How the heck as a society are we going to be able to move forward when we can't just license that one gene, we've got to license the whole genome and all the carbohydrate markers and the secondary processing of the proteins when we've got this web of patents blocking productive use?
FC: It's a difficult topic. I tend to agree with you, but I also acknowledge the cost of going forward after the initial discovery and up to the point where you can have a diagnostic test. This cost is tremendous. In order to support this activity, you need IP. It's very difficult and I don't have a magic solution to that. Not sure if anybody does.
Perhaps to go back to your first question about proliferation of studies and papers about biomarkers: I am mostly interested in the multivariate studies where people consider multiple biomarkers together as a statistical or machine-learning model to predict a clinical endpoint. I would say that from the point of view of the literature, the reporting of those studies, I feel that there is a lack of documentation about the procedures and also the model that is being produced. Sometimes you read papers and the paper indicates cross-validation was used to derive performance estimates, to name just one technical approach, but the paper does not specify which precise variant of cross-validation was used. There are many flavors of cross-validation, evaluations and so on. Sometimes you wish investigators were a bit more precise in reporting their results than they are at the moment.
I think we could, as a field, gain from standardizing a little bit the way we develop models, the way we validate them, the way we report about the model and evaluation study so that it is easier for the reviewers, but also for other investigators who read the paper after it is published to see what has been done, to evaluate the strengths and weaknesses of the approaches and to get a better understanding of what is going on.
What I'd love to see in those papers is a supplementary electronic archive that contains one file which entirely contains the model being published, and that tells you how you can use software X to read this file in order to make predictions on new samples, for instance to validate the model on an independent dataset. For most articles being published, we are still far from that. At the moment, people tell you, 'Oh, well you know you can get this R code from this website and if you use it like we did' — but they don't tell you how precisely — 'then you get our results.' But there's no guarantee.
GT: Do you think there should be something like the MIAME or MIAPE-type standards for this?
FC: Yes. There should be something like that. Minimal information required to report a multivariate biomarker model. Yes, absolutely.
SD: I agree.
FC: Obviously the FDA has barriers in place to try to make sure that you cannot use models in the clinic that are not reliable enough, but it's a lot of wasted effort to actually just realize by validating a model in another patient cohort that it didn't work and probably there was a bug somewhere in the software that was used to produce that model. You'd like to fix those as early as possible so that you focus on testing the models that are more likely to validate. There are enough challenges in this field without adding software bugs on top of that.