By Turna Ray
Online services that match patients to ongoing personalized drug trials based on the molecular characteristics of their disease may bias drug/diagnostic codevelopment studies and make the treatment and test appear more efficacious than they actually are, a US Food and Drug Administration biostatistics expert recently cautioned.
According to Thomas Gwise, deputy director of the FDA's biometrics division, online tools such as the Targeted Therapy Finder for Melanoma released by software firm CollabRx are "basically advertising for prescreening" for drug/diagnostic codevelopment trials.
This can pose a problem, he said, because prescreening patients for clinical trial enrollment based on the results of previously conducted laboratory tests with varying specificity and sensitivity may cause investigators to overestimate the performance of investigational personalized medicine products.
CollabRx's Targeted Therapy Finder is a free online tool that asks patients to provide information such as disease stage, primary site of disease origin, metastases sites, and genomic mutations associated with the disease or treatment response. Based on these data, the application generates a list of drugs and clinical trials that may be appropriate for the patient and advises them to discuss these options with their doctor. CollabRx has developed Targeted Therapy Finder applications for melanoma, lung cancer, and colorectal cancer. The company believes that by connecting physicians, researchers, drug developers, and patients in an online forum it can inspire new models for personalized drug development that improve patient outcomes and reduce drug failure rates for industry.
Although such services promise to be a useful tool for patients, the agency's evaluation criteria for personalized drugs is still evolving, so the emergence of such services may pose a statistical headache from a regulatory standpoint.
Gwise said that he happened upon the CollabRx service while surfing the web a few days before the FDA was about to approve Roche/Plexxikon's personalized melanoma drug Zelboraf and the companion BRAF-mutation test for picking out best responders to the treatment. After inputting some staging information, he received a list of FDA-approved and investigational drugs, which included Zelboraf, as well as information about clinical trials that fit the patient characteristics he had provided for the search.
"This sort of stopped me," Gwise said last week at a conference on individualized therapeutics hosted by the FDA and the Drug Information Association. Since it appeared likely that the CollabRx tool was providing information about drug trials recruiting BRAF-mutated melanoma patients while the pivotal study for Zelboraf was ongoing, Gwise suggested that such services may confound clinical trials involving drug/test combination products with various forms of bias.
"This could be very good for patients," Gwise acknowledged. "On the other hand, it could cause some headaches for the evaluation of the diagnostic test, and maybe even cause some bias in the estimation of the performance of the therapy."
Bias resulting from the way patients are recruited to clinical trials has long plagued drug development investigations, and with the growing prevalence of genetic testing it is unlikely to go away. The FDA is cognizant of this reality. Gwise admitted that many times these types of biases are outside the sponsors' and FDA's control, but urged industry to try to adjust patient recruitment procedures to account for the possibility of prescreening.
"One thing we can't let slip out of view is that we're using this test to define subgroup, and that's the subgroup for whom we believe the treatment will work," he noted. "So, the fuzzier the test is, the fuzzier that subgroup is, and that could also possibly impact the performance estimate of the drug, of the treatment effect."
As genetic testing becomes more accessible and affordable to the general public through the growing popularity of direct-to-consumer testing services such as 23andMe, more and more people will have knowledge of their genetic makeup before volunteering for clinical trials. In response to this trend, online tools such as CollabRx's Targeted Therapy Finder are cropping up as a resource for patients and doctors who are interested in trying out emerging personalized medicine options.
However, drug developers may not be so keen to engage in these new collaborative paradigms if the FDA frowns upon enrolling patients with knowledge of their mutation status. Some drug and diagnostics firms who have successfully taken an Rx/Dx product through the FDA are already on alert for such sources of bias.
Speaking at the same meeting as Gwise, Karen Long, division VP of medical, clinical, and regulatory affairs at Abbott Laboratories, said that the company watches "very carefully" for clinical trial sites that may be screening for patients that fit a specific profile. Abbott Molecular recently gained approval for a companion test that gauges which non-small cell lung cancer patients with ALK mutations will respond to Pfizer's drug Xalkori.
"We know this happens and we really try [to] prevent prescreening of patients coming into a … study for [drug/diagnostic] codevelopment," Long said. "We know when a site is prescreening. We can tell by looking at the database and our pharma partner contacts that investigator and instructs them to please cease the prescreening of these patients." She acknowledged that prescreening for patients who carry the marker of interest in a personalized medicine trial "could impact not only the statistics, but also the project in general."
PGx Reporter contacted Roche to ask whether prescreening was a problem in clinical trials for its recently approved melanoma drug Zelboraf, indicated for patients with BRAF mutations, but company officials were unavailable for comment.
Although regulation doesn't change as quickly as technology, the FDA is working to address the issues raised by molecularly targeted drug development. The agency has announced plans to align the activities of its drug and diagnostics divisions, is encouraging prospective sample requisition from participants of all drug trials, and is crafting guidance documents on the regulatory and study design challenges of developing drugs intended for a small patient subset.
Gwise's comments regarding Targeted Therapy Finder suggest that FDA regulators are thinking about how traditional drug development paradigms fit within larger cultural shifts in the way people make healthcare decisions. This is exactly what the developers of Targeted Therapy Finder wanted.
"It's great that the FDA recognizes and is getting ahead of this issue, [but] this is an issue that doesn't have to do just with CollabRx," Smruti Vidwans, the company's chief scientist, told PGx Reporter. "This is part of a much bigger trend in which healthcare is moving toward a personalized medicine approach. The practice of medicine is changing as we speak, and that means clinical trial execution is going to be a part of it."
An Unavoidable Problem
According to Gwise, spectrum bias, self-selection bias, and sampling bias are some of the possible ways that trials for drug/diagnostic combination products can be confounded if patients are volunteering because they know they harbor the molecular marker being investigated.
Gwise said that prescreening patients can introduce spectrum bias in drug/diagnostic codevelopment trials due to the variability in the lab tests used to test patients ahead of the trial. "Differences in operator performance, differences in devices, differences specimen handling and preparation — you can think of all sorts of sources of variability that go into the different laboratories," he explained.
He used the example of an imaging test to illustrate the problem. Under this scenario, a researcher may pick up a "spectrum" of tumors of varying sizes, but naturally the big tumors will be easier to see than the really small ones. However, a small tumor that the imaging test doesn't identify may be just as significant as the large one that the test did find.
In the same way, in a drug trial where the patients are stratified with the help of an investigational molecular diagnostic, prescreening of those patients with another test increases the likelihood that the investigational test will pick up the obvious marker-positive and -negative patients. As a result, the sensitivity and specificity of the investigational test "is going to go way up and you're going to have a really good looking test," Gwise said.
Self-selection bias, which is inherently an issue in clinical trials since participants have a choice as to whether to ultimately participate in the study or not, may be further exacerbated by prior genetic testing and could impact the treatment effect in an Rx/Dx codevelopment trial, Gwise said.
When patients decide to participate in a study based on their known marker status, "the prevalence of marker-positive [patients] will be inflated," Gwise pointed out. "This should be a tip-off if the only people coming in to be tested for the clinical trial are marker positives," and that should signal to study investigators "that something is going on."
[ pagebreak ]
Conversely, prescreening can also discourage patients from participating in a trial if they don't carry the marker of interest, Gwise pointed out. This could hinder investigators from recruiting the necessary number of marker-positive and marker-negative patients, which is the study design that the FDA ideally prefers when validating an association between a molecular marker and treatment response.
The third kind of bias that can arise as a result of prescreening is sampling bias, which occurs when investigators aren't testing patients they want to target for a trial. In this scenario, "what very well could happen is some false negatives may be going by the wayside and we won't be able to look at them with the test, and this will bias the estimates of performance for the marketed test," Gwise said.
To account for prescreening in personalized medicine trials, he recommended that drug and test developers survey potential participants to find out if they've been previously tested, try to monitor for the prevalence of marker-positive and marker-negative patients, and explore the potential impact of such biases in their studies.
"I would suggest that [drug and test] developers think about this when they're planning the clinical trial," Gwise said, acknowledging that often, "there is not much you can do about this."
CollabRx's Vidwans viewed as positive the FDA's efforts to raise awareness of the statistical issues that will be increasingly problematic in personalized drug development, noting that pharma sponsors "should definitely take into consideration" Gwise's recommendations in order to reduce these types of bias.
Need for New Paradigms
As drug and diagnostic firms develop more individualized treatments, many have recognized that established clinical trial models are often unfeasible from an ethical standpoint or prohibitive in terms of patient recruitment given the rarity of many of the molecular marker being investigated. As such, industry is increasingly urging FDA to come up with new clinical trial models that are better suited to personalized drug development.
Targeted Therapy Finder was developed as part of a larger effort, called Cancer Commons, that aims to bring researchers, industry, and patients together around the common goal of advancing personalized oncology treatments. CollabRx CEO and cancer survivor Jay Tenenbaum launched Cancer Commons with a vision to move oncology drug development toward more open and collaborative research models, and away from the rigid randomized-controlled trials that are generally considered the gold standard by the FDA and the broader research community.
As a melanoma patient, Tenenbaum was enrolled in a clinical trial for a drug that failed late-stage studies but from which he was benefiting. This experience led him to realize that there are some patients that respond to drugs that don't show benefit in a large study population and that there needs to be a system for identifying these patient subsets. Through the Targeted Therapy Finder tool CollabRx is hoping to advance new drug development paradigms by spurring small, proof-of-concept trials for individualized treatments (PGx Reporter 3/30/2011).
CollabRx's effort to match patients to treatments and clinical trials is part of a larger patient-centric movement in healthcare research, where study participants are increasingly demanding more control over their data and how that information is used. "Using web-based tools, what you're doing is you're kind of democratizing what used to be cutting-edge research and cutting-edge tools. So, someone at a small community center who may not have access to this information, may now have access," Vidwans said.
Leading the consumer-driven research movement is DTC genetic testing company 23andMe. The Google-backed firm is challenging standard clinical trial paradigms by pursing a research model that is essentially funded by its customers, who pay 23andMe to analyze their genes for disease risk and drug response associations. The company mines the genomic data of its more than 100,000 customers to conduct studies to validate genotype-phenotype associations.
The phenotypic information used in these studies is self-reported by customers. Although self-reported data has its flaws, 23andMe has said that because customers have paid for the genetic testing service, they are much more likely to complete surveys about their traits and disease characteristics than patients randomly recruited in a traditional research setting.
Last year, the company published its first genome-wide association findings in PLoS Genetics. In that study, involving more than 9,000 customers, researchers from 23andMe and Columbia University were able to verify previously reported associations for five traits and identified new SNPs linked to four of the traits, including hair curl, freckling, sneezing in response to light, and the ability to detect asparagus metabolite odors in urine. Using a similar approach, 23andMe this year also reported data from a gene-association study on Parkinson's disease, and is investigating gene associations for other diseases.
While self-selection bias is undoubtedly an issue in this type of study design, 23andMe's consumer-driven model has allowed it to rapidly populate large studies that otherwise would have taken many years to recruit and been prohibitively expensive to conduct.
Meanwhile, early adopters of personalized medicine strategies, such as the pharmacy-benefit manager Medco, are conducting pharmacogenetic investigations that challenge conventional research models and shake up established ideas about how new interventions are adopted into the healthcare system. For example, in a Medco-sponsored study evaluating whether PGx testing to administer the anticoagulant warfarin improved patient outcomes over standard monitoring methods, the study investigators did not randomize the study population or blind physicians to which patients were genetically tested.
The lack of blinding was viewed by many in the research community as a study design flaw, since the benefit of genetic testing may have been positively bolstered by the so-called Hawthorne effect — a scenario in which study subjects improve behavior simply in response to being studied and not as a result of the intervention.
In response to critics, Robert Epstein, Medco's chief medical officer, said that because the adoption of genetic testing requires physicians to change their behavior, the fact that doctors in the Mayo/Medco study may have followed their patients more closely because of genetic testing shouldn't undervalue the clinical utility of the intervention.
"If, in fact, receiving genotyping information helps the physician pay closer attention to the patient, like make them do more INRs because it points out the person has a rare genotype and needs to be tracked closer, I don't think that's a bad thing. I think that's a good thing," Epstein said at a major medical conference last year, defending against a peer-review challenge (PGx Reporter 3/17/2010).
Vidwans noted that CollabRx is interested in working with drug developers and the FDA to figure out new drug development paradigms that are better than traditional study designs at establishing the safety and efficacy of molecularly targeted drugs. Despite Gwise's reservations about clinical trial prescreening through online services such as Targeted Therapy Finder, Vidwans said the company hasn't heard any criticism about the tool from the agency or drug developers.
Targeted Therapy Finder has the support of professional societies. The molecular disease models that underlie the tool are mined from sources such as the American Society of Clinical Oncology's database of cancer drugs trials. ASCO, under a data-sharing agreement with CollabRx, has made Targeted Therapy Finder available to its members and through affiliated patient information sites (PGx Reporter 4/27/2011).
"Patient empowerment through the web is here to stay," Vidwans said. "We're all going to have to work together to figure out how to leverage this and not just think of it as a problem. There are other opportunities that could help reduce the bias and facilitate recruitment of patients that never would have been part of the equation."
Have topics you'd like to see covered in Pharmacogenomics Reporter? Contact the editor at tray [at] genomeweb [.] com.