As the US Food & Drug Administration refines its policies regarding in vitro diagnostic multivariate index assays, attention to assay reproducibility issues and selection of appropriate intended use populations will be key to the success of proteomics-based tests, FDA staff fellow Joshua Levin suggested at a talk this week discussing the agency's 2009 review of two mock 510(k) submissions for protein IVDMIAs.
Speaking at the National Institutes of Health's Proteomic Interest Group Fellows Forum, Levin, who is part of the FDA's Office of In Vitro Diagnostic Device Evaluation and Safety in the Center for Devices and Radiological Health, offered the agency's take on work done by scientists from the National Cancer Institute's Clinical Proteomics Technology Assessment for Cancer program to publish a pair of mock submission guides designed to help translational proteomics researchers submit protein-based IVDMIAs for regulatory approval (PM 1/15/10).
The two mock submissions were described in a study published Dec. 10, 2009, in the online edition of Clinical Chemistry and covered two platforms — a multiplex immunoaffinity mass-spec platform for protein quantification dubbed PepCa10; and an immunological array platform for glycoprotein isoform quantification called SDIA. The PepCa10 efforts were led by Plasma Proteome Institute CEO Leigh Anderson, and the SDIA project was led by Purdue University analytical chemistry professor Fred Regnier.
In Levin's discussion of the submissions, assay reproducibility emerged as a significant FDA concern, a focus driven by the fact that platforms like mass spectrometry and protein microarrays — used in the PepCA10 and SDIA work, respectively — typically demonstrate higher coefficients of variation than ELISA-based methods.
In particular, Levin said, CVs for mass spec sample prep are notoriously high — an issue that translational proteomics continues to struggle with as it works to move multiplexed MS-based tests into the clinic (PM 10/22/10).
High CVs were also a problem with the mock data from the microarray-based SDIA submission, he said. More broadly, FDA has "seen issues with microarrays and reproducibility" in several of the initial protein IVDMIA submissions it has received, Levin noted. The agency requires that 510(k) applicants show new tests are "substantially equivalent" to other tests already on the market, and a number of companies have argued that their microarray assays are the equivalent of already existing ELISAs. The microarrays' higher CVs, however, call that comparison into question, he suggested.
"We have companies that are coming in with microarray assays and saying, 'We want to compare this to an ELISA.' But their reproducibility variance is much higher, and that's going to make us say, 'Well, this ELISA is in clinical practice, but it has CVs of 5 percent and the microarray has [CVs] of 15 to 20 percent,' so we need to help the company think through that and what that means in terms of patient care," he said.
Importantly, Levin noted, the exact level of variation the FDA might find acceptable depends on the limit of detection at which it occurs and how close that limit is to the test's clinical cutoff.
"If you have a limit of detection that's way below where your clinical cutoff is, and you have a CV of 20 percent down there, we're not going to be too concerned," he said. "But if you have a CV of 20 percent at the clinical cutoff, that's going to make us worry."
Variability is especially a concern in the case of quantitative assays where a patient's results are given a numerical score, Levin said. With qualitative assays that just report a "yes-no decision, we can be a little more flexible about the reproducibility," he said. However, he added, even for qualitative assays FDA is "still going to want reproducibility in and around the [clinical] cutoff – that's going to be very important."
Quantitative tests reporting a numerical score will also typically draw more scrutiny with regard to their linear range, Levin said.
"If it's a quantitative test we want to know what the linear range is, that the [clinical] cutoff is going to fall in that linear range. If you're saying, this person is a 15 and this person is a 20, that's much different from saying, this person is low risk and this person is high risk," he said, citing as an example the numerical score provided by Genomic Health's gene-based breast cancer test Oncotype DX, which FDA has not reviewed.
"If we were to review Oncotype DX, we would have greater concerns about their use of a linear recurrence score than if they were to come in and just say we have a low [risk score] and a high [risk score]."
Levin also provided thoughts on how the tests' intended clinical uses might affect their regulatory paths. In particular, he noted that the SDIA assay, which aims to assess whether a mammogram subject should undergo a biopsy based on the levels of eight glycoproteins in the blood, would be considered a high-risk Class III device, and as such would have to go through the pre-market approval process as opposed to the less stringent 510(k) process.
"What [the SDIA] test is doing is saying, 'We're going to take this subset of patients and say they don't need biopsies,'" he said. "Well, if you're wrong, you've missed a cancer. So we thought this is a high-risk test that would probably require pre-market approval."
This concept of intended use "is key to how FDA reviews these tests," Levin said. "You might have the same technology, the same test, but depending on how it's used this might be a high-risk test [Class III, requiring PMA] or a medium-risk test [Class II, requiring just 510(k) approval]."
"If you had a multiplex proteomic breast cancer assay, this might be a high-risk test or a medium-risk test," he added. "If it's used for screening of asymptomatic patients, that's considered a high-risk indication because you're taking people and telling them you have cancer or don't have cancer, and if you're wrong that's a big deal. But if you take already diagnosed patients and you're just using the test to monitor the patients, that's considered moderate risk."
It's a distinction with significant financial implications, Levin noted. While a 510(k) application fee costs $4,348 — with companies with less than $100 million in gross revenue eligible for a small business rate of $2,174 — the application fee for a PMA is $236,298, with a small business rate of $59,075. This is in addition to the cost of the clinical trials needed to meet the higher standards of the PMA process.
These expenses put PMAs outside the range of many small companies, he said, noting that "there are some user fee negotiations going on right now between FDA and industry," that could make the process more accessible to smaller firms.
Levin also briefly addressed the issue of laboratory-developed tests, which the FDA has said it plans to begin regulating after years of taking a hands-off approach (PGx Reporter 7/21/2010). The agency will take a "risk-based approach" to their regulation, Levin said, noting that "we're going to say that the highest-risk [LDTs] that are out there should undergo FDA scrutiny."
FDA is currently developing a guidance document on these tests that it expects to release sometime this spring, he said.
As for the regulatory status of IVDMIAs, the FDA released a draft guidance in 2006 outlining its intent to regulate these assays (PM 9/14/2006), but has yet to issue a final guidance. More recently, the agency has signaled that it intends to address IVDMIAs as part of its broader LDT guidance (PGx Reporter 6/23/2010).
Have topics you'd like to see covered in ProteoMonitor? Contact the editor at abonislawski [at] genomeweb [.] com.