CHICAGO (GenomeWeb) – The recent update to the Broad Institute's Connectivity Map has given hope to computational biologists that they soon will be able to solve the vexing problem of detecting drug-induced liver injury in preclinical studies. For now, though, they are still relying on the previous iteration of CMap, which has done little to advance DILI detection, even with machine learning technologies.
This problem was on display here this week at the annual Intelligent Systems for Molecular Biology (ISMB) conference of the International Society for Computational Biology. One of the ISMB tracks featured the 17th annual International Conference on Critical Assessment of Massive Data Analysis, or CAMDA, which issues data analysis challenges to the scientific community each year.
For 2018, challenges included figuring out a computational method to predict drug-induced liver injury by integrating or comparing data from multiple cell lines from the CMap database. CAMDA received seven entries in the form of article abstracts for this, most of which relied on machine learning. None performed particularly strongly, though.
Witold Rudnicki, a machine learning expert at the University of Bialystok's Institute of Informatics and Computational Center in Poland, said that his team "obtained weakly predictive models." Though he reported that the models were "robust" and worked across cell lines, his method produced results "no better than random guess."
Traditional testing from pathology and other clinical chemistry reports are able to predict about 60 percent of DILI cases in the preclinical studies, according to another entrant, University of South Dakota data analyst Zhixiu Lu.
"In general, we try to find very hard questions," explained challenge coordinator David Kreil, chair of bioinformatics at the University of Natural Resources and Life Sciences, in Vienna, known by its German acronym, BOKU. While CAMDA has been running such contests for years, this is among the first to introduce large-scale, multi-omics problems, he said.
"It's still a question that's highly open," Kreil said. "We don't go into questions where there's a fixed answer," simply looking to get scores a little bit higher than previous studies.
"We tell people, 'Here's a dataset that's challenging, that's hard, and we don't know how to solve it. Now go and do your work," he explained.
CAMDA is also trying to get researchers to be creative.
"There's a consequence to that," Kreil noted. "Very often, the best presentation is not the one that gets the highest score on some metric, because it's very hard to classify how you're doing unless there are very clear prediction cases."
He said the CAMDA audience does not always need to see successes. "We also tell stories about what failed and why it failed and how it failed, and we think we're closing a gap here that is not really covered by other parts of the conference," Kreil explained.
Rex Sumison, a student in bioinformatics and statistics at Brigham Young University, is among those who struggled. "We were having a hard time reaching above baseline accuracy" in creating classification algorithms, he said.
The same goes for Margherita Francescatto, a postdoctoral researcher in computational biology at Fondazione Bruno Kessler, a research institute in Trento, Italy. She explored whether deep learning could help predict DILI from CMap gene expression profiles using the MCF7 and PC3 cancer cell lines and Affymetrix GeneChip arrays.
Nothing worked particularly well. "We are not learning [of] a compound likely to make DILI in humans," Francescatto said during her presentation. "The test set remains completely difficult to understand."
A major problem, she later told GenomeWeb, was that the CMap dataset was inadequate.
"With deep learning, in general, you need to train a lot of parameters," Francescatto noted. "If you have only 180 samples, you don't really have enough to train all these parameters."
Participants in the CAMDA CMap challenge also had to opt into a meta-analysis being conducted by the US Food and Drug Administration. "This is a small portion of a big study we are designing at FDA," said Shraddha Thakkar, a staff scientist at the FDA's National Center for Toxicological Research.
Another CAMDA leader, Harvard computational biologist Wenzhong Xiao, noted that the FDA study will move to the next-generation CMap in 2019, so researchers will have 20 cell lines and more than 1,000 drugs with DILI annotations to work with. "We hope that we will have much better outcomes next year," Xiao said.
This year, challenge participants had just two cell lines and 276 drugs annotated with DILI labels at their disposal. Both cell lines performed poorly for DILI-negative drugs, Thakkar said.
The research will evolve next year into a program called isDILI, a community-wide crowdsourcing effort to assess in silico DILI prediction, according to Thakkar.