Take Me Heme, Binding Modes: Wrapping up the CYP Inhibition Challenge

Authors: Maria Castellanos, Jon Swain and Hugo MacDermott-Opeskin
DOI: 10.5281/zenodo.23985044

Cytochrome P450 (CYP) enzymes are the human body's primary workhorses for drug metabolism, making CYP inhibition a critical liability to model when predicting potential drug-drug interactions. To benchmark modern computational tools, OpenADMET generated high-throughput, high-fidelity inhibition data for 6,899 compounds across 4 CYPs.

Participants predicted activities across four crucial CYP isoforms that metabolize the vast majority of marketed drugs: CYP3A4, CYP2D6, CYP2C9, and CYP1A2. The challenge was divided across three different tracks:

• Direct Inhibition Track: A regression task to predict direct inhibition pIC50
• Time-Dependent Inhibition (TDI) Track: A classification task to identify compounds that exhibit time-dependent inhibition.
• Structure Prediction Track: A pose-prediction task evaluated against 20 novel CYP3A4 Cryo-EM complex structures.

The community turned out in full force for this challenge. With over 6,000 submissions across the three tracks, we saw a total of 271 participants in Direct Inhibition, 155 in TDI, and 50 in the Structure track. We also just fully unblinded the entire dataset; now available on Hugging Face. Whether you participated or not, you can now access the complete set of inhibition data and structures to run your own post hoc analyses and train your models.

Now, without further ado, let’s look at the final leaderboards!

We’re crowning the Heme-coming King and Queen

We are excited to announce the winners of the CYP Inhibition Blind Challenge! Below, you will find the top 20 entries for each of the three tracks. Full interactive leaderboard tables are now available on the challenge Hugging Face space. Entries are grouped into tiers, with one tier containing all consecutive entries that were not statistically distinct from each other, and the next tier starting with the first entry found to be statistically distinct.

Note that these final leaderboards differ from their live versions available through the challenge. First, the ranking below reflects evaluation on the entire test set, while the live leaderboard evaluates only half of the test set; as a result, some entries may have shifted up or down compared to the live rankings. Second, we filtered the leaderboard to include only entries with a valid model report and a valid Hugging Face username, as explained in the challenge rules. Third, we removed teams suspected of cheating by submitting from multiple accounts (against the challenge rules), and that provided no valid justification for their actions.

How much did that reshuffle things? For the two activity tracks, remarkably little. Scoring the full test set instead of half moves the typical entry up one place in Direct Inhibition, up four in TDI, and keeps nine of the ten entries that led the live board in the final top ten. The Structure track saw the largest shifts, with only half of the live top ten remaining at the top, which is expected since the live board was scored on ten protein-ligand complexes only.

Direct inhibition leaderboard

Our regression task has 103 entries on the final leaderboard, spanning 29 performance tiers. Congratulations to whippoorwill and preheat-to-450, who make up Tier 1. You get to rock out as the stars of Heme-coming.

TDI leaderboard

Our classification task had 62 entries on the final leaderboard, spanning 5 performance tiers. Congratulations to rasayan-labs, nova, and NV-deCYPher, who make up Tier 1.

A classification task contains much less information than a regression task (only one bit per compound, compared to a wide range of possible values per compound for a regression task). In addition, the CYP2D6 ground truth had only a small number of positives. This made overfitting to the live leaderboard more likely. Some teams tried new techniques, checked whether this changed performance on the live leaderboard, and reversed output classes if performance fell. Even keeping half the dataset hidden wasn’t enough to penalize this, since the final leaderboard included the compounds used for the live leaderboard.

The table below illustrates the overfitting. All top teams showed a noticeable drop-off in performance between their final leaderboard MA-MCC and an MA-MCC calculated post hoc on only the hidden compounds. This was a limitation of our challenge design. Since this was our first classification challenge, we are still learning the best way to score these. The published leaderboard is valid, as these teams performed best under the defined rules, but for future challenges we will use only the hidden compounds for the final leaderboard, not the entire test set. An extra congratulations to SC38, jeremy, and wikke69, who were our top three when scoring only on the hidden compounds!

Structure leaderboard

Our Structure task has 38 entries on the final leaderboard, spanning 4 performance tiers. With a much smaller dataset than the regression or classification tasks, it was harder to show statistically significant differences between entries, meaning the tiers spanned a wide range of entries on the final leaderboards.

How did we do?

We chose a "tiering" system to rank the entries on the leaderboard, which used a paired bootstrap method with the Benjamini-Hochberg procedure to control the false discovery rate to group entries that are not significantly distinct from each other, allowing us to accurately present the differences between entries at the top of the leaderboard. The three tracks had different tier distributions (Figure 1): Direct Inhibition was more finely resolved, with 103 entries split into 29 tiers and the top 5 spanning only 3 tiers. With 750 compounds scored across four isoforms, the bootstrap uncertainty on a single entry is only about ±0.01 ST-RAE, so even small gaps near the top resulted in real differences. In contrast, TDI performance was more homogeneous, with 62 entries split into five tiers, and teams ranked 4 to 57 sitting in just two tiers. The structure track was an even closer match, with 38 entries divided between four tiers, and Tier 1 holding the first 13 entries. Because the structure dataset spanned only 20 complexes, each entry has an error bar of about ±0.04 LDDT-PLI, almost as wide as the 0.07 separating the best entry (0.48) from the 13th (0.42).

One notable observation in the final performance was that some isoforms were significantly harder to model than others. While 101 of the 103 final Direct Inhibition entries beat a naive mean prediction baseline (i.e., predicting the mean of the test set for all compounds, ST-RAE < 1), CYP2D6 proved to be a challenging target, with a median ST-RAE of 0.71. On the other hand, CYP2C9 and CYP3A4 were the most tractable (median ST-RAE 0.44 and 0.46), and CYP1A2 sat in the middle (0.52). In terms of ranking, the difference is even more stark, with the other three isoforms having a median Spearman correlation of 0.80-0.85, and CYP2D6 scoring 0.44. As we discussed in our interim leaderboard release, CYP2D6 was harder to predict because the train and test sets had completely different distributions due to the way we designed the test set (CYP2D6 Dose Response Curves were not available for expansion). As a result, the CYP2D6 test set was dominated by inactives (65% of compounds having a pIC50 below 4), compared to a more evenly distributed training set (9% below pIC50 4). As a result, models trained on this higher-potency training set systematically overpredicted the test set, and the entries that performed well on CYP2D6 were largely those that recalibrated downward. The TDI track showed a similar pattern. CYP3A4 TDI was reasonably learnable (median MCC 0.44, with the best at 0.55), while CYP2D6 TDI was harder (median 0.22, best 0.49). The culprit was class imbalance: only 44 of the 748 test compounds (5.9%) were CYP2D6 time-dependent inhibitors, compared with 21.6% in the training data, leaving very little positive signal.

The Structure track proved the hardest of the three. The best entry reached an LDDT-PLI of 0.48 (median 0.41), and only 8.9% of all submitted poses were within 2 Å BiSyRMSD of the experimental ligand position. Interestingly, with a median LDDT-PLI of 0.82, participants overwhelmingly found the right binding site but placed the ligand incorrectly within it. CYP3A4's notably large and flexible active site makes finding the correct pose challenging.

Head-to-head comparisons

We have also added a head-to-head tab to the challenge’s Hugging Face space. You can pick any two entries and compare them side by side, now that the full ground truth is public. It is the quickest way to see whether a rank difference comes from a handful of outliers or from a consistent edge across the test set.

The CYP inhibition challenge had great participation across the three tracks

Y’all worked incredibly hard! The challenge ran from mid-August to the 3rd of November and collected 6,034 submissions in total: 3,741 from Direct Inhibition, 1,695 from TDI, and 598 in the Structure track. Together, this totaled a whopping 13.8 million individual predictions! Submissions came from 650 accounts, of which only 291 were eligible participants; the rest were repeated “ghost” accounts from teams that didn’t follow the challenge rules.

Figure 3 shows submission rates climbed steadily through September, jumped around the interim leaderboard, and then increased faster through the final week, with 207 submissions (eligible for the final leaderboard) received on the last day alone. Most participants submitted multiple times, with a median of 3 valid submissions in Direct Inhibition, 2 in TDI, and 4 in Structure, while the most persistent of you submitted 116 times across the challenge.

Plenty of you didn’t listen to us about not leaving your final submissions to the last minute. We received 271 submissions on the last day, 40 in the final hour, and one participant even submitted 4 seconds before the deadline (!!).

As always, these challenges wouldn't happen without you, our participants. Thank you for taking part, especially the 17 hard-working teams who appear on all three final leaderboards!

What’s next

The challenge may be officially over, but the fun is just getting started! Here is what you can look forward to in the coming weeks:

• Unblinded dataset available now: The complete, unblinded dataset across all three tracks is officially live on Hugging Face!

• Follow-up analysis & post-mortems: We’re diving deeper into the challenge results to extract key takeaways and performance trends across all three tracks, so stay tuned!

• Webinars with the top participants: We'll host live webinars featuring the top-performing participants and teams, where you can hear directly about their modeling pipelines and lessons learned.

• ML Innovation Award: In this challenge, we are also selecting a few submissions for the ML Innovation Award to highlight the competition's most creative approaches. The winners will be announced soon!

• New OpenADMET Challenges: We’re far from done benchmarking ADMET prediction. Keep an eye out for the next challenge–it could launch as soon as next month!

Join our Discord channel and subscribe to our newsletter so you don't miss upcoming announcements and dataset releases!

Acknowledgements

We would like to thank our funders for their support of OpenADMET, in particular ARPA-H, Radial (part of the Astera Institute (https://ror.org/00ydx1s47)), Schrödinger Inc., the Gates Foundation and the OpenAI Foundation. We would also like to thank our partners Enamine, Hugging Face, OpenEye, CDD Vault, Discovery Life Sciences, and the beamline staff at NSLS-II for their support.

This work is supported by the Advanced Research Projects Agency for Health (ARPA-H) under AVOID-OME, and Award Number 1AY1AX000035. The contents are those of the authors. They may not reflect the policies of the Department of Health and Human Services or the U.S. government. The content is solely the responsibility of the authors and does not necessarily represent the official views of the Advanced Research Projects Agency for Health.