Open-sourcing our blind challenge platform

Authors: Jon Swain and Maria Castellanos
DOI: https://doi.org/10.5281/zenodo.22761182

At OpenADMET, we're in the business of running blind challenges: time-limited competitions where teams predict properties of molecules they've never seen, scored against experimental data they can’t know until the challenge is over. Over the last year, we've built a fair amount of infrastructure to make those challenges reproducible, tamper-resistant, and easy to spin up. Today we're releasing it for the community to use.

OpenADMET/blind-challenge-template is a template repository (including a complete scoring backend, the infrastructure to run it, and a participant-facing web app) that you can copy to launch your own blind challenge in a single afternoon. This is the same code we use in production, completely open source under an Apache 2.0 license.

We want to thank the team at Radial (part of the Astera Institute) in particular for their support of our work to make blind challenges easier, more open, and more accessible.

Why a blind challenge needs its own infrastructure

The central point of a blind challenge is evaluation integrity. If participants can see the test labels, the results mean nothing. A blind challenge is only as credible as the machinery behind it. That machinery has a few jobs that are easy to underestimate:

• Keep the ground truth genuinely hidden. Test labels, per-compound uncertainty, and the phase in which each compound is unblinded all have to live somewhere a participant can never reach. These are deliberately kept separate from the frontend that the user interacts with, significantly reducing the chance of accidental unblinding or leakage.
• Score every submission the same way, automatically. No human in the loop deciding what counts or reformatting submissions. A submission lands, triggering validation and scoring, and a score is generated. The exact same pipeline generates every score on the leaderboard.
• Say whether a difference is real. "Model A beat Model B by 0.02" is not a useful result unless you can attach a confidence to it. That means implementing bootstrapped metrics and robust pairwise significance testing, not just point estimates.
• Be affordable and reusable. A challenge runs for a few months, and then it's over. The infrastructure should cost close to nothing at rest and tear down cleanly.

We rebuilt a variation of this for each challenge before finally extracting and generalizing it as a template. Importantly, it is very easy to adapt this infrastructure to variations on our challenge design with modern agentic coding.

The architecture

The most critical design decisions can be broken down into four components.

1. The frontend and backend are completely decoupled

Participants interact with a Gradio app hosted on Hugging Face Spaces, where they can read the challenge rules and FAQs, upload predictions, and view the live leaderboard. That app has narrowly scoped, write-mostly credentials: it can drop a submission into an S3 prefix and read the published leaderboards, and that's essentially it.

Important components (the ground-truth datasets, the scoring code, the raw per-compound scores) live in an isolated, private AWS account that the frontend has no general access to. Scoring runs in Lambda, triggered by the submission landing in S3. The web app never sees a test label because it never has permission to.

While Hugging Face gives us a familiar, zero-maintenance home for the UI that the ML community already knows how to use, AWS gives us isolation. Keeping them apart means a bug or a leak in the public-facing app can't compromise the evaluation.

2. Every challenge gets its own scoped identity, via OIDC

There are no long-lived AWS keys in GitHub. The deploy pipeline authenticates with GitHub's OIDC provider and assumes an IAM role whose trust policy names exactly one repository and whose permissions are scoped to exactly that challenge's resources: its bucket, its Lambdas, its secrets; nothing else.

This means a mistake in one repo's pipeline cannot touch another's data or delete another's stack. The impact of any single credential leak is isolated to one challenge.

3. Infrastructure is code, with per-challenge state

All AWS resources are defined in OpenTofu (the open-source fork of Terraform). State lives in a shared S3 bucket with a DynamoDB lock table, but each challenge writes to its own state file. This design ensures that concurrent apply’s from different challenges can't corrupt each other, and tearing one challenge down leaves the others untouched.

4. One variable names everything

AWS resource names are derived from a single OpenTofu variable, challenge_name. There's no second "environment" or "stage" dimension to keep in sync. Set challenge_name = "kinase-challenge" and you get kinase-challenge, kinase-challenge-regression-evaluator, kinase-challenge-hf-space, and so on, consistently, everywhere. This means a new challenge gets minted through a simple rename, rather than error-prone find-and-replace operations. Moreover, it lets a single scoped IAM policy cover all of a challenge's resources with a single wildcard.

What's in the box

We are releasing templates across the various submission tracks we have supported in blind challenges thus far: regression, classification, and structure prediction. If and when new tracks are established, we will update the challenge template repository.

Bootstrapped scoring. Every metric is computed over 1,000 bootstrap resamples of the test set, with the resample indices shared across endpoints so between-model comparisons are paired. This enables pairwise significance testing with corrections for multiple comparisons and report tiers rather than a potentially spurious strict ranking.

Uncertainty-aware scoring. The template ships a small library of custom scoring functions that take experimental error bounds into account. Swapping between them is a one line change to the metric list in backend/config.py. The default is soft-thresholded relative absolute error (ST-RAE). A prediction that lands inside a compound's experimental error bounds scores zero error, and a prediction outside it is penalised only by the distance to the nearer bound, not by the distance to the point estimate. Models aren't punished for disagreeing with a number the assay itself isn't sure about.

Macro-averaged scores. For multi-endpoint tracks, a synthetic macro-averaged (MA) endpoint is scored alongside the real ones in every bootstrap sample, a simple mean across endpoints, so one easy endpoint can't dominate the overall score.

Phased leaderboards. Submissions are scored against two compound sets: the full blinded test set and the subset used for a live leaderboard. The live leaderboard updates automatically from the subset during the challenge; interim and final leaderboards are generated from the full set at the organiser's cutoffs. At OpenADMET, we have shown that the live leaderboard strongly drives participation

The pieces:

• backend/: submission validation, the bootstrapped scoring pipeline, leaderboard and manifest generation, and the Lambda handlers that glue them to S3 and EventBridge.
• hf_space/: the Gradio app, with the submission form and the per-endpoint and overall leaderboards.
• opentofu: the full AWS stack (S3, Lambda as container images in ECR, EventBridge schedules, IAM, Secrets Manager).
• docs/: how a submission becomes a score, with pipeline diagrams.

Using it

Spin up a challenge

1. Create a repo from the template (GitHub's "Use this template", or clone and re-init).
2. Work through the TODOs. backend/config.py is the single source of truth: your endpoint names, test-set size, metric choices, and deadlines. Every fill-in point in the repo is marked with a TODO comment; a grep (or the Todo Tree extension) lists them all. hf_space/config.py mirrors the endpoint lists and holds the page content participants see.
3. Run the AWS setup in docs/SETUP.md:

• a one-time per-account bootstrap (state backend, GitHub OIDC provider), then per-challenge steps to create the scoped IAM identities and the deploy role. Nothing organisation-specific is committed; your account IDs, bucket names, and Hugging Face org come from gitignored config files locally and GitHub Actions variables in CI.

4. Create the Hugging Face Space and (optionally) a Discord channel for submission notifications.
5. Push to main. GitHub Actions builds the Lambda image, applies the OpenTofu stack, and deploys the Space. From then on, every merge redeploys.

Then upload your ground-truth datasets, and you have a running challenge. We’re excited to see what you do with it!

Adapt it

The template is deliberately configuration-driven, but it's a starting point, not a framework. If your challenge needs a metric we don't ship, add it to the metric list. If your scoring is fundamentally different, the pipeline is just a few hundred lines of readable pandas and NumPy code, so fork it. The test suite (pytest tests/backend/) runs on fully synthetic fixtures, regenerated from your config by a script, so you can validate changes without touching real data.

Tear it down

When the challenge ends, the docs/SETUP.md file includes a teardown checklist. Run every step and it removes everything specific to that challenge: the AWS resources, the deploy role, the Discord channels, and the Hugging Face Space, leaving only the shared account bootstrap for the next one. If you want the components to persist as a public record, treat the checklist as a menu instead: archive the data, tear down just the compute so it stops costing anything, and leave the S3 bucket, the Hugging Face Space, and Discord channel in place.

Bonus release: challenge-data-checker

All the infrastructure above keeps the ground truth labels secure, but it can't tell you whether your train and test sets are actually independent. Releasing your data publicly can be daunting. What if during data preparation a molecule was mistakenly registered under two different identifiers and now appears in both your training and test datasets? Or even worse, when staging your data, what if instead of clicking on training_data.csv, you’ve selected test_data_UNBLINDED.csv? That kind of leakage doesn't just inflate a leaderboard; it silently invalidates the blind challenge's entire premise, and it's the sort of thing that's easy to miss by eye in a 10,000-row spreadsheet.

We're also releasing challenge-data-checker alongside the template to catch exactly this before you ever blind a dataset. It's a small CLI (and Python API) that parses every SMILES with RDKit, canonicalises it, and checks train against test for exact leakage on raw SMILES, canonical SMILES, InChIKey, or any identifier column you configure. The checker features a fallback mechanism that calculates bulk pairwise Tanimoto similarity to catch near-duplicates and stereo mismatches that an InChIKey comparison would miss. It also includes data quality checks, such as internal duplicates within a split, identifier-namespace problems (one ID pointing at two structures, or one structure with two IDs), and common issues like mixtures, salts, metal complexes, and suspiciously bare fragments left over from a preprocessing step.

Point it at local files or a URL (including a staged HuggingFace Hub dataset), and it prints a pass/fail dashboard plus a detailed report with the exact rows responsible. It's Apache-2.0 licensed and installable with pip, so it's just as easy to drop into your own pipeline as the challenge template is.

Contributing

We'd like this to be useful beyond OpenADMET. Issues and pull requests are welcome, see CONTRIBUTING.md. A few things worth knowing up front: contributions are under Apache-2.0 with a DCO (no CLA), PRs should arrive review-ready, and AI-assisted contributions are fine as long as you've actually read and tested the output. There's a SECURITY.md for reporting anything sensitive privately.

Acknowledgements

We would like to thank our funders for their support of OpenADMET, in particular ARPAH, Radial (part of the Astera Institute (https://ror.org/00ydx1s47)), Schrödinger Inc, and the Gates Foundation. We would also like to thank our partners Enamine, HuggingFace, OpenEye, CDD Vault, Discovery Life Sciences, and the beamline staff at NSLS-II for their support.

This work is supported by the Advanced Research Projects Agency for Health (ARPA-H) under AVOID-OME, and Award Number 1AY1AX000035. The contents are those of the authors. They may not reflect the policies of the Department of Health and Human Services or the U.S. government. The content is solely the responsibility of the authors and does not necessarily represent the official views of the Advanced Research Projects Agency for Health.

Get the code:

• github.com/OpenADMET/blind-challenge-template
• github.com/OpenADMET/challenge-data-checker

Questions or ideas? Open an issue or find us on the OpenADMET Discord.