ADMET endpoint calibration — to your assays, not a public split

Abstract

Public ADMET model evaluations consistently over-state real-world performance because customer compound libraries are not drawn from the same distribution as public chemistry databases. We describe the distribution-shift problem, the calibration interface Aether Bio ^[1] ships with, the per-endpoint headline numbers and failure modes, and the operational guidance that distinguishes a customer who gets the headline performance from one who is disappointed. The conclusion is unambiguous: un-calibrated production use is misuse, and we treat it as such — the platform actively surfaces the calibration step.

1. The distribution-shift problem

Public ADMET evaluations look great. Most published ADMET models report ROC-AUC well above 0.9 on public splits of well-characterised endpoints — hERG inhibition, Caco-2 permeability, CYP3A4 inhibition. Many beat 0.95 on these splits. The numbers are real but they are not what production looks like.

The problem is that customer compound libraries are not drawn from the same distribution as the public chemistry databases. A drug-discovery team working on a specific target class assembles a library that is enriched in a particular chemotype, depleted in certain functional groups, and may include scaffolds that are entirely absent from any public training corpus. The assay protocol the customer uses for hERG measurement may differ from the protocol that produced the public hERG dataset — different ion-channel preparation, different patch-clamp setup, different incubation times.

The consequence is a measurable performance gap. A model that hits 0.94 ROC-AUC on a public hERG split lands routinely at 0.75–0.82 on customer's actual incoming compounds before calibration. The gap is not a model failure; it is a distribution-shift cost. Pretending it does not exist serves no-one.

2. What we ship

Aether Bio's ADMET endpoints ship with a calibration interface as a first-class capability, not as an optional tooling extension. The customer provides an unlabeled set of recent compounds — typically the last 6–12 months of their incoming chemistry — plus a small labeled tail (50–200 measured compounds is sufficient). The runtime fits an isotonic calibration layer on top of the base model output. Subsequent predictions report calibrated probabilities, point estimates with confidence intervals, and an explicit out-of-distribution flag for compounds outside the calibration support.

Calibration is a property of the deployment, not of the underlying base-model checkpoint. Two customers running the same Aether Bio release have two different calibrated configurations, each accurate against their own chemistry. The base model is not modified; the per-customer calibration lives in the customer's tenant and is auditable ^[2].

Calibration is monitored. The runtime tracks calibrated-prediction-versus-measured drift across new compounds the customer assays; when the drift rises, the runtime surfaces a recommendation to refresh calibration. Customers running rapid chemistry iteration typically refresh quarterly; customers in stable lead-optimisation modes can go longer.

3. Headline numbers

On an external curated set drawn from compound series outside the training distribution, calibrated hERG classification reaches ROC-AUC 0.91. With light customer fine-tuning — a small number of in-chemotype measured compounds added to the calibration step — ROC-AUC rises to 0.94. Aqueous solubility regression hits 0.45 log units RMSE on the same external set; both numbers are competitive with specialised endpoint models.

These are headline numbers, not floor numbers. Customer-specific calibration on chemotypes well-represented in the calibration data typically beats these by 1–3 percentage points; customer-specific calibration on chemotypes weakly represented can underperform. The variance is real and we surface it through the confidence intervals attached to every prediction.

4. Endpoint coverage

Aether Bio covers the following endpoints out of the box. Calibration is supported for every endpoint; out-of-distribution flagging is supported for all of them.

• Aqueous solubility (logS) — regression.
• Caco-2 permeability — regression.
• hERG inhibition — binary classification with paired IC50 regression.
• CYP3A4 / CYP2D6 / CYP2C9 inhibition — binary classification.
• Microsomal stability (intrinsic clearance) — regression.
• Plasma protein binding (% free) — regression.
• Hepatotoxicity (DILI severity) — binary classification.
• Mutagenicity (Ames test outcome) — binary classification.
• Plasma clearance and terminal half-life — regression.
• Volume of distribution at steady state — regression.
• Brain penetration (logBB) and CNS penetration probability — regression + classification.
• Plasma stability — regression.

5. Where the model is wrong

Failure modes by endpoint, in order of practical severity. We publish these so customers know which predictions to weight more cautiously.

Microsomal stability. We under-predict intrinsic clearance at the high-clearance tail by roughly 0.4 log units. Compounds that turn over very quickly (intrinsic CL > 50 µL/min/mg microsomal protein) are systematically under-flagged. The mechanism is corpus imbalance — extremely-high-clearance compounds are over-represented in failures and under-represented in training data because they tend to be filtered out early.

Plasma protein binding. Well-calibrated for typical druglike compounds; mis-calibrated strongly for charged macrocycles (cyclic peptides above ~800 Da) and for highly lipophilic compounds (logP > 5). The model's structural representation under-uses three-dimensional shape for these classes.

CYP3A4 inhibition. Lowest absolute ROC-AUC of the cytochrome panel (≈0.84 calibrated). The training distribution under-represents time-dependent inhibition relative to its clinical importance. We are extending the training set with time-dependent-inhibition cases; the 1.1 release is expected to close the gap.

Brain penetration. Reasonable on small-molecule druglike chemistry; mis-calibrated for active-transport substrates where the prediction has to account for efflux. The model does not currently incorporate transporter substrate prediction; we recommend customers running CNS programmes layer a transporter-prediction tool on top.

6. Customer-specific fine-tuning

Beyond isotonic calibration, customers can run a light fine-tune on their measured compounds. The fine-tune is scoped to the customer's tenant and never reused across customers ^[1]. Compute cost is modest — typically 4–8 GPU-hours per endpoint per fine-tune cycle — and the lift over plain calibration is 1–3 ROC-AUC points on classification endpoints and 0.05–0.1 log units RMSE on regression endpoints, varying with the size and diversity of the customer's labeled data.

Fine-tunes are versioned alongside the agent runtime ^[3]. Each prediction in production carries a reference to the fine-tune version that produced it; auditors can verify which compounds the fine-tune saw and reproduce the prediction exactly.

7. Out-of-distribution flagging

Predictions on compounds outside the calibration support are surfaced explicitly. The runtime computes a structural-distance metric between the query compound and the calibration set; compounds beyond a configurable threshold are flagged and the prediction is returned with a wider confidence interval.

We deliberately do not refuse predictions on out-of-distribution compounds. The right call is to communicate the uncertainty rather than to fail to answer. Customers can configure the runtime to gate downstream automation on the OOD flag — refusing to synthesise an OOD-flagged hit without human review, for example — but the prediction itself is always available.

8. Why we ship calibration tooling, not just numbers

A customer who reads our headline ROC-AUC and uses the model as-is on their chemistry will probably be disappointed. A customer who runs the calibration on their compound stream and tracks deltas over time gets the performance the headline implies. We document the second path; we do not try to hide that the first one is harder.

Operationally we treat un-calibrated production use as misuse. The platform actively surfaces the calibration step during onboarding; the customer dashboard shows calibration status as a first-class metric. We have rejected customer requests to disable the calibration surface in favour of "just give me predictions" — the predictions without calibration are misleading, and downstream programmes that build on misleading numbers compound the misleading-ness.

9. Limitations

Three meaningful limitations. First, calibration depends on the customer having a labelled tail — a small number of measured compounds in their chemotype. Customers in genuinely early discovery, with very few measured compounds, get less out of calibration. Second, isotonic calibration corrects for ranking shift but not for systematic bias far outside the calibration support; OOD compounds remain harder to predict. Third, calibration is per-endpoint; cross-endpoint correlations the customer's data shows are not currently exploited.

Reproducibility: the external evaluation sets, the calibration protocol, and the public-split baseline numbers are open under the Research tier. Customer-specific calibrated configurations are not shareable across customers; this is by design ^[1].

10. Conclusion

Public ADMET numbers consistently over-state real-world performance because customer chemistry is distribution-shifted from the public splits. Aether Bio addresses this by shipping a calibration interface as a first-class capability — the customer's compound stream and labeled tail are inputs to calibration that runs in their tenant, and the resulting calibrated predictions are what production uses. Per-endpoint failure modes are documented honestly. Customers who follow the calibration workflow get the headline performance; customers who skip it will be disappointed, and we surface that risk through the platform UX rather than hiding it in documentation.