Failure Passports: Turning Model Limits into Collective Intelligence

Human-body models assembled from submodels inherit each component's limits. Identically named signals can describe different compartments, timescales, or conditions. SBML enables model exchange [1], FAIR promotes machine-actionable reuse [2], and BioModels supports sharing and reproducibility [3]. Building on these foundations, I propose Failure Passports: versioned records of where a model has been tested, failed, or remains untested.

Each passport would identify a model version, intended use, biological context, inputs, outputs, units, compartments, timescales, parameter ranges, and assumptions. Provenance would distinguish measured, fitted, transferred, and assumed parameters. Test records would link predictions to comparators, uncertainty, acceptance criteria, software environments, and outcomes; retrospective criteria would be labelled. Passed, failed, untested, and disputed states would remain distinct. Failure in one context would not discredit a model universally; absent evidence would never imply success.

The collective-intelligence mechanism is cumulative boundary knowledge. Researchers would contribute limitations with source locations and executable checks where possible. A minimal checker would flag unit conflicts, compartment mismatches, and extrapolation beyond documented conditions. Semantic warnings would request expert judgment rather than automatically declare biological incompatibility. AI could suggest fields from public documentation, but accepted claims would require human verification. Contributor credit, corrections, and competing interpretations would persist.

A prospective six-month pilot would screen public BioModels entries for human relevance, runnable SBML, and accessible documentation, targeting twelve models. Selection and exclusions would be published before evaluation. Twenty-four reuse tasks would include documented compatible interfaces and deliberately seeded incompatibilities. Seeded changes would be labelled benchmark manipulations, never defects of the original models. Two independent reviewers would establish task answers; unresolved cases would be excluded under preregistered rules.

Approximately thirty volunteer evaluators would receive randomized, counterbalanced task sets, seeing each task once. The comparison would present identical facts as flat documentation or structured passports with checking. Blinded assessors would score justifications. Preregistered outcomes would be incompatibility-detection sensitivity, false-alarm rate, review time, and confidence calibration. A practical target would be a fifteen-percentage-point sensitivity gain with at most five points more false alarms. Uncertainty estimates would account for repeated evaluators and tasks. This pilot would assess feasibility, not establish general effectiveness.

Recruitment would require ethics review, informed consent, and minimal identifiers. Reviewers and model-reuse permissions are prerequisites, not secured resources. Deliverables would be an open schema, minimal checker, benchmark, and contributor guide. Model authors would be invited to correct passports and assess documentation burden. Null findings and disagreements would remain visible. The aim is a shared memory of limitations that guides defensible model connections and useful experiments. No simulations, recruitment, or empirical evaluation have been completed.

AI assistance: OpenAI Codex assisted concept development, drafting, and reference verification; prompts and public responses accompany this proposal.

References

[1] Hucka M et al. (2003). The systems biology markup language (SBML). Bioinformatics 19:524-531. doi:10.1093/bioinformatics/btg015.

[2] Wilkinson MD et al. (2016). The FAIR Guiding Principles for scientific data management and stewardship. Scientific Data 3:160018. doi:10.1038/sdata.2016.18.

[3] Malik-Sheriff RS et al. (2020). BioModels - 15 years of sharing computational models in life science. Nucleic Acids Research 48:D407-D415. doi:10.1093/nar/gkz1055.