Config Validation with Pydantic #1912
Draft
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Supersedes #1547
Changes proposed in this Pull Request
This PR revives and supersedes #1547. This time using pydantic which can export to JSON Schema instead of writing the JSON Schema directly.
What we can do with this new approach:
bool,list[str]), Literals (e.g. for logging levelsLiteral["DEBUG", "INFO", "WARNING", "ERROR", "CRITICAL"], as well as any custom validation you can come up with. I added as an example that names either has to match unix date format or start withcool_name(see here)config.default.yamland define them in the model, which then can get exported to a similar yaml again. See here the automatically exported config.default based on the Pydantic Schema.This is a very minimal example right now, and not integrated into the Snakefile. I want to discuss two things first:
config.default.yamlcan still be automatically generated to not break any existing workflows, but if you wanna change defaults you would need to touch the model directly. This is what I vote for and what is done in this minimal version. We could also keep theconfig.default.yamland only use the model for validation, so keep defaults seperated from docs, validation, types etc. This needs to be decided now. @fneum @coroa @FabianHofmannlib/validation/configand not in anyscripts. Similar to the example I have right now. Nothing we need to decide right now and I just wanna hear some opinionsChecklist
pixi.toml(usingpixi add <dependency-name>).config/config.default.yaml.doc/configtables/*.csv.doc/data_sources.rst.doc/release_notes.rstis added.