Stabilize observability

[Liam-DeVoe]

Closes #4387.

• @settings(observability=...) takes True, False, or ObservabilitySettings. If not set, inherits from HYPOTHESIS_OBSERVABILITY, which is either true or false (no ObservabilitySettings parsing/control).
• HYPOTHESIS_EXPERIMENTAL_OBSERVABILITY_NOCOVER removed with no deprecation period
• HYPOTHESIS_EXPERIMENTAL_OBSERVABILITY and HYPOTHESIS_EXPERIMENTAL_OBSERVABILITY_CHOICES kept for backwards-compat, but will be removed eventually
• observability.OBSERVABILITY_COLLECT_COVERAGE and observability.OBSERVABILITY_CHOICES replaced with observability.OBSERVABILITY_SETTINGS

For a future PR: with observability callbacks, there's no way to control ObservabilitySettings. You don't want to set @settings(observability=ObservabilitySettings(...)) on the test, because this would write to .hypothesis/observed, and you just want the observations in-memory.

You can do this by monkeypatching observability.OBSERVABILITY_SETTINGS. (indeed, hypofuzz does this right now). I'd like to get rid of that var, and add add_observability_callback(options: ObservabilitySettings). But semantic questions like "what if two callbacks where only one sets coverage=True" have to be answered. I do think this can be done, I just don't want to do it right now. So I've left the var in, but undocumented.

old description

• Disable coverage by default
  • IMO this is too much data (and too high risk for collision with sys.monitoring tools / slowdowns, but that's a weaker argument) to enable by default, but I'm open to discussion
• @settings(observability=...) takes three possible values:
  • observability=True enables observability
  • observability=False disables observability
  • observability=[option1, option2], like observability=["coverage"], enables observability, and also enables whatever options are specified
• if @settings(observability=...) is not set, inherit from HYPOTHESIS_OBSERVABILITY envvar
• HYPOTHESIS_EXPERIMENTAL_OBSERVABILITY_NOCOVER removed with no deprecation period
• HYPOTHESIS_EXPERIMENTAL_OBSERVABILITY and HYPOTHESIS_EXPERIMENTAL_OBSERVABILITY_CHOICES kept for backwards-compat because it was easy, but are now-undocumented and will be removed at some point

I'd like to keep a very clear separation between @settings(observability=...), and add_observability_callback. IMO the former should always write to/be related to .hypothesis/observed, and the latter should never be (ie it's sent in-memory).

This brings up a non-trivial issue, probably for a future PR: when adding observability callbacks, there's no way to control the coverage and choices options. You don't want to set @settings(observability=["choices"]) on the test/profile, because this would write to .hypothesis/observed, and you just want the observations in-memory. Previously this was done with observability.OBSERVABILITY_COLLECT_COVERAGE and observability.OBSERVABILITY_CHOICES. I'd like to get rid of those vars eventually, and add add_observability_callback(options=["coverage"]). But semantic questions like "what if two callbacks where only one sets options=["coverage"] have to be answered. I do think this can be done, I just don't want to do it right now. So I've left those vars in, though now un-documented.

---

• update hypofuzz monkeypatching to use new OBSERVABILITY_SETTINGS

I'd also love to write a hypothesis.works blog post about this, at some point..

[cc @hgoldstein95]

[Zac-HD]

(incomplete thoughts)

[Zac-HD]

Suggested change

	coverage: bool = True
	choices: bool = False
	callbacks: tuple[Callable, ...] = field(default=(_deliver_to_file,))
	coverage: Literal["line", "branch"] | Collection[Literal["line", "branch"]] | None = None
	choices: Literal["values", "nodes"] | None = None
	callbacks: tuple[Callable[[dict], None], ...] = (_deliver_to_file,)

plus validation to ensure that callbacks is nonempty, and a __or__/.union() method to merge configs

Alternatively we could make it easy to get the standard callback by making it a staticmethod on the ObservabilitySettings class 🤔

[Zac-HD]

let's also note_deprecation about this, so there's an explicit warning.

Alternatively I'd be fine with immediately and loudly breaking it; IMO "silently does nothing" is the worst option.

[Zac-HD]

Some more notes after our in-person conversation; settings issue coming soon.

[Zac-HD]

Needs to note that callbacks can be customized / this is the default only.

[Zac-HD]

We probably want a similar heading for Coverage metadata, now that it's off by default?

And also note that the format of that is unstable, since we might want to switch representations

[Zac-HD]

observability_enabled() should also be deleted, in favor of checking settings().observability is not None as we discussed.

[Zac-HD]

I think here we want to explicitly error if it's set and not in True/true/1/False/false/0; otherwise you can get the awkward situation where =yes is silently accepted, and then treated as false.