Add search parameter to query GA and technical preview packages

[juliaElastic]

Based on user feedback, we should make it easier for users to see packages that are in technical preview (not yet GA) without having to toggle on beta integrations on Fleet UI.
Simply removing the toggle from the UI and defaulting to show prerelease integrations is not feasible, as it would show prerelease versions of GA packages too, which is risky and not intended for most users.

Currently it's not easy query packages with GA versions with packages on technical preview.
The search parameter prerelease:false includes only GA packages, prerelease:true includes prerelease versions of GA packages too.

We could add a third state to the prerelease parameter that would return GA versions and technical preview packages.
Alternatively we could change the meaning of prerelease:false that would return technical preview packages too.

[jsoriano]

We could add a third state to the prerelease parameter that would return GA versions and technical preview packages.
Alternatively we could change the meaning of prerelease:false that would return technical preview packages too.

I will call @nimarezainia for this decision 😇:

• Adding a third state to prerelease (or a new parameter) will allow to introduce the change in a more controlled way, as it will be available only on the future versions of Fleet that adopt this new option. It would also allow to keep some kind of setting, so users can chose the behavior.
• Changing the meaning of prerelease: false will apply to all deployments, without needing any change in Fleet.

The second option seems riskier, but would require less changes and would bring the new approach also to current users of Fleet.

[jen-huang]

Changing the meaning of prerelease: false will apply to all deployments, without needing any change in Fleet.

TBH I like this idea the most, but it will affect every version of Kibana, including old versions where it is possible that the number of preview packages exceed GA ones (think late 7.x/early 8.x), so I don't think this behavior should apply there.

Adding a third state to prerelease (or a new parameter) will allow to introduce the change in a more controlled way, as it will be available only on the future versions of Fleet that adopt this new option. It would also allow to keep some kind of setting, so users can chose the behavior.

For that reason I would prefer a third state/new param.

Also, I want to point out that I think we should still do the actual UI work of removing the beta toggle and underlying enable prerelease flag as originally requested. I elaborate more here: https://github.com/elastic/ingest-dev/issues/4790#issuecomment-3444300109

[jsoriano]

Changing the meaning of prerelease: false will apply to all deployments, without needing any change in Fleet.

TBH I like this idea the most, but it will affect every version of Kibana, including old versions where it is possible that the number of preview packages exceed GA ones (think late 7.x/early 8.x), so I don't think this behavior should apply there.

Old versions of kibana use experimental=true, what leads to a legacy code path in Package Registry that doesn't filter out prerelease packages as we do now with prerelease=false. So in principle behavior wouldn't change for these old versions.
I think experimental=true was used until 8.6 (till elastic/kibana#143853).

It would be still a change for versions over 8.6, and wouldn't allow to customize behavior per user config as a third state would do. But maybe this is an acceptable trade-off for the other possible benefits (less changes in Kibana, and availability of packages without GA versions to more deployments).

Also, I want to point out that I think we should still do the actual UI work of removing the beta toggle and underlying enable prerelease flag as originally requested. I elaborate more here: elastic/ingest-dev#4790 (comment)

Ah yes, you are right, some work is needed in Kibana in any case.

[nimarezainia]

@jamiehynds @cpascale43 @daniela-elastic @lalit-satapathy fyi

[nimarezainia]

Can I ask a naive question, who uses (or is interested in using) pre-release versions? I'd imagine it's purely there for the integration developer. If so can we find a way that developers can see what they need - perhaps using a well known feature flag.

Here we have 2 main options for packages: GA & Beta/Tech-Preview. Within each we have pre-release versions. I don't think the end user should ever get their hands on a pre-release. To that end I would propose:

1. remove the beta toggle, so GA and techpreview packages are shown together
2. Filter out pre-release (if needed show them via a feature flag). Sounds like for this we need a 3rd state.

If we agree on the above then let's change the meaning of the pre-release flag to apply to all GA and TechPreview, instead of introducing a new 3rd state).

[jsoriano]

Can I ask a naive question, who uses (or is interested in using) pre-release versions?

Looking to the initial [internal] request (https://github.com/elastic/ingest-dev/issues/4790), the intention was to show in-development packages to users. They would be warned by the beta labels and so on.

Filter out pre-release (if needed show them via a feature flag).

Do you mean to filter-out from Fleet side?

Sounds like for this we need a 3rd state.

We need a third state if we want to query GA and prerelease packages together without using all=true, that is too big. Alternatively Fleet could make two requests, one with prerelease=false and another one with prerelease=true. If the toggle/feature flag is disabled, then it would only use prerelease=false, that is the current default.

If we agree on the above then let's change the meaning of the pre-release flag to apply to all GA and TechPreview, instead of introducing a new 3rd state).

Not sure if I understand this, would it be to show GA and TechPreview if a feature flag is enabled and only GA by default?

[nimarezainia]

Filter out pre-release (if needed show them via a feature flag).

Do you mean to filter-out from Fleet side?

Yes. if the intention is not to show them, filter out the pre-release. Show all GA and Tech Preview.

If we agree on the above then let's change the meaning of the pre-release flag to apply to all GA and TechPreview, instead of introducing a new 3rd state).

Not sure if I understand this, would it be to show GA and TechPreview if a feature flag is enabled and only GA by default?

I think we need to by default show GA and TechPreview and have an option to filter out the pre-release options for both. that's what I was asking.

[daniela-elastic]

@jamiehynds @cpascale43 @daniela-elastic @lalit-satapathy fyi

I left a comment here which should trickle down to the implementation details of what parameters we use in order to meet the contextual use cases described

[jsoriano]

filter out the pre-release. Show all GA and Tech Preview.

by default show GA and TechPreview and have an option to filter out the pre-release options for both

We don't distinguish now between pre-release and tech preview, package versions are now GA or prerelease. We would need to define a difference between prerelease and tech preview.

[nimarezainia]

I think we need to by default show GA and TechPreview and have an option to filter out the pre-release options for both. that's what I was asking.

@jsoriano and @juliaElastic, I go back to this comment above. By default show GA and Tech-preview packages, and filter out pre-release .

if a developer wants to see the pre-release versions what can we do? I want to avoid another toggle.

[juliaElastic]

if a developer wants to see the pre-release versions what can we do? I want to avoid another toggle.

We could add a kibana config (like a feature flag) to turn on prerelease versions.

[nimarezainia]

if a developer wants to see the pre-release versions what can we do? I want to avoid another toggle.

We could add a kibana config (like a feature flag) to turn on prerelease versions.

that is acceptable I think.

[jamiehynds]

Just to confirm prerelease in this sense does not equate to Tech Preview? I think we're on the same page that Tech Preview and GA integrations should be shown by default. I'm just not sure what a pre-release package actually is.

In the Integrations UI, when an integration is marked as Tech Preview, there's a banner that calls them pre-release, hence the confusion.