Add spec.min and spec.max query parameter in search endpoint #1058

mrodm · 2023-07-20T14:19:54Z

This PR adds two new filter parameters spec.min and spec.max for the search endpoint. These values will be used to filter packages according to their format_version value.

If no spec.min nor kibana.version are in the query, the default value for spec.min is going to be 3.0.0.

How to test

Update some test packages to build different packages with different version of format_version

Remember to update for each built package the version in manifest and changelog (e.g. 0.0.9) too
As an example, elastic_package_registry could be modified as:

 $ git diff
diff --git packages/elastic_package_registry/manifest.yml packages/elastic_package_registry/manifest.yml
index b8a41f9b5..0d819ff0e 100644
--- packages/elastic_package_registry/manifest.yml
+++ packages/elastic_package_registry/manifest.yml
@@ -1,4 +1,4 @@
-format_version: 1.0.0
+format_version: 2.10.0
 name: elastic_package_registry
 title: "Elastic Package Registry"
 version: 0.0.7

Build the packages
```
elastic-package build
```

Build a new local docker image for package-registry

cd /path/to/package-registry   
docker build -t docker.elastic.co/package-registry/package-registry:v1.21.0-rc1 .

Run local package-registry using the packages directory with the built packages as a volume mount point:

docker run --rm -p 8080:8080 -v /home/mariorodriguez/Coding/work/integrations/build/packages/:/packages/package-registry -it docker.elastic.co/package-registry/package-registry:v1.21.0-rc1

Execute some queries against this local package-registry:

# filter packages with spec.min and spec.max
curl -s "http://localhost:8080/search?spec.min=2.2.0&spec.max=3.0.0&prerelease=true" | jq -r '.[]|"\(.name) - \(.version)"'
# filter packages with spec and kibana.version
curl -s "http://localhost:8080/search?spec.min=2.2.0&spec.max=3.0.0&kibana.version=8.6.0&prerelease=true" | jq -r '.[]|"\(.name) - \(.version)"'

mrodm · 2023-07-20T14:32:57Z

packages/packages.go


 	// Deprecated, release tags to be removed.
 	Experimental bool
 }

 // Apply applies the filter to the list of packages, if the filter is nil, no filtering is done.
-func (f *Filter) Apply(ctx context.Context, packages Packages) Packages {
+func (f *Filter) Apply(ctx context.Context, packages Packages) (Packages, error) {


Update return parameters, since semver.NewConstraint could fail too with an error.

mrodm · 2023-07-20T14:33:46Z

packages/packages.go

-		if f.Capabilities != nil {
-			if valid := p.WorksWithCapabilities(f.Capabilities); !valid {
-				continue
-			}
-		}
-


Removed from legacy. These clients should not send capabilities in their queries.

mrodm · 2023-07-20T14:37:14Z

packages/packages_test.go

+	}
+}
+
+func TestPackagesSpecMinMaxFilter(t *testing.T) {


Create a new test based on TestPackagesFilter in order to be able to set specific packages.

CHANGELOG.md

mrodm · 2023-07-20T18:42:27Z

packages/package.go

+	if specMin == nil && kibanaVersion == nil {
+		specMin = semver.MustParse("3.0.0")
+	}


Not totally sure about this, this could be changed depending on what it is decided here #1050 (comment)

Removed this default minimum spec.
So in case there is no spec.min in the default query, it will return all the packages filtered just by spec.max

mrodm · 2023-07-24T11:02:36Z

main_test.go

+		{"/search?capabilities=observability,security&prerelease=true", "/search", "search-prerelease-capabilities-observability-security.json", searchHandler(testLogger, indexer, testCacheTime)},
+		{"/search?capabilities=none&prerelease=true", "/search", "search-prerelease-capabilities-none.json", searchHandler(testLogger, indexer, testCacheTime)},
+		{"/search?spec.min=1.1.0&spec.max=2.10.0&prerelease=true", "/search", "search-spec-min-1.1.0-max-2.10.0.json", searchHandler(testLogger, indexer, testCacheTime)},
+		{"/search?spec.max=2.10.0&prerelease=true", "/search", "search-spec-max-2.10.0.json", searchHandler(testLogger, indexer, testCacheTime)},


Added some tests for search endpoint using capabilities and spec parameters.

Some versions of the example and foo test packages have been updated:

adding capabilities

updating format_version number

jsoriano · 2023-07-24T22:20:55Z

main_test.go

@@ -90,6 +90,10 @@ func TestEndpoints(t *testing.T) {
 		{"/search?type=input&prerelease=true", "/search", "search-input-packages.json", searchHandler(testLogger, indexer, testCacheTime)},
 		{"/search?type=input&package=integration_input&prerelease=true", "/search", "search-input-integration-package.json", searchHandler(testLogger, indexer, testCacheTime)},
 		{"/search?type=integration&package=integration_input&prerelease=true", "/search", "search-integration-integration-package.json", searchHandler(testLogger, indexer, testCacheTime)},
+		{"/search?capabilities=observability,security&prerelease=true", "/search", "search-prerelease-capabilities-observability-security.json", searchHandler(testLogger, indexer, testCacheTime)},
+		{"/search?capabilities=none&prerelease=true", "/search", "search-prerelease-capabilities-none.json", searchHandler(testLogger, indexer, testCacheTime)},
+		{"/search?spec.min=1.1.0&spec.max=2.10.0&prerelease=true", "/search", "search-spec-min-1.1.0-max-2.10.0.json", searchHandler(testLogger, indexer, testCacheTime)},


I see I didn't copy this to the description of the issue, sorry. In the original proposal we thought about using only <major>.<minor>, so if a version of Kibana supports 2.0, it supports any 2.0.X version. Patch versions shouldn't include differences that break compatibility, or new features.

So the API would work like this:

Suggested change

{"/search?spec.min=1.1.0&spec.max=2.10.0&prerelease=true", "/search", "search-spec-min-1.1.0-max-2.10.0.json", searchHandler(testLogger, indexer, testCacheTime)},

{"/search?spec.min=1.1&spec.max=2.10&prerelease=true", "/search", "search-spec-min-1.1.0-max-2.10.0.json", searchHandler(testLogger, indexer, testCacheTime)},

Ok, it makes sense.

Updated the tests accordingly with that.

I also added a validation to check that versions in the query parameter are just major.minor (added also a test for that).

CHANGELOG.md

mrodm · 2023-07-25T07:56:24Z

search.go

+func getSpecVersion(version string) (*semver.Version, error) {
+	// version must cointain just <major.minor>
+	if len(strings.Split(version, ".")) != 2 {
+		return nil, fmt.Errorf("invalid version '%s': it should be <major.version>", version)
+	}
+	specVersion, err := semver.NewVersion(version)
+	if err != nil {
+		return nil, fmt.Errorf("invalid spec version '%s': %w", version, err)
+	}
+	return specVersion, nil
+}


Ensure that versions in query parameters are as expected. That means with the pattern major.minor

packages/package.go

jsoriano · 2023-07-25T17:50:18Z

packages/package.go

+	}
+
+	versionSplit := strings.Split(p.FormatVersion, ".")
+	versionSplit[2] = "0"


Nit. Check length to avoid panics?

Or maybe we can just build this version based on the previously parsed one.

specMajorMinorVersion := fmt.Sprintf("%d.%d.0", p.specSemVer.Major(), p.specSemVer.Minor())

elasticmachine · 2023-07-26T09:36:46Z

💚 Build Succeeded

Buildkite Build
Commit: 0b01436

History

💔 Build #311 failed deab506
💚 Build #309 succeeded 85b5a31
💚 Build #301 succeeded a18750c
💚 Build #300 succeeded 27350b7

cc @mrodm

packages/package.go

mrodm added 4 commits July 20, 2023 15:02

Add spec.min and spec.max query parameters

701a1a4

Add error return parameter in Apply

50af49b

Update tests

c36b0f6

Update msg error

ea70991

mrodm self-assigned this Jul 20, 2023

mrodm added 2 commits July 20, 2023 16:21

Add changelog entry

c655d28

Remove capabilities and spec.min/max filters from legacyApply

ed4afff

mrodm commented Jul 20, 2023

View reviewed changes

Remove error from legacyApply

403dfa1

mrodm commented Jul 20, 2023

View reviewed changes

Update changelog entry

b8f49a5

mrodm commented Jul 20, 2023

View reviewed changes

CHANGELOG.md Outdated Show resolved Hide resolved

mrodm added 2 commits July 20, 2023 19:40

Add tests for search endpoint (capabilities and spec)

10b0298

Delete unused test generated file

146b77a

mrodm commented Jul 20, 2023

View reviewed changes

mrodm marked this pull request as ready for review July 20, 2023 18:46

mrodm requested a review from a team July 20, 2023 18:46

Remove spec.min default value if it is not defined

2a15c15

mrodm commented Jul 24, 2023

View reviewed changes

mrodm mentioned this pull request Jul 24, 2023

Add spec.min, spec.max to categories endpoint #1059

Merged

jsoriano reviewed Jul 24, 2023

View reviewed changes

mrodm added 2 commits July 25, 2023 09:47

Allow just versions major.minor

27350b7

Remove link from changelog entry

a18750c

mrodm commented Jul 25, 2023

View reviewed changes

mrodm requested a review from jsoriano July 25, 2023 07:59

mrodm added 2 commits July 25, 2023 15:58

Merge remote upstream/main into add_spec_min_max_search

85b5a31

Set patch version to zero in spec to compare with spec.min/max

deab506

jsoriano approved these changes Jul 25, 2023

View reviewed changes

Skip specMajorMinorSemVer precalculation for proxied packages

0b01436

mrodm force-pushed the add_spec_min_max_search branch from 04b1e17 to 0b01436 Compare July 26, 2023 09:28

mrodm commented Jul 26, 2023

View reviewed changes

packages/package.go Show resolved Hide resolved

mrodm requested a review from jsoriano July 26, 2023 09:54

jsoriano approved these changes Jul 26, 2023

View reviewed changes

mrodm merged commit d009793 into elastic:main Jul 26, 2023
1 check passed

mrodm deleted the add_spec_min_max_search branch July 26, 2023 16:27

andrewkroh mentioned this pull request Sep 13, 2023

[docs] Add missing /search query params to docs (spec.min, spec.max, etc) #1087

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add spec.min and spec.max query parameter in search endpoint #1058

Add spec.min and spec.max query parameter in search endpoint #1058

mrodm commented Jul 20, 2023 •

edited

Loading

mrodm Jul 20, 2023

mrodm Jul 20, 2023

mrodm Jul 20, 2023

mrodm Jul 20, 2023

mrodm Jul 24, 2023

mrodm Jul 24, 2023

jsoriano Jul 24, 2023 •

edited

Loading

mrodm Jul 25, 2023

mrodm Jul 25, 2023

jsoriano Jul 25, 2023

jsoriano Jul 25, 2023

elasticmachine commented Jul 26, 2023

	{"/search?spec.min=1.1.0&spec.max=2.10.0&prerelease=true", "/search", "search-spec-min-1.1.0-max-2.10.0.json", searchHandler(testLogger, indexer, testCacheTime)},
	{"/search?spec.min=1.1&spec.max=2.10&prerelease=true", "/search", "search-spec-min-1.1.0-max-2.10.0.json", searchHandler(testLogger, indexer, testCacheTime)},

Add spec.min and spec.max query parameter in search endpoint #1058

Add spec.min and spec.max query parameter in search endpoint #1058

Conversation

mrodm commented Jul 20, 2023 • edited Loading

How to test

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jsoriano Jul 24, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

elasticmachine commented Jul 26, 2023

💚 Build Succeeded

History

mrodm commented Jul 20, 2023 •

edited

Loading

jsoriano Jul 24, 2023 •

edited

Loading