Reduce total time of booting up the registry

[mtojek]

As the number of packages is growing the package-registry needs more time to validate and load all packages before starting the HTTP server.

There are multiple ways of fixing this issue, including disabling package verification, loading packages once needed or exposing two separate endpoints for health and readiness.

[mtojek]

I looked into the source code -

@ruflin do you think we can disable the validation in package-registry (or add a switch for that in package-storage Docker images)? Packages are validated on the Integrations level, I didn't see any situation in which the package-registry notified about the problem. It will result in decreasing the total boot time.

The other option, separate endpoints for health and readiness, will play better with k8s orchestrator, but it won't decrease the deployment time, so I would vote for the first option.

[ruflin]

If we have a "distribution" package with all the packages backed in, ++ on disabling the validation and doing it during the Docker build.

I think there is still value for validation in the case the registry is run to read in packages from a volume where validation was not done before. Could we make validation just a flag?

I assume package-registry still has its own validation or did we switch over the package-spec?

[mtojek]

If we have a "distribution" package with all the packages backed in, ++ on disabling the validation and doing it during the Docker build.

Yes, that's the plan.

I think there is still value for validation in the case the registry is run to read in packages from a volume where validation was not done before. Could we make validation just a flag?

Yes, I was thinking about the flag, which is disabled in the distribution mode (as all packages have been already validated).

I assume package-registry still has its own validation or did we switch over the package-spec?

It's still there as we need to research if there (where) are validation gaps, so we can switch to package-spec. In general the package-spec catches 99% package errors.

[ruflin]

Would be great to get rid of the package-registry own implementation for validation and have parity with package-spec.

[mtojek]

Let me split into half and work on the flag first.

Regarding switching to the package-spec - we have to remember that this will introduce another upstream dependency in the package-registry, 3 branches of the package-storage and integrations. In other words, every commit pushed to the package-spec will trigger an avalanche of updates. I wonder if we can louse coupling here or simply accept it (/cc @ycombinator ).

[ruflin]

At the moment the "problem" is that each commit means a release (more or less) and in general a package-spec change is driven by a very specific need to the release has to happen immeditately. I wonder if over time the releases become less frequent? I personally like that we don't magically update but avalanche of updates is not great. Automation possible maybe?

[ycombinator]

I'm +1 for switching the validation in the package registry to use the package spec validator (orthogonal to the flag work, of course). As for the update avalanche problem, we should be able to automate this fairly easily by creating downstream projects in Jenkins so each time there's a successful package-spec build, we kick off builds for elastic-package, package-registry, etc.?

[mtojek]

As for the update avalanche problem, we should be able to automate this fairly easily by creating downstream projects in Jenkins so each time there's a successful package-spec build, we kick off builds for elastic-package, package-registry, etc.?

Do you see any magical way to update Go dependency on package-spec (go.mod, go.sum)? It might be tricky as some parts of code are compiled not just embedded (schemas). Of course it will bring us to another problem, when people forget about updating the binaries :) Probably we have to spend time to analyze this issue.

[ycombinator]

Do you see any magical way to update Go dependency on package-spec (go.mod, go.sum)?

I admit I haven't experimented with this in depth but I think go get should upgrade the dependency on package-spec to the next available version (ref: https://golang.org/ref/mod#mvs-upgrade)? Of course, the key with such upgrades being safe is that package-spec follows semantic versioning in that it doesn't break backwards compatibility within a major version.

That said, I see your point that it's not as simple as configuring downstream Jenkins builds. We will need further automation to run the go get and if there's a diff in the go.mod file due to a package-spec version upgrade, to automatically create the PR for it.

Such automatic upgrades will solve the problem of ensuring that any tools that depend on package-spec are always using the latest available version of package-spec. They will not solve the problem of making sure these tools are using the latest features available in package-spec - for this, a manual PR will be necessary.

[ycombinator]

Of course it will bring us to another problem, when people forget about updating the binaries

I assume you are thinking of elastic-package here? I think we've discussed this in the past: elastic/elastic-package#32.

[mtojek]

I'm deploying new releases of package-registry. Once finished I will close this issue as we've one opened for package-spec validation: #626 . The main issue has been mitigated, the boot up time went down to 3-4 secs.