Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Discuss: Remove public dir support #524

Closed
wants to merge 1 commit into from

Conversation

ruflin
Copy link
Member

@ruflin ruflin commented Jun 18, 2020

The public dir was needed in the past to build packages. But with the recent changes to the registry this has become obsolete. Instead the registry can be directly pointed at a directory with packages. In the package storage this directory is called packages, I suggest we use this as a default. The registry itself could have its packages in the same directory.

This PR is not complete, it is more to discuss the change.

The public dir was needed in the past to build packages. But with the recent changes to the registry this has become obsolete. Instead the registry can be directly pointed at a directory with packages. In the package storage this directory is called `packages`, I suggest we use this as a default. The registry itself could have its packages in the same directory.

This PR is not complete, it is more to discuss the change.
@@ -39,6 +40,7 @@ func catchAll(public http.FileSystem, cacheTime time.Duration) func(w http.Respo
func determineResourcePath(r *http.Request, public http.FileSystem) (string, error) {
path := r.URL.Path

path = strings.Replace(path, "/package", "", 1)
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This change is odd but is needed as now all is at the top level

@ruflin
Copy link
Member Author

ruflin commented Jun 18, 2020

@mtojek I assume this also ties into your change you have planned for multiple paths?

@elasticmachine
Copy link

elasticmachine commented Jun 18, 2020

💔 Build Failed

Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: [Branch indexing]

  • Start Time: 2020-06-19T12:18:18.852+0000

  • Duration: 3 min 25 sec

Steps errors

Expand to view the steps failures

  • Name: Check out from version control

    • Description: [2020-06-19T12:20:42.239Z] using credential f6c7695a-671e-4f4f-a331-acdce44ff9ba
      [2020-06-19T12:20:

    • Duration: 0 min 6 sec

    • Start Time: 2020-06-19T12:20:39.777+0000

    • log

  • Name: Check out from version control

    • Description: [2020-06-19T12:21:05.957Z] using credential f6c7695a-671e-4f4f-a331-acdce44ff9ba
      [2020-06-19T12:21:

    • Duration: 0 min 4 sec

    • Start Time: 2020-06-19T12:21:05.939+0000

    • log

  • Name: Check out from version control

    • Description: [2020-06-19T12:21:39.684Z] using credential f6c7695a-671e-4f4f-a331-acdce44ff9ba
      [2020-06-19T12:21:

    • Duration: 0 min 3 sec

    • Start Time: 2020-06-19T12:21:39.667+0000

    • log

Log output

Expand to view the last 100 lines of log output

[2020-06-19T12:21:08.719Z]  > git fetch --no-tags --progress --prune git@github.com:elastic/package-registry.git +refs/pull/524/head:refs/remotes/origin/PR-524 +refs/heads/master:refs/remotes/origin/master # timeout=15
[2020-06-19T12:21:09.398Z] Merging remotes/origin/master commit 98d7184cfe5b1d271e1bef848a5720073959888e into PR head commit 8d3f4b1f7b6560585bfdd57177ed8e42d437b5ad
[2020-06-19T12:21:09.641Z] ERROR: Execution failed
[2020-06-19T12:21:09.641Z] Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to JNLP4-connect connection from beats-ci-immutable-ubuntu-1604-1592569108447807524.c.elastic-ci-prod.internal/10.224.0.129:50862
[2020-06-19T12:21:09.641Z] 		at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1788)
[2020-06-19T12:21:09.641Z] 		at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:356)
[2020-06-19T12:21:09.641Z] 		at hudson.remoting.Channel.call(Channel.java:998)
[2020-06-19T12:21:09.641Z] 		at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.execute(RemoteGitImpl.java:146)
[2020-06-19T12:21:09.641Z] 		at sun.reflect.GeneratedMethodAccessor1108.invoke(Unknown Source)
[2020-06-19T12:21:09.641Z] 		at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[2020-06-19T12:21:09.641Z] 		at java.lang.reflect.Method.invoke(Method.java:498)
[2020-06-19T12:21:09.641Z] 		at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.invoke(RemoteGitImpl.java:132)
[2020-06-19T12:21:09.641Z] 		at com.sun.proxy.$Proxy120.execute(Unknown Source)
[2020-06-19T12:21:09.641Z] 		at jenkins.plugins.git.MergeWithGitSCMExtension.decorateRevisionToBuild(MergeWithGitSCMExtension.java:120)
[2020-06-19T12:21:09.641Z] 		at hudson.plugins.git.GitSCM.determineRevisionToBuild(GitSCM.java:1063)
[2020-06-19T12:21:09.641Z] 		at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1168)
[2020-06-19T12:21:09.641Z] 		at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:125)
[2020-06-19T12:21:09.641Z] 		at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:93)
[2020-06-19T12:21:09.641Z] 		at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:80)
[2020-06-19T12:21:09.641Z] 		at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
[2020-06-19T12:21:09.641Z] 		at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
[2020-06-19T12:21:09.641Z] 		at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[2020-06-19T12:21:09.641Z] 		at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[2020-06-19T12:21:09.641Z] 		at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[2020-06-19T12:21:09.641Z] hudson.plugins.git.GitException: Command "git merge 98d7184cfe5b1d271e1bef848a5720073959888e" returned status code 1:
[2020-06-19T12:21:09.641Z] stdout: Removing util/dataset_test.go
[2020-06-19T12:21:09.641Z] CONFLICT (rename/delete): packages/yamlpipeline/1.0.0/dataset/log/fields/base-fields.yml deleted in 98d7184cfe5b1d271e1bef848a5720073959888e and renamed in HEAD. Version HEAD of packages/yamlpipeline/1.0.0/dataset/log/fields/base-fields.yml left in tree.
[2020-06-19T12:21:09.641Z] CONFLICT (rename/delete): packages/log/0.9.0/manifest.yml deleted in 98d7184cfe5b1d271e1bef848a5720073959888e and renamed in HEAD. Version HEAD of packages/log/0.9.0/manifest.yml left in tree.
[2020-06-19T12:21:09.641Z] CONFLICT (rename/delete): packages/log/0.9.0/img/icon.svg deleted in 98d7184cfe5b1d271e1bef848a5720073959888e and renamed in HEAD. Version HEAD of packages/log/0.9.0/img/icon.svg left in tree.
[2020-06-19T12:21:09.641Z] CONFLICT (rename/delete): packages/log/0.9.0/docs/README.md deleted in 98d7184cfe5b1d271e1bef848a5720073959888e and renamed in HEAD. Version HEAD of packages/log/0.9.0/docs/README.md left in tree.
[2020-06-19T12:21:09.641Z] CONFLICT (rename/delete): packages/log/0.9.0/dataset/log/manifest.yml deleted in 98d7184cfe5b1d271e1bef848a5720073959888e and renamed in HEAD. Version HEAD of packages/log/0.9.0/dataset/log/manifest.yml left in tree.
[2020-06-19T12:21:09.641Z] CONFLICT (rename/delete): packages/log/0.9.0/dataset/log/agent/stream/stream.yml.hbs deleted in 98d7184cfe5b1d271e1bef848a5720073959888e and renamed in HEAD. Version HEAD of packages/log/0.9.0/dataset/log/agent/stream/stream.yml.hbs left in tree.
[2020-06-19T12:21:09.641Z] Automatic merge failed; fix conflicts and then commit the result.
[2020-06-19T12:21:09.641Z] 
[2020-06-19T12:21:09.641Z] stderr: 
[2020-06-19T12:21:09.641Z] 	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2430)
[2020-06-19T12:21:09.641Z] 	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:2044)
[2020-06-19T12:21:09.641Z] 	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1944)
[2020-06-19T12:21:09.641Z] 	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandWithCredentials(CliGitAPIImpl.java:1935)
[2020-06-19T12:21:09.641Z] 	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$600(CliGitAPIImpl.java:81)
[2020-06-19T12:21:09.641Z] 	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$3.execute(CliGitAPIImpl.java:903)
[2020-06-19T12:21:09.641Z] 	at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:161)
[2020-06-19T12:21:09.641Z] 	at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:154)
[2020-06-19T12:21:09.641Z] 	at hudson.remoting.UserRequest.perform(UserRequest.java:212)
[2020-06-19T12:21:09.641Z] 	at hudson.remoting.UserRequest.perform(UserRequest.java:54)
[2020-06-19T12:21:09.641Z] 	at hudson.remoting.Request$2.run(Request.java:369)
[2020-06-19T12:21:09.641Z] 	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
[2020-06-19T12:21:09.641Z] 	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
[2020-06-19T12:21:09.641Z] 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[2020-06-19T12:21:09.641Z] 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[2020-06-19T12:21:09.641Z] 	at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:93)
[2020-06-19T12:21:09.641Z] 	at java.lang.Thread.run(Thread.java:748)
[2020-06-19T12:21:09.641Z] Retrying
[2020-06-19T12:21:09.655Z] Sleeping for 30 sec
[2020-06-19T12:21:09.397Z]  > git config core.sparsecheckout # timeout=10
[2020-06-19T12:21:09.402Z]  > git checkout -f 8d3f4b1f7b6560585bfdd57177ed8e42d437b5ad # timeout=15
[2020-06-19T12:21:09.470Z]  > git remote # timeout=10
[2020-06-19T12:21:09.474Z]  > git config --get remote.origin.url # timeout=10
[2020-06-19T12:21:09.479Z] using GIT_SSH to set credentials GitHub user @elasticmachine SSH key
[2020-06-19T12:21:09.483Z]  > git merge 98d7184cfe5b1d271e1bef848a5720073959888e # timeout=10
[2020-06-19T12:21:09.565Z]  > git config core.sparsecheckout # timeout=10
[2020-06-19T12:21:09.569Z]  > git checkout -f 8d3f4b1f7b6560585bfdd57177ed8e42d437b5ad # timeout=15
[2020-06-19T12:21:39.684Z] using credential f6c7695a-671e-4f4f-a331-acdce44ff9ba
[2020-06-19T12:21:39.700Z] Wiping out workspace first.
[2020-06-19T12:21:39.748Z] Cloning the remote Git repository
[2020-06-19T12:21:39.748Z] Using shallow clone with depth 4
[2020-06-19T12:21:39.748Z] Avoid fetching tags
[2020-06-19T12:21:39.748Z] Cloning repository git@github.com:elastic/package-registry.git
[2020-06-19T12:21:39.749Z]  > git init /var/lib/jenkins/workspace/Beats_package-registry_PR-524 # timeout=10
[2020-06-19T12:21:39.759Z] Fetching upstream changes from git@github.com:elastic/package-registry.git
[2020-06-19T12:21:39.759Z]  > git --version # timeout=10
[2020-06-19T12:21:39.767Z] using GIT_SSH to set credentials GitHub user @elasticmachine SSH key
[2020-06-19T12:21:39.772Z]  > git fetch --no-tags --progress git@github.com:elastic/package-registry.git +refs/heads/*:refs/remotes/origin/* # timeout=15
[2020-06-19T12:21:42.109Z] Cleaning workspace
[2020-06-19T12:21:42.144Z] Using shallow fetch with depth 4
[2020-06-19T12:21:42.144Z] Pruning obsolete local branches
[2020-06-19T12:21:42.087Z]  > git config remote.origin.url git@github.com:elastic/package-registry.git # timeout=10
[2020-06-19T12:21:42.090Z]  > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
[2020-06-19T12:21:42.098Z]  > git config remote.origin.url git@github.com:elastic/package-registry.git # timeout=10
[2020-06-19T12:21:42.105Z]  > git rev-parse --verify HEAD # timeout=10
[2020-06-19T12:21:42.120Z] No valid HEAD. Skipping the resetting
[2020-06-19T12:21:42.121Z]  > git clean -fdx # timeout=10
[2020-06-19T12:21:42.141Z] Fetching upstream changes from git@github.com:elastic/package-registry.git
[2020-06-19T12:21:42.141Z] using GIT_SSH to set credentials GitHub user @elasticmachine SSH key
[2020-06-19T12:21:42.145Z]  > git fetch --no-tags --progress --prune git@github.com:elastic/package-registry.git +refs/pull/524/head:refs/remotes/origin/PR-524 +refs/heads/master:refs/remotes/origin/master # timeout=15
[2020-06-19T12:21:42.861Z] Merging remotes/origin/master commit 98d7184cfe5b1d271e1bef848a5720073959888e into PR head commit 8d3f4b1f7b6560585bfdd57177ed8e42d437b5ad
[2020-06-19T12:21:43.593Z] Running on Jenkins in /var/lib/jenkins/workspace/Beats_package-registry_PR-524
[2020-06-19T12:21:43.795Z] �[39;49m[INFO] getVaultSecret: Getting secrets�[0m
[2020-06-19T12:21:43.839Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2020-06-19T12:21:42.858Z]  > git config core.sparsecheckout # timeout=10
[2020-06-19T12:21:42.861Z]  > git checkout -f 8d3f4b1f7b6560585bfdd57177ed8e42d437b5ad # timeout=15
[2020-06-19T12:21:42.929Z]  > git remote # timeout=10
[2020-06-19T12:21:42.934Z]  > git config --get remote.origin.url # timeout=10
[2020-06-19T12:21:42.937Z] using GIT_SSH to set credentials GitHub user @elasticmachine SSH key
[2020-06-19T12:21:42.941Z]  > git merge 98d7184cfe5b1d271e1bef848a5720073959888e # timeout=10
[2020-06-19T12:21:43.026Z]  > git config core.sparsecheckout # timeout=10
[2020-06-19T12:21:43.029Z]  > git checkout -f 8d3f4b1f7b6560585bfdd57177ed8e42d437b5ad # timeout=15
[2020-06-19T12:21:44.585Z] + chmod 755 generate-build-data.sh
[2020-06-19T12:21:44.586Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/package-registry/PR-524/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/package-registry/PR-524/runs/5 FAILURE 205473
[2020-06-19T12:21:44.586Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/package-registry/PR-524/runs/5/steps/?limit=10000 -o steps-info.json

@mtojek
Copy link
Contributor

mtojek commented Jun 18, 2020

Yes, it's draft currently: #525

@mtojek
Copy link
Contributor

mtojek commented Jun 18, 2020

I planned to leave PublicDir for legacy purposes:

type Config struct {
	PublicDir           string        `config:"public_dir"` // left for legacy purposes
	PackagesPaths       []string      `config:"packages_paths"`
	CacheTimeSearch     time.Duration `config:"cache_time.search"`
	CacheTimeCategories time.Duration `config:"cache_time.categories"`
	CacheTimeCatchAll   time.Duration `config:"cache_time.catch_all"`
}

@ruflin
Copy link
Member Author

ruflin commented Jun 18, 2020

Directly closing this as #525 already covers the idea from this PR .

@mtojek Lets try to get rid of all legacy before 7.9, we will build up enough over the next months :-)

@ruflin ruflin closed this Jun 24, 2020
@ruflin ruflin deleted the packages-default-dir branch June 24, 2020 07:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants