Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[JENKINS-41758] Add a Declarative Agent extension for Kubernetes #127

Merged
merged 9 commits into from
May 4, 2017

Conversation

abayer
Copy link
Member

@abayer abayer commented Feb 7, 2017

JENKINS-41758

Ok, I think this is ready to go.

cc @reviewbybees esp @carlossg

@abayer
Copy link
Member Author

abayer commented Feb 7, 2017

fwiw, I can't get any of the tests using minikube to run locally, so I can't actually verify anything at all. grr.

@abayer
Copy link
Member Author

abayer commented Feb 7, 2017

...of course, just after I say that, it works and I get my new test to pass. heh.

@carlossg
Copy link
Contributor

carlossg commented Feb 8, 2017

can you provide an example of the declarative syntax that this would enable?

@abayer
Copy link
Member Author

abayer commented Feb 8, 2017

Take a look at the test I added?

@carlossg
Copy link
Contributor

carlossg commented Feb 8, 2017

doh!

@abayer
Copy link
Member Author

abayer commented Feb 9, 2017

So jenkinsci/pipeline-model-definition-plugin#109 will simplify the "should we do checkout?" bit and let us trim down the dependencies again.

Also worth noting - I currently don't have a good way of providing a List to a DeclarativeAgent field - working on that, but until that's the case, I'm not going to bother supporting volumes at all. The semantics of Declarative don't match one to one with the k8s plugin's Pipeline semantics as well - i.e., specifying multiple ContainerTemplates in Declarative would be pointless, since we'd be re-running the whole podTemplate etc call for each agent anyway. I'm still thinking about that.

@abayer
Copy link
Member Author

abayer commented Feb 21, 2017

Moving to Declarative 1.0.2 and simplifying the dependencies and logic a bit.

@ghost
Copy link

ghost commented Feb 25, 2017

This pull request originates from a CloudBees employee. At CloudBees, we require that all pull requests be reviewed by other CloudBees employees before we seek to have the change accepted. If you want to learn more about our process please see this explanation.

@rsandell
Copy link
Member

🐝

@iocanel
Copy link
Contributor

iocanel commented Apr 4, 2017

@abayer: this is not work in progress any more, right? Should we try to get it in?

@abayer
Copy link
Member Author

abayer commented Apr 7, 2017

Lemme update it first. =)

abayer added 6 commits April 7, 2017 12:54
Note that this is just the first work-in-progress commit at this point
- I want to make some changes in Declarative and release those before
actually merging this (to simplify the dependencies and checkout
process) but this is a start.
@abayer abayer closed this Apr 7, 2017
@abayer abayer reopened this Apr 7, 2017
@abayer
Copy link
Member Author

abayer commented Apr 7, 2017

The error looks to be transient noise...

private String cloud;
private String inheritFrom;

private int instanceCap;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These should probably have a centralized config with defaults, same as dockerLabel et.al.
The pipeline dev should not have to know all these things, looks more like admin stuff.

@abayer
Copy link
Member Author

abayer commented May 3, 2017

@carlossg @iocanel ping? The failures are stupid InjectedTest tricks that may or may not keep happening...

@carlossg
Copy link
Contributor

carlossg commented May 3, 2017

I get the same test error locally, changing the pom to

<jenkins-workflow-step-api.version>2.9</jenkins-workflow-step-api.version>

helped to fix all but one

org.csanchez.jenkins.plugins.kubernetes.pipeline.KubernetesPipelineTest
java.lang.NoSuchMethodError: No such DSL method 'pipeline' found among steps ...

@abayer
Copy link
Member Author

abayer commented May 3, 2017

Ok, will revisit tomorrow.

@abayer
Copy link
Member Author

abayer commented May 4, 2017

Ok, the workflow-step-api thing was legit. And the other, I believe, was due to a missing test dependency. Whoops.

Copy link
Contributor

@carlossg carlossg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tests passing \o/

@carlossg carlossg merged commit c12cb70 into jenkinsci:master May 4, 2017
@carlossg
Copy link
Contributor

carlossg commented May 4, 2017

Looking at the example in the test it still needs something. At least needs a way to specify that you want to run mvn -version in the maven container

pipeline {
  agent {
    kubernetes {
      cloud 'minikube'
      label 'mypod'
      containerTemplate {
        name 'maven'
        image 'maven:3.3.9-jdk-8-alpine'
        ttyEnabled true
        command 'cat'
      }
    }
  }
  stages {
    stage('Run maven') {
      steps {
        sh 'mvn -version'
      }
    }
  }
}

@abayer
Copy link
Member Author

abayer commented May 5, 2017

Ok, I'll file a followup PR.

@berni2288
Copy link

Sorry for bumping this closed PR, I compiled the kubernetes plugin from the master branch and tried to run the above pipeline example, but I get a ClassNotFoundException. The plugin seems to work when not using the declarative pipeline syntax, but I prefer the declarative syntax. Did I do something wrong?

java.lang.ClassNotFoundException: org.csanchez.jenkins.plugins.kubernetes.pipeline.KubernetesDeclarativeAgentScript at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:677) at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:787) at groovy.lang.GroovyClassLoader.loadClass(GroovyClassLoader.java:775) at org.jenkinsci.plugins.pipeline.modeldefinition.withscript.WithScriptDescribable.getScript(WithScriptDescribable.java:53) at org.jenkinsci.plugins.pipeline.modeldefinition.agent.DeclarativeAgent.getScript(DeclarativeAgent.java:50) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93) at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325) at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1215) at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1024) at org.codehaus.groovy.runtime.callsite.PojoMetaClassSite.call(PojoMetaClassSite.java:47) at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48) at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113) at com.cloudbees.groovy.cps.sandbox.DefaultInvoker.methodCall(DefaultInvoker.java:18) at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.inDeclarativeAgent(jar:file:/var/jenkins_home/plugins/pipeline-model-definition/WEB-INF/lib/pipeline-model-definition.jar!/org/jenkinsci/plugins/pipeline/modeldefinition/ModelInterpreter.groovy:361) at org.jenkinsci.plugins.pipeline.modeldefinition.ModelInterpreter.call(jar:file:/var/jenkins_home/plugins/pipeline-model-definition/WEB-INF/lib/pipeline-model-definition.jar!/org/jenkinsci/plugins/pipeline/modeldefinition/ModelInterpreter.groovy:74) at WorkflowScript.run(WorkflowScript:1) at ___cps.transform___(Native Method) at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:57) at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:109) at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:82) at sun.reflect.GeneratedMethodAccessor182.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72) at com.cloudbees.groovy.cps.impl.PropertyishBlock$ContinuationImpl.get(PropertyishBlock.java:76) at com.cloudbees.groovy.cps.LValueBlock$GetAdapter.receive(LValueBlock.java:30) at com.cloudbees.groovy.cps.impl.PropertyishBlock$ContinuationImpl.fixName(PropertyishBlock.java:66) at sun.reflect.GeneratedMethodAccessor187.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72) at com.cloudbees.groovy.cps.impl.ConstantBlock.eval(ConstantBlock.java:21) at com.cloudbees.groovy.cps.Next.step(Next.java:83) at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:173) at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:162) at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:122) at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:261) at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:162) at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$001(SandboxContinuable.java:19) at org.jenkinsci.plugins.workflow.cps.SandboxContinuable$1.call(SandboxContinuable.java:35) at org.jenkinsci.plugins.workflow.cps.SandboxContinuable$1.call(SandboxContinuable.java:32) at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.GroovySandbox.runInSandbox(GroovySandbox.java:108) at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:32) at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:174) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:330) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$100(CpsThreadGroup.java:82) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:242) at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:230) at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:64) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:112) at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)

@carlossg
Copy link
Contributor

Yeah I tried to demo it yesterday and got the same error. I guess the tests pass because the classpath is built from the maven project but packaging must be wrong. @abayer ?

@berni2288
Copy link

The question is also, would multiple containers work with declarative pipelines? How would you define multiple container templates?

@abayer
Copy link
Member Author

abayer commented Jun 14, 2017

I'll take a look. That's weird.

@abayer
Copy link
Member Author

abayer commented Jun 14, 2017

Ok, the packaging is fine, I think it's some classloader problem. Experimenting further.

@abayer
Copy link
Member Author

abayer commented Jun 14, 2017

There's something really weird here with the classloader - the UberClassLoader for Jenkins can't find any resource in kubernetes.jar (from kubernetes.hpi). It's fine for any other plugin, just not kubernetes. I...have no idea what's going on at this point.

@carlossg
Copy link
Contributor

Maybe because the pom has pluginFirstClassLoader=true ?

@abayer
Copy link
Member Author

abayer commented Jun 14, 2017

That could be. But I'm not sure how that would mess with the global classloader...

@berni2288
Copy link

I think pluginFirstClassLoader=false made it working for me now.

@abayer
Copy link
Member Author

abayer commented Jun 14, 2017

Ah-ha! So yeah, https://github.com/jenkinsci/jenkins/blob/d13b1361e1650494528a7b46f82fe25d7fb5e3c3/core/src/main/java/hudson/PluginManager.java#L1890-L1890 doesn't work for kubernetes, presumably due to the pluginFirstClassLoader=true. Digging!

@berni2288
Copy link

Ok, I aready wondered if you did test driven development only :)

Next error: Jenkins doesn’t have label mypod

Could this be because of 400d1ed ?
KubernetesDeclarativeAgentScript.groovy probably needs to get an update then.

@abayer
Copy link
Member Author

abayer commented Jun 14, 2017

Yup, ClassLoaderReflectionToolkit._findResource doesn't work with PluginFirstClassLoader. Because PluginFirstClassLoader doesn't have a findResource method!

@carlossg
Copy link
Contributor

carlossg commented Jun 14, 2017

pluginFirstClassLoader was added because some dependencies of kubernetes-client were already in core in older versions

@berni2288
Copy link

berni2288 commented Jun 19, 2017

Thanks for the fix!

@diewuq
Copy link

diewuq commented Sep 12, 2017

error: Jenkins doesn’t have label mypod

This error still occur in kubernetes 1.0 with jenkins 2.60.2/2.60.3. Could you help a look at this issue?

@carlossg
Copy link
Contributor

Read https://github.com/jenkinsci/kubernetes-plugin#declarative-pipeline

Declarative Pipeline support requires Jenkins 2.66+

@diewuq
Copy link

diewuq commented Sep 12, 2017

@carlossg Thanks for your advice, now I try with jenkins 2.78, at first there is no label issue, jenkins-slave aways offline, I expose the port like below:
jenkins url: http://168.*.*.131:30384/ jenkins tunnel: http://168.*.*.131:31524, the offline issue still go on;
and then another try, the "Jenkins doesn’t have label mypod" occur again.
Any ideas?

script.checkout script.scm
}
}
script.container(describable.containerTemplate.name) {
Copy link
Contributor

@marvinthepa marvinthepa Oct 10, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not sure that wrapping every call in container(describable.containerTemplate.name){ } is such a good idea.

All commands are executed inside the one single container template that can be specified, while the pod will at least contain a 'jnlp' container in addition. For users to execute scripts in the jnlp container, they need to specify it:

step {
   container('jnlp') {
      sh 'echo run on the slave'
   }
}

Also, the current example contains an(other) explicit container, see

container('maven') {
.

I would vote for removing the container here, making the default "execution location" the jnlp container (like it is for non-declarative pipeline). That will also enable us to add support for multiple containerTemplates in declarative pipeline.

Or was it a conscious decision to keep the declarative syntax simpler and therefore less powerful?

@carlossg what do you think?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just saw #127 (comment), i.e. multiple containerTemplates may not make sense..

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe I know the answer (from following the above thread) but does the current declarative support for this plugin support podTemplate inheritance? Such that you can specify X number of containers you want in the POD? (I think the answer is no).

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

multiple containers are now possible with the yaml syntax

@vaibhavjain882
Copy link

I am getting "Jenkins doesn’t have label mypod" though it run when I did initial setup. Any Idea?

@EamonZhang
Copy link

@snackycracky
Copy link

@marvinthepa @abayer why wouldn't multiple containerTemplates not make sense ? Will you support volumes in the near future ? Did you think about this as you said in #127 (comment) ?

@chetanddesai
Copy link

@snackycracky @marvinthepa @abayer I'm running into the same issue of wanting to run multiple containerTemplates in a declarative pipeline. Something like the ability to have podTemplate support in the agent configuration. I'm trying to do something simple like getting both maven and node on the same slave. It seems declarative pipeline is more future-proof so shouldn't we make sure everything is declarative first?

@carlossg
Copy link
Contributor

[JENKINS-50533] Support multiple containers in declarative pipeline is implemented in #306

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.