Error while testing GitLab's Jenkins CI Integration with Jenkins' GitLab Plugin

[radium226]

Hello,

I'm trying to make GitLab 8.7 (using directly https://gitlab.com) and Jenkins 2.0 work together, but I got this error from Jenkins when I test (using the Test button of GitLab's Jenkins CI Service interface):

java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:320)
    at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:163)
    at org.kohsuke.stapler.MetaClass$11.dispatch(MetaClass.java:378)
    at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)
    at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
    at org.kohsuke.stapler.MetaClass$11.dispatch(MetaClass.java:380)
    at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746)
    at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876)
    at org.kohsuke.stapler.Stapler.invoke(Stapler.java:649)
    at org.kohsuke.stapler.Stapler.service(Stapler.java:238)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
    at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1669)
    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:135)
    at hudson.plugins.greenballs.GreenBallFilter.doFilter(GreenBallFilter.java:59)
    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132)
    at hudson.plugins.scm_sync_configuration.extensions.ScmSyncConfigurationFilter$1.call(ScmSyncConfigurationFilter.java:49)
    at hudson.plugins.scm_sync_configuration.extensions.ScmSyncConfigurationFilter$1.call(ScmSyncConfigurationFilter.java:44)
    at hudson.plugins.scm_sync_configuration.ScmSyncConfigurationDataProvider.provideRequestDuring(ScmSyncConfigurationDataProvider.java:106)
    at hudson.plugins.scm_sync_configuration.extensions.ScmSyncConfigurationFilter.doFilter(ScmSyncConfigurationFilter.java:44)
    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:132)
    at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:126)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
    at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:49)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
    at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at jenkins.security.BasicHeaderProcessor.success(BasicHeaderProcessor.java:140)
    at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:82)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
    at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:76)
    at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
    at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
    at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:81)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
    at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1652)
    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:585)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
    at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:553)
    at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:223)
    at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
    at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
    at org.eclipse.jetty.server.Server.handle(Server.java:499)
    at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)
    at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
    at org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
    at winstone.BoundedExecutorService$1.run(BoundedExecutorService.java:77)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.NullPointerException
    at com.dabsquared.gitlabjenkins.webhook.ActionResolver.onPost(ActionResolver.java:93)
    at com.dabsquared.gitlabjenkins.webhook.ActionResolver.resolveAction(ActionResolver.java:53)
    at com.dabsquared.gitlabjenkins.webhook.ActionResolver.resolve(ActionResolver.java:47)
    at com.dabsquared.gitlabjenkins.webhook.GitLabWebHook.getDynamic(GitLabWebHook.java:44)
    ... 73 more

I looked at the source code and it seems that the X-GitLab-Event is missing from the HTTP request.

Do you guys have an idea what going on?

[crussell52]

I am seeing the same behavior with a local jenkins 2.0 install and Gitlab 8.5.5-ee.

Same stacktrace.

Plugin version 1.2.0

[crussell52]

Downgraded the plugin to 1.1.32 in jenkins. Successfully works around this issue.

[4kochi]

Have same the problem. Downgraded to v1.1.32 and now it works fine.

[coder-hugo]

I tested the plugin in Jenkins 2 running in the bundled Jetty and a Tomcat 8 and for both I hadn't trouble that the X-GitLab-Event header is missing. So I assume this is somehow related to your Jenkins/GitLab setup. Can you provide some details of your setup. Are there any systems in between Jenkins and GitLab, e.g. a proxy. Maybe you could try to use a packet sniffer (tcpdump) to see if the HTTP request from GitLab contains this header.

[pyther]

I'm running Gitlab 8.6.4 and I'm not seeing any X-GitLab headers in the request sent from Gitlab -> Jenkins.

On my gitlab server 127.0.0.1:8090 maps -> jenkin2.example.com:8080 which is the raw Jetty server that is bundled with Jenkins.

Request

.POST /project/website/olcf HTTP/1.1
Content-Type: application/json
Authorization: Basic amVua2luczpkOTYzNjk0ODBhNTUxZWRiYTlhYjMwNDZmOTUzYmFhYg==
Connection: close
Host: 127.0.0.1:8090
Content-Length: 2858

{"object_kind":"push", <REDACTED>}

Response

.HTTP/1.1 500 Server Error
Date: Sat, 30 Apr 2016 21:05:12 GMT
X-Content-Type-Options: nosniff
Expires: Thu, 01 Jan 1970 00:00:00 GMT
Cache-Control: no-cache,no-store,must-revalidate
X-Hudson-Theme: default
Content-Type: text/html;charset=UTF-8
Set-Cookie: JSESSIONID.e4f376e5=13uk8h13avml418djq3tme7xmz;Path=/;HttpOnly
X-Hudson: 1.395
X-Jenkins: 2.0
X-Jenkins-Session: 95b0fe99
X-Hudson-CLI-Port: 59437
X-Jenkins-CLI-Port: 59437
X-Jenkins-CLI2-Port: 59437
X-Frame-Options: sameorigin
X-Instance-Identity: MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAntZ0W95stGDFeeuLAATV2ekTLn0/0/x1es0VGCM0J72r4U6DIfY5OHCA1WRobR7g8TgCK7xZ8k3QpI6Z5jTc6S5A2oTZP2bkzZs0G+wi+qvOyXfxGGWLJXnmhb5lYTAembwn66NDOFBejWkVAe2KFySLHUlzyeDdQnZy+gi4fj+9w
fvdBAw73Y8bqHk8RRwCugWjAMonD25GwDrg3XtZWURJNs0bUPRQ41sFYyuFiXt2RcDbs88+cQA7Nwlx4F8ypk7NuIq4dr0sKUFE3HEJ9rQVru2aAT4jZh+nxj8r4aUnH8qvEx19QLa8U2wz2eIaMzWNE53zPBm0eeM2V3AC0wIDAQAB
X-SSH-Endpoint: jenkins2.example.com:39643
Connection: close
Server: Jetty(9.2.z-SNAPSHOT)

[crussell52]

I am actually using apache mod_proxy to deliver over port 80.

If the root cause is indeed a missing header, maybe it is being stripped by apache. I won't be able to examine it until Monday.

[coder-hugo]

I just tested this at work with GitLab EE and the Jenkins CI service. In this case the X-GitLab-Event header is missing for me, too. As a workaround you can configure a webhook instead of the Jenkins CI service in your GitLab.
@dblessing IIRC you were involved in the implementation of the Jenkins CI service in GitLab. Is there any reason why this doesn't send the X-GitLab-Event header?

[alexgrist]

Also experiencing this issue, had to downgrade to v1.1.32. Not using apache or nginx as a reverse proxy, just direct request to jenkins.

[crussell52]

As stated earlier, I am using apache as a reverse proxy to allow access to jenkins on port 80 (for user convenience).

To eliminate this as a variable, I modified my configuration in GitLab to point directly to the jenkins port. This bypasses apache entirely, but did not correct the issue.

Latest test with plugin: v1.2.1

[crussell52]

Recreated once again, this time with:

Jenkins 2.1
Gitlab Jenkins CI Plugin: 1.2.1
Gitlab EE: 8.7.1

I also updated my gitlab configuration to access Jenkins by IP (internal network) to take the internal DNS out of the equation. Like my earlier test, I used my Jenkins port instead of port 80 to take the apache reverse-proxy out of the equation as well.

I got the same results. So I ran tcpdump on my gitlab server as suggested by @coder-hugo.

sudo tcpdump -nnvvASs 1514 dst 1.2.3.4

The relevant portion of the output from the GitLab service's "Test Settings" button:

(The IP has been changed to protect the innocent.)

...i.|..POST /project/<redacted> HTTP/1.1
Content-Type: application/json
Authorization: Basic <redacted>
Connection: close
Host: 1.2.3.4:8081
Content-Length: 5272

This does seem to confirm the absence of a X-GitLab-Event header at the point of origin.

[coder-hugo]

@crussell52 as mentioned here #272 (comment) this seems to be a bug in the GitLab EE Jenkins CI Service. If you add a webhook instead of the Jenkins CI Service the request should contain the X-GitLab-Event header.

[coder-hugo]

Add link to GitLab EE issue

[crussell52]

@coder-hugo Thank you for the followup and for raising the issue with GitLab. :)