Capture coverage failing when using docker

[kenjones-cisco]

Description

When attempting to publish coverage.xml, report.xml, and test.xml to SonarQube after generating the individual xml files, I'm seeing the following happen.

1. The coverage is not honoring the sonar.exclusions entry, as all the files in my .glide/** and vendor/** are being processed.
2. unable to create InputFile object: gitscm.cisco.com/ccdev/cicdtest/version/info.go
3. java.lang.StringIndexOutOfBoundsException: String index out of range: -1

Steps to Reproduce

1. Build Dockerfile for running SonarQube locally

FROM sonarqube:6.5-alpine

RUN wget -O "$SONARQUBE_HOME/extensions/plugins/sonar-golang-plugin-1.2.11-rc3.jar" \
      https://github.com/uartois/sonar-golang/releases/download/v1.2.11-rc3/sonar-golang-plugin-1.2.11-rc3.jar

docker build -t cicdtest_sonar -f Dockerfile.cd .

2. Start SonarQube

docker run -d --name sonarqube -p 9000:9000 -p 9092:9092 cicdtest_sonar

3. Run sonar-scanner

sonar-scanner -X \
  -Dsonar.host.url=http://localhost:9000 \
  -Dsonar.login=534221ba8db9121b9afe15a322a5fa1f17d3c9df

Content of your sonar-project.properties

https://gist.github.com/kenjones-cisco/8b5f55f4f5197d3ddfba66142aeefbc1#file-sonar-project-properties

It will help us to check how you configured the plugin.

Log of sonar-scanner related to the plugin

https://gist.github.com/kenjones-cisco/8b5f55f4f5197d3ddfba66142aeefbc1#file-scanner-log

The plugin logs major events, including which report files have been found and analyzed.

Versions

SonarQube Server

            "Env": [
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/lib/jvm/java-1.8-openjdk/jre/bin:/usr/lib/jvm/java-1.8-openjdk/bin",
                "LANG=C.UTF-8",
                "JAVA_HOME=/usr/lib/jvm/java-1.8-openjdk",
                "JAVA_VERSION=8u131",
                "JAVA_ALPINE_VERSION=8.131.11-r2",
                "SONAR_VERSION=6.5",
                "SONARQUBE_HOME=/opt/sonarqube",
                "SONARQUBE_JDBC_USERNAME=sonar",
                "SONARQUBE_JDBC_PASSWORD=sonar",
                "SONARQUBE_JDBC_URL="
            ],

sonar-scanner

INFO: Scanner configuration file: C:\Users\kenjones\.local\bin\sonar-scanner-3.0.3.778\bin\..\conf\sonar-scanner.properties
INFO: Project root configuration file: C:\Users\kenjones\projects\examples\cicdtest\sonar-project.properties
INFO: SonarQube Scanner 3.0.3.778
INFO: Java 1.8.0_131 Oracle Corporation (64-bit)
INFO: Windows 10 10.0 amd64

go version
go version go1.9 linux/amd64

gometalinter
c039b9f1fbff9a38595e19e9d97474ed57270cb8 Last commit as of yesterday

OS

PRETTY_NAME="Debian GNU/Linux 9 (stretch)"
NAME="Debian GNU/Linux"
VERSION_ID="9"
VERSION="9 (stretch)"
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

All go, gometalinter, etc. are run inside of Docker containers

Versions of your sonar installation (server, sonar-scanner), gometalinter, go and OS.

Additional Information

All *.xml are in the same gist as the other files:
https://gist.github.com/kenjones-cisco/8b5f55f4f5197d3ddfba66142aeefbc1

Seems similar to #14

Sample reports may be necessary to reproduce the issue. Please attach if possible reports files (report.xml, coverage.xml, test.xml) corresponding to the feature which is not working as expected.

[kenjones-cisco]

To be completely clear.
SonarQube is running inside of a container.
All development tasks are executed inside of a container.
The only command running locally is sonar-scanner.

Once SonarCube is upgraded to within my CI/CD platform. SonarQube is run on a VM. Jenkins provides the sonar-scanner to integrate with SonarQube, but all the go related tasks will happen in the exact same container that I'm using locally (provides the abstract layer that avoids the "it works on my laptop", as Docker runs on linux, MacOS, and Windows).

[kenjones-cisco]

Just in case it is helpful or relevant, the development container is defined as follows:

FROM golang:1.9

RUN apt-get update && apt-get install -y --no-install-recommends \
        gettext-base \
        zip \
    && rm -rf /var/lib/apt/lists/*

ENV YAML_BIN_VERSION 1.13.1
ENV JQ_BIN_VERSION 1.5

RUN wget -O /usr/local/bin/jq "https://github.com/stedolan/jq/releases/download/jq-${JQ_BIN_VERSION}/jq-linux64" \
    && chmod 755 /usr/local/bin/jq \
    && wget -O /usr/local/bin/yaml "https://github.com/mikefarah/yaml/releases/download/${YAML_BIN_VERSION}/yaml_linux_amd64" \
    && chmod 755 /usr/local/bin/yaml

ARG DEVELOPER

RUN git config --global user.name "${DEVELOPER}" \
    && git config --global user.email "${DEVELOPER}@cisco.com" \
    && git config --list

ENV GLIDE_VERSION v0.12.3

RUN go get github.com/alecthomas/gometalinter \
    && gometalinter --install --vendor \
    && go get golang.org/x/tools/cmd/goimports \
    && go get github.com/mitchellh/gox \
    && go get github.com/davecheney/godoc2md \
    && go get github.com/axw/gocov/gocov \
    && go get github.com/t-yuki/gocov-xml \
    && go get github.com/tebeka/go2xunit \
    && go get github.com/go-swagger/go-swagger/cmd/swagger \
    && curl -Lo /tmp/glide.tgz https://github.com/Masterminds/glide/releases/download/$GLIDE_VERSION/glide-$GLIDE_VERSION-linux-amd64.tar.gz \
    && tar -C /usr/bin -xzf /tmp/glide.tgz --strip=1 linux-amd64/glide \
    && rm /tmp/glide.tgz

ENV CGO_ENABLED 0
ENV GOPATH /go:/cicdtest

[danielleberre]

1. happens because we manually look for XML cover files in the project subdirectory in the plugin.

@thibaultfalque we need to retrieve the sonar.exclusions property and prevent our search to enter those directories.

[danielleberre]

We just added the code to prevent files in the exclusion path to be analyzed. The next RC will allow to check that it fixes your problem.

[thibaultfalque]

Please check release 1.2.11-RC8.

[thibaultfalque]

@kenjones-cisco could you check with the last release and close the issue if the problem is resolved ?

[kenjones-cisco]

Issues 1 and 2 still exist

Even though I have the following defined:
sonar.exclusions=vendor/**,.git/**,.glide/**,**/*.xml
It still keeps scanning the vendor/, .git/, and the .glide/ directories.

And when the coverage file is generated the filename inside the coverage.xml shows up as gitscm.cisco.com/ccdev/cicdtest/version/info.go since it includes the package. But given the file is actually in the local directory as the top level package is "virtual" due to the use of docker, the error unable to create InputFile object: gitscm.cisco.com/ccdev/cicdtest/version/info.go is still happening.

[danielleberre]

@thibaultfalque we need to check if sonar.exclusions is properly honored in each sensor.

Regarding the second issue, I have really no idea how to fix it since I am not a regular docker user. Any idea would be welcome.

[kenjones-cisco]

@danielleberre I wondering if there is an assumption about what file paths look like for a golang code base.

Most have their source control within github which results in a base import package of github.com/<username|orgname>/<project_name>. When cloning a project from github the code is by default in a directory named <project_name>.

As such if your code base has multiple packages you will have to provide the full import path. Therefore when you build, test, etc. The code needs to appear as if it is in src/<import_path> (ex. src/github.com/uartois/sonar-golang).

Therefore when mounting the code base into a docker container, it will be mounted on a path that looks like /opt/src/<import_path>. As such when the coverage file is generated the full package name is captured under package name="" and then the file name looks like <package_name>/<file>.go

As such I have tried running sonar-scanner inside a container with my source code mounted in a similar way with the sonar-project.properties file at the root of my project, but even then it seems to be unable to match the filename path to the actual files.

It seems as if sonar-scanner expects to be run from src/ and then walks through down to the code located inside of <import_path>. That of course resulted in a huge problem because sonar-scanner wanted to walk through every directory within src/ even if I specified in the sonar-project.properties that the source is located at <import_path> or tried to exclude every directory within src/ except for the one I wanted scanned. As a result the lack of exclusion resulted in 6 or 7 projects with their own vendor/ directories being scanned and uploading 28MB of results but because the data set for a single project was now so large it was difficult to determine if the file was picked properly or not.