Problem building

[NicoHood]

With version 1.3.x the folder structure was arduino-builder/src/arduino.cc/arduino-builder/main.go

Now the src folder is missing, which makes go unable to find the module when downloading the signed tar archive and building this locally. I am no go expert, but I need to move all files into a new src folder. Could you please give me another solution? You also compile offline, so there must be some other way I assume?

==> Starting build()...
src/arduino.cc/arduino-builder/main.go:47:2: cannot find package "github.com/arduino/arduino-builder" in any of:
	/usr/lib/go/src/github.com/arduino/arduino-builder (from $GOROOT)
	/build/arduino-builder/src/build/src/github.com/arduino/arduino-builder (from $GOPATH)
src/arduino.cc/arduino-builder/main.go:48:2: cannot find package "github.com/arduino/arduino-builder/gohasissues" in any of:
	/usr/lib/go/src/github.com/arduino/arduino-builder/gohasissues (from $GOROOT)
	/build/arduino-builder/src/build/src/github.com/arduino/arduino-builder/gohasissues (from $GOPATH)
src/arduino.cc/arduino-builder/main.go:49:2: cannot find package "github.com/arduino/arduino-builder/grpc" in any of:
	/usr/lib/go/src/github.com/arduino/arduino-builder/grpc (from $GOROOT)
	/build/arduino-builder/src/build/src/github.com/arduino/arduino-builder/grpc (from $GOPATH)
src/arduino.cc/arduino-builder/main.go:50:2: cannot find package "github.com/arduino/arduino-builder/i18n" in any of:
	/usr/lib/go/src/github.com/arduino/arduino-builder/i18n (from $GOROOT)
	/build/arduino-builder/src/build/src/github.com/arduino/arduino-builder/i18n (from $GOPATH)
src/arduino.cc/arduino-builder/main.go:51:2: cannot find package "github.com/arduino/arduino-builder/types" in any of:
	/usr/lib/go/src/github.com/arduino/arduino-builder/types (from $GOROOT)
	/build/arduino-builder/src/build/src/github.com/arduino/arduino-builder/types (from $GOPATH)
src/arduino.cc/arduino-builder/main.go:52:2: cannot find package "github.com/arduino/arduino-builder/utils" in any of:
	/usr/lib/go/src/github.com/arduino/arduino-builder/utils (from $GOROOT)
	/build/arduino-builder/src/build/src/github.com/arduino/arduino-builder/utils (from $GOPATH)
==> ERROR: A failure occurred in build().
    Aborting...
==> ERROR: Build failed, check /var/lib/archbuild/extra-x86_64/arch/build
[arch@talloniv trunk]$ ls /var/lib/archbuild/extra-x86_64/arch/build/arduino-builder/src/build/src/arduino.cc/arduino-builder/
main.go

build() {
	mkdir -p "${srcdir}/build/src/arduino.cc"
	cp -a "${srcdir}/${pkgname}-${pkgver}/." "${srcdir}/build/src/arduino.cc"
	export GOPATH="${srcdir}/build"
	cd "${srcdir}/build"
	# go get -fix github.com/go-errors/errors
	# go get -fix github.com/stretchr/testify
	# go get -fix github.com/jstemmer/go-junit-report
	# go build arduino.cc/arduino-builder
	#
	go get github.com/go-errors/errors
	go get github.com/stretchr/testify
	go get github.com/jstemmer/go-junit-report
	go get -u github.com/arduino/go-properties-map
	go get -u github.com/arduino/go-timeutils
	go get google.golang.org/grpc
	go get github.com/golang/protobuf/proto
	go get golang.org/x/net/context
	go get github.com/fsnotify/fsnotify
	#go get github.com/arduino/arduino-builder
	go build arduino.cc/arduino-builder

	# TODO Fix ctags test environment at build time
	# https://github.com/arduino/arduino-builder/issues/202
}

[facchinm]

Hi Nico, the new structure is the one needed to make the project a go-gettable project. Incidentally, this makes building supereasy.

mkdir arduino-builder-new && cd  arduino-builder-new
export GOPATH=$PWD
go get github.com/arduino/arduino-builder/arduino-builder

The last command retrieves all the dependencies and leaves the final binary in bin where you can grab it

[NicoHood]

But it will download the arduino-builder from github directly, right? I do not want to let the tool download the binaries, because then they are not validated by gpg.

[facchinm]

The last command does this:

• creates src folder
• clones github.com/arduino/arduino-builder (master branch)
• clones all dependencies (as git repos)

From there you can, for example, cd into src/github.com/arduino/arduino-builder and checkout the last validated tag.
When done, running go build github.com/arduino/arduino-builder/arduino-builder will recreate the binary in bin folder based on the actual tag.

[NicoHood]

Checking a tag of a software downloaded (go get) repository involves manual gpg validation, which is not recommended for the build system. I'd like to copy the data from the release inside the src folder, as shown above. However that is still not functional. Any ideas why? The copying should similar to your recommended git tag checkout.

Beside this, using only git master branches for the dependencies will very likely not give use reproducible builds and other problems. I do not know go good enough but I see this as potential problem.

[facchinm]

Reproducible builds is in fact a huge problems for go ecosystem that is being addressed by go dep in recent versions; however, simply recompiling with go1.10 is not enough since we need to add a dependency file and some other stuff. We would be glad to accept such a pull request 😉

[NicoHood]

I am sorry, I aint got time to work into that. I cannot get the build running and gave up frustrated for now.

[cmaglie]

Doing what is written in README doesn't work for you?
https://github.com/arduino/arduino-builder#building-from-source

About reproducible builds... everything is in GOPATH, exactly as before. (maybe having the GOPATH set to the repository folder gave you the impression to have a more reproducible build? technically, nothing changed from before... you can always set your GOPATH to everywhere you want, even a temporary empty folder).

I cannot get the build running and gave up frustrated for now.

As @facchinm said, golang has just deployed modules tooling with go1.11 (for now is an experimental OPT-IN feature) that should solve all the reproducible build issues and we are going to adopt it soon so... brace yourself, the build steps will change again :-)

[NicoHood]

From my understand I need to skip the step:

go get github.com/arduino/arduino-builder

Because I want to use a local archive that i downloaded and validated with gpg. However I am unable to place it into the right/same directory as the go command would have done it.

[cmaglie]

You can pgp-validate with git also:

~$ export GOPATH=`mktemp -d`
~$ echo $GOPATH
/tmp/tmp.3r0XnE4FEB
~$ go get github.com/arduino/arduino-builder
~$ cd $GOPATH/src/github.com/arduino/arduino-builder
/tmp/tmp.3r0XnE4FEB/src/github.com/arduino/arduino-builder$ git checkout 1.4.1
Note: checking out '1.4.1'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:

  git checkout -b <new-branch-name>

HEAD is now at f9d69df Release 1.4.1
$ git verify-tag 1.4.1 --raw -v
object f9d69df69fea02a73d3a73647505f216447a19e2
type commit
tag 1.4.1
tagger Arduino Packages <support@arduino.cc> 1536570220 +0200

Release 1.4.1
[GNUPG:] NEWSIG
[GNUPG:] KEY_CONSIDERED 326567C1C6B288DF32CB061A95FA6F43E21188C4 0
[GNUPG:] SIG_ID 296ULSz0vD21BR2MLGgJYPkXtqg 2018-09-10 1536570226
[GNUPG:] KEY_CONSIDERED 326567C1C6B288DF32CB061A95FA6F43E21188C4 0
[GNUPG:] GOODSIG 95FA6F43E21188C4 Arduino Packages <support@arduino.cc>
[GNUPG:] VALIDSIG 326567C1C6B288DF32CB061A95FA6F43E21188C4 2018-09-10 1536570226 0 4 0 1 10 00 326567C1C6B288DF32CB061A95FA6F43E21188C4
[GNUPG:] TRUST_UNDEFINED 0 pgp
[GNUPG:] VERIFICATION_COMPLIANCE_MODE 23
/tmp/tmp.3r0XnE4FEB/src/github.com/arduino/arduino-builder$ echo $?
0

This will also reduce the pressure on our side (one script less to maintain to publish the arduino-builder releases files with checksum and signature for each).

[NicoHood]

I finally got the build working. I started again a week later with fresh energy. I was really frustrated, sorry for that. And thanks for all the great work! I am really straight forward in my issue reports/wording, but I really appreciate all your help! Just wanted to clarify that, because the cooperation/help from you is really helpful! :)