make: pkg: introduce git-cache #5112

kaspar030 · 2016-03-20T12:11:09Z

Currently, every package get's checked out from remote repositories on every build, sometimes multiple times.

This PR adds support for git-cache [1], transparently using a local copy of the requested SHA1, or reverting to the remote location if the cache doesn't exist, fails, or doesn't have the commit.

This should not affect anything unless git-cache is actually used.

In order to make use of the cache, install the git-cache binary into path.
Currently, repositories have to be added to the cache manually by executing
# git cache init once per user, then # git cache add <name> <url> once per repository.

git cache update fetches new commits from cached remote repos.

[1] https://github.com/kaspar030/git-cache

kaspar030 · 2016-03-20T12:13:27Z

One thing I don't know how to handle nicely is the git exec path. git picks up anything starting with "git-" as command, so in order for the build system to pick up git-cache, it has to be in PATH or GIT_EXEC_PATH has to point to it. I opted for the latter, but then git cannot find it's "https" remote helper, so currently I'm unsetting GIT_EXEC_PATH within git-cache, which seems hacky.
The other option would be to change PATH, which I somehow don't like either.
Ideas?

kaspar030 · 2016-03-20T12:15:15Z

An open question would be if the build should still fail if the cache cannot be updated, as that might mean that a remote repository is actually dead.

kaspar030 · 2016-03-20T12:26:14Z

One thing I don't know how to handle nicely is the git exec path

Seems like in docker things are even more complicated.

kaspar030 · 2016-03-20T12:40:52Z

Ok, modifying PATH is the only way I see.

kaspar030 · 2016-03-20T12:41:15Z

squashed WIP commits

OlegHahm · 2016-03-20T13:10:18Z

Currently, every package get's checked out from remote repositories on every build, sometimes multiple times.

For the CI, you mean?

OlegHahm · 2016-03-20T13:11:58Z

Makefile.include

@@ -20,6 +20,10 @@ RIOTPKG ?= $(RIOTBASE)/pkg
 RIOTPROJECT ?= $(shell git rev-parse --show-toplevel 2>/dev/null || pwd)
 RIOTPROJECT := $(abspath $(RIOTPROJECT))

+# add git tools directory to PATH so git can pick it up
+PATH:=$(PATH):$(RIOTBASE)/dist/tools/git
+export PATH


I'm not sure I would like it if a build script would modify my PATH variable. Maybe you can store the old value and restore it after the build?

Also: Wouldn't this line append the additional path on every run?

The export is only valid for sub-processes if make.

-----Original Message-----
From: Oleg Hahm notifications@github.com
To: RIOT-OS/RIOT RIOT@noreply.github.com
Cc: Kaspar Schleiser kaspar@schleiser.de
Sent: So., 20 März 2016 14:12
Subject: Re: [RIOT] make: pkg: introduce git-cache (#5112)

@@ -20,6 +20,10 @@ RIOTPKG ?= $(RIOTBASE)/pkg
RIOTPROJECT ?= $(shell git rev-parse --show-toplevel 2>/dev/null || pwd)
RIOTPROJECT := $(abspath $(RIOTPROJECT))

+# add git tools directory to PATH so git can pick it up
+PATH:=$(PATH):$(RIOTBASE)/dist/tools/git
+export PATH

I'm not sure I would like it if a build script would modify my PATH variable. Maybe you can store the old value and restore it after the build?

You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub:
https://github.com/RIOT-OS/RIOT/pull/5112/files/8b363905f6854a7f841f884bf5b6e71c4010782a#r56764211

OlegHahm · 2016-03-20T13:16:07Z

Ok, modifying PATH is the only way I see.

Couldn't you replace the calls with absolute paths?

kaspar030 · 2016-03-20T14:40:35Z

yep.

-----Original Message-----
From: Oleg Hahm notifications@github.com
To: RIOT-OS/RIOT RIOT@noreply.github.com
Cc: Kaspar Schleiser kaspar@schleiser.de
Sent: So., 20 März 2016 14:10
Subject: Re: [RIOT] make: pkg: introduce git-cache (#5112)

Currently, every package get's checked out from remote repositories on every build, sometimes multiple times.

For the CI, you mean?

You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHub:
#5112 (comment)

kaspar030 · 2016-03-20T17:46:42Z

Couldn't you replace the calls with absolute paths?

Yeah, excellent idea!

kaspar030 · 2016-03-20T18:44:04Z

@OlegHahm Better? Never have failed builds because of unreachable remote repositories anymore! :)

OlegHahm · 2016-03-20T19:35:13Z

👍

kaspar030 · 2016-03-20T19:48:10Z

squashed

cgundogan · 2016-03-21T14:27:17Z

dist/tools/git/git-cache

+        git clone "${REMOTE}" "${TARGET_PATH}"
+        git -C "${TARGET_PATH}" checkout $SHA1
+    fi
+}


OlegHahm · 2016-03-21T16:43:28Z

@kaspar030, it's not clear to me how I can test this locally. I tried with the microcoap_server example: building it, then disconnect my computer from the network and try to build again, but it failed. I would have assumed that the git-cache clone command would handle this.

kaspar030 · 2016-03-21T17:24:21Z

@OlegHahm did you see the instructions in the beginning of the pr?

put git-cache in path
run 'git cache init'
run 'it cache add microcoap microcoap url
run 'git cache update'
can be skipped if you call the script with its full path.

OlegHahm · 2016-03-21T17:30:21Z

No I didn't. From looking at the diff it was clear to me if this information was still up to date and I couldn't figure if I would have to run this from pkg or $RIOTBASE.

run 'git cache add microcoap

Isn't there the URL parameter missing?

OlegHahm · 2016-03-21T17:52:02Z

Okay, got it. Can you add these instructions to the README? ACK once this is addressed.

kaspar030 · 2016-03-21T20:25:37Z

addressed comments, immediately squashed

make: pkg: introduce git-cache

kaspar030 added the State: WIP State: The PR is still work-in-progress and its code is not in its final presentable form yet label Mar 20, 2016

kaspar030 force-pushed the use_git_cache branch from 9d5da00 to 8b36390 Compare March 20, 2016 12:39

kaspar030 removed the State: WIP State: The PR is still work-in-progress and its code is not in its final presentable form yet label Mar 20, 2016

kaspar030 assigned jnohlgard Mar 20, 2016

OlegHahm reviewed Mar 20, 2016
View reviewed changes

kaspar030 assigned OlegHahm and unassigned jnohlgard Mar 20, 2016

kaspar030 force-pushed the use_git_cache branch from 8d16f11 to 12a1c5e Compare March 20, 2016 19:47

kaspar030 removed the Discussion: RFC The issue/PR is used as a discussion starting point about the item of the issue/PR label Mar 20, 2016

kaspar030 mentioned this pull request Mar 21, 2016

pkg: add support for git reference repository #5074

Closed

cgundogan reviewed Mar 21, 2016
View reviewed changes

dist/tools/git/git-cache

git clone "${REMOTE}" "${TARGET_PATH}"

git -C "${TARGET_PATH}" checkout $SHA1

fi

}

Copy link

Member

cgundogan Mar 21, 2016

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

set +ex

kaspar030 added 2 commits March 21, 2016 21:22

dist: tools: add git-cache

7726572

pkg: adapt to use git-cache

812da2f

kaspar030 force-pushed the use_git_cache branch from 12a1c5e to 812da2f Compare March 21, 2016 20:25

OlegHahm added a commit that referenced this pull request Mar 21, 2016

Merge pull request #5112 from kaspar030/use_git_cache

16a3f13

make: pkg: introduce git-cache

OlegHahm merged commit 16a3f13 into RIOT-OS:master Mar 21, 2016

This was referenced Mar 21, 2016

pkg: u8glib: add initial support for graphics library #5126

Closed

git-cache: problem when building for multiple platforms #5130

Closed

pkg: initial import of emb6 stack #4713

Merged

kaspar030 deleted the use_git_cache branch February 7, 2017 22:15

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

make: pkg: introduce git-cache #5112

make: pkg: introduce git-cache #5112

kaspar030 commented Mar 20, 2016

kaspar030 commented Mar 20, 2016

kaspar030 commented Mar 20, 2016

kaspar030 commented Mar 20, 2016

kaspar030 commented Mar 20, 2016

kaspar030 commented Mar 20, 2016

OlegHahm commented Mar 20, 2016

OlegHahm Mar 20, 2016

OlegHahm Mar 20, 2016

kaspar030 Mar 20, 2016

OlegHahm commented Mar 20, 2016

kaspar030 commented Mar 20, 2016

kaspar030 commented Mar 20, 2016

kaspar030 commented Mar 20, 2016

OlegHahm commented Mar 20, 2016

kaspar030 commented Mar 20, 2016

cgundogan Mar 21, 2016

OlegHahm commented Mar 21, 2016

kaspar030 commented Mar 21, 2016

OlegHahm commented Mar 21, 2016

OlegHahm commented Mar 21, 2016

kaspar030 commented Mar 21, 2016

make: pkg: introduce git-cache #5112

make: pkg: introduce git-cache #5112

Conversation

kaspar030 commented Mar 20, 2016

kaspar030 commented Mar 20, 2016

kaspar030 commented Mar 20, 2016

kaspar030 commented Mar 20, 2016

kaspar030 commented Mar 20, 2016

kaspar030 commented Mar 20, 2016

OlegHahm commented Mar 20, 2016

OlegHahm Mar 20, 2016

Choose a reason for hiding this comment

OlegHahm Mar 20, 2016

Choose a reason for hiding this comment

kaspar030 Mar 20, 2016

Choose a reason for hiding this comment

OlegHahm commented Mar 20, 2016

kaspar030 commented Mar 20, 2016

kaspar030 commented Mar 20, 2016

kaspar030 commented Mar 20, 2016

OlegHahm commented Mar 20, 2016

kaspar030 commented Mar 20, 2016

cgundogan Mar 21, 2016

Choose a reason for hiding this comment

OlegHahm commented Mar 21, 2016

kaspar030 commented Mar 21, 2016

OlegHahm commented Mar 21, 2016

OlegHahm commented Mar 21, 2016

kaspar030 commented Mar 21, 2016