Feature: add option --disable-apt-cacher (following #249) #253

AbcSxyZ · 2021-09-16T21:34:12Z

Following #249 from @micaelmalta, this PR creates the option --disable-apt-cacher,
to disable the use of apt cacher for dependencies cache.

It also adds some documentation on the README about the option. Documentation would need at least an update for none ubuntu/debian distros.

…ocker

AbcSxyZ · 2021-09-18T18:16:14Z

Do not work on lxc, within the container, source.list is done between following lines :

gitian-builder/libexec/make-clean-vm

Line 66 in 32b1145

on-target -u root bash < target-bin/bootstrap-fixup

The file bootstrap-fixup.in contain source.list configuration, and expect to use a mirror.

Fix needed

AbcSxyZ · 2021-09-18T20:36:41Z

Just added some code to configure lxc container apt/source.list to avoid using mirror.

Within a running container, I ended up with :

deb http://archive.ubuntu.com/ubuntu trusty main universe
deb http://security.ubuntu.com/ubuntu trusty-security main universe
deb http://archive.ubuntu.com/ubuntu trusty-updates main universe

instead of :

deb http://$MIRROR_HOST:PORT/archive.ubuntu.com/ubuntu trusty main universe
deb http://$MIRROR_HOST:PORT/security.ubuntu.com/ubuntu trusty-security main universe
deb http://$MIRROR_HOST:PORT/archive.ubuntu.com/ubuntu trusty-updates main universe

Actually, it's removing references to apt cacher server, but I'm unable to test it. It looks like having lxc container connected to Internet is a bit tricky, I crash because of the internet connection on apt commands.

I'm not sure if lxc containers are more used without internet, or packages installed when building base, I don't know the software and ignore how a configuration like that should work...
I'm trying to use lxc, not necessary need it but would be great, but I'm unable to do it for now

AbcSxyZ · 2021-09-19T21:16:33Z

Add bug fix for apt cacher configuration within the Dockerfile, see following comment for details.

AbcSxyZ · 2021-09-19T22:37:18Z

libexec/config-bootstrap-fixup

+if [ $APT_CACHER = "1" ]; then
+    sed "s;HOSTIP;$MIRROR_HOST;g" < target-bin/bootstrap-fixup.in > target-bin/bootstrap-fixup
+else
+    sed "s;HOSTIP:3142/;;g" < target-bin/bootstrap-fixup.in > target-bin/bootstrap-fixup
+fi


This is actually not working.

config-boostrap-fixup is called with bin/make-base-vm, which have APT_CACHER variable through --disable-apt-cacher.
It is called also with bin/gbuild, who use it with make-clean-vm. At this moment, they do not have APT_CACHER variable anymore.

Creating the following error :

bash: [: =: unary operator expected

APT_CACHER will be replaced by an unquoted empty value.

As a possible solution, would it be possible to configure bootstrap-fixup.in only when using make-base-vm ? And then reuse generated bootstrap-fixup file instead of recreating one in make-clean-vm.

The feature was implemented by commit 5785dfc, I suppose we can do it as I suggest in the case GITIAN_HOST_IP doesn't change between make-base-vm & gbuild.
Actually, without this situation, it looks like it's creating twice the same file.

It can make sure configuration is more done through make-base-vm.
Maybe deleting the following line can do the job (?) :

gitian-builder/libexec/make-clean-vm

Line 65 in 4609325

libexec/config-bootstrap-fixup

Not sure if it's the right solution, but if it is I just added the commit d8d370d to implement this fix.

AbcSxyZ · 2021-09-20T17:24:12Z

bin/make-base-vm

@@ -193,12 +199,19 @@ if [ $DOCKER = "1" ]; then
    base_image="$DISTRO:$SUITE"
  fi

+  apt_cacher=""


Variable declaration useless

AbcSxyZ · 2021-10-23T15:43:16Z

Hi 👋,

I wanted to know if you are planning to review this PR ? You were really reactive for the feature, but I do not have any reply since September.

Actually, the PR need tests for LXC & KVM, both with and without --disable-apt-cacher + I had a question in a previous comment about the implementation. If anyone can test it with his build, I wasn't able to install/use lxc & kvm...

We did it while working on Dogecoin, there is a pending PR reworking the gitian process and it uses this feature, would be great to know what you want to do 🙃

devrandom · 2021-11-02T18:28:58Z

I do intend to review, but have to allocate some time for testing with the different configurations...

devrandom

I found a couple of issues right away.

devrandom · 2021-11-22T09:16:49Z

libexec/config-bootstrap-fixup

@@ -12,4 +12,8 @@ if [ -z "$MIRROR_HOST" ] || [ "$MIRROR_HOST" == "127.0.0.1" ]; then
  MIRROR_HOST=$GITIAN_HOST_IP
 fi

-sed "s;HOSTIP;$MIRROR_HOST;g" < target-bin/bootstrap-fixup.in > target-bin/bootstrap-fixup
+if [ $APT_CACHER = "1" ]; then


two issues:

missing quotes around $APT_CACHER

APT_CACHER is not actually exported as an environment variable

devrandom · 2021-11-22T09:23:34Z

Thinking about this further, and given that Bitcoin is moving to a different build system, I am planning to move this repo to maintenance mode. Only serious bugs and security issues will be considered going forward.

AbcSxyZ · 2021-11-22T11:39:14Z

Can you tell me what is this new build system, please ? I do not find information related to this change, in the current gitian docs from bitcoin-core repo is still related to your repo, and it's redirecting to a gitian-build.py on bitcoin/bitcoin which do not exists anymore.

EDIT: saw your readme indicating guix, thx.

micaelmalta and others added 5 commits July 13, 2021 23:42

Docker apt-cacher

f9aa177

make-base-vm: switch apt cacher option, using cacher enabled by default

0b10618

make-base-vm: allow disabling apt cacher for lxc & kvm

f9a993e

apt-cacher: reset default mirror and avoid use of host network with d…

f58f681

…ocker

apt-cacher: add readme doc for --disable-apt-cacher

68dd756

AbcSxyZ requested a review from devrandom as a code owner September 16, 2021 21:34

apt-cacher: avoid use of host network with docker

7568db8

AbcSxyZ changed the title ~~Feature: disable apt cacher default (following #249)~~ Feature: add option --disable-apt-cacher (following #249) Sep 17, 2021

apt-cache: avoid use mirror with lxc

3793b65

AbcSxyZ mentioned this pull request Sep 19, 2021

Docs4Doge: update of gitian builder script + documentation (follow #2396) dogecoin/dogecoin#2579

Closed

apt-cacher: fix configuration error of the Dockerfile

525e33a

AbcSxyZ commented Sep 19, 2021

View reviewed changes

apt-cacher: avoid configuration of bootstrap-fixup.in in make-clean-vm

d8d370d

AbcSxyZ commented Sep 20, 2021

View reviewed changes

AbcSxyZ mentioned this pull request Nov 2, 2021

Open items for 1.14.5 release dogecoin/dogecoin#2655

Closed

27 tasks

devrandom reviewed Nov 22, 2021

View reviewed changes

devrandom closed this Nov 22, 2021

AbcSxyZ mentioned this pull request Nov 27, 2021

gitian-builder: add patches to disable apt-cacher dogecoin/dogecoin#2723

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Feature: add option --disable-apt-cacher (following #249) #253

Feature: add option --disable-apt-cacher (following #249) #253

Uh oh!

AbcSxyZ commented Sep 16, 2021 •

edited

Loading

Uh oh!

AbcSxyZ commented Sep 18, 2021

Uh oh!

AbcSxyZ commented Sep 18, 2021 •

edited

Loading

Uh oh!

AbcSxyZ commented Sep 19, 2021

Uh oh!

AbcSxyZ Sep 19, 2021

Uh oh!

AbcSxyZ Sep 20, 2021 •

edited

Loading

Uh oh!

AbcSxyZ Sep 20, 2021 •

edited

Loading

Uh oh!

AbcSxyZ Sep 20, 2021

Uh oh!

AbcSxyZ commented Oct 23, 2021 •

edited

Loading

Uh oh!

devrandom commented Nov 2, 2021

Uh oh!

devrandom left a comment

Uh oh!

devrandom Nov 22, 2021

Uh oh!

devrandom commented Nov 22, 2021

Uh oh!

AbcSxyZ commented Nov 22, 2021 •

edited

Loading

Uh oh!

Uh oh!

Feature: add option --disable-apt-cacher (following #249) #253

Feature: add option --disable-apt-cacher (following #249) #253

Uh oh!

Conversation

AbcSxyZ commented Sep 16, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

AbcSxyZ commented Sep 18, 2021

Uh oh!

AbcSxyZ commented Sep 18, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

AbcSxyZ commented Sep 19, 2021

Uh oh!

AbcSxyZ Sep 19, 2021

Choose a reason for hiding this comment

Uh oh!

AbcSxyZ Sep 20, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

AbcSxyZ Sep 20, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

AbcSxyZ Sep 20, 2021

Choose a reason for hiding this comment

Uh oh!

AbcSxyZ commented Oct 23, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

devrandom commented Nov 2, 2021

Uh oh!

devrandom left a comment

Choose a reason for hiding this comment

Uh oh!

devrandom Nov 22, 2021

Choose a reason for hiding this comment

Uh oh!

devrandom commented Nov 22, 2021

Uh oh!

AbcSxyZ commented Nov 22, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

AbcSxyZ commented Sep 16, 2021 •

edited

Loading

AbcSxyZ commented Sep 18, 2021 •

edited

Loading

AbcSxyZ Sep 20, 2021 •

edited

Loading

AbcSxyZ Sep 20, 2021 •

edited

Loading

AbcSxyZ commented Oct 23, 2021 •

edited

Loading

AbcSxyZ commented Nov 22, 2021 •

edited

Loading