Upgrade to 4.1 #3

lapo-luchini · 2023-01-26T15:09:18Z

I have some work-in-progress patches for that.
It includes changes from mvn-pre-fetch branch.

As stated in [Reproducible builds in FreeBSD](https://www.bsdcan.org/2016/schedule/attachments/375_2016-06-11-BSDCan-2016-Reproducible-Builds.pdf) `bsdtar` doesn't allow changing mtime, so I frist added a line to do this: ``` cd ${REPO_DIR} && ${FIND} . -print0 | xargs -0 touch -d 2022-08-08T20:40:04Z ``` This is not enough as file traversal order changes from time to time; to solve this I used `find -s` to produce a sorted file list: ``` cd ${WRKDIR} && \ ${FIND} -s repository/ -print >list.txt && \ ${TAR} czf ${DISTDIR}/apache-${PORTNAME}-${DISTVERSION}-repo.tar.gz -nT list.txt ``` Still not enough, as `gzip` header includes a timestamp; this can be solved by using [`gzip -n`](https://www.freebsd.org/cgi/man.cgi?query=gzip&sektion=1) but instead I tried two approaches and it seems to work fine with both: 1. using `7zz` (always sorts traversal order, so `list.txt` is not needed) `7zz a -mx9 ${DISTDIR}/apache-${PORTNAME}-${DISTVERSION}-repo.7z repository/` 2. using `xz` `${TAR} cJf ${DISTDIR}/apache-${PORTNAME}-${DISTVERSION}-repo.tar.xz -nT list.txt` I decided for the latter as it is included in the base system.

lapo-luchini · 2023-01-27T23:39:11Z

Most differences solved, still need to massage plist file.

===> Checking for items in STAGEDIR missing from pkg-plist
Error: Orphaned: %%ETCDIR%%/credentials.sample
Error: Orphaned: %%DATADIR%%/lib/asm-9.1.jar
Error: Orphaned: %%DATADIR%%/lib/caffeine-2.9.2.jar
Error: Orphaned: %%DATADIR%%/lib/checker-qual-3.10.0.jar
Error: Orphaned: %%DATADIR%%/lib/error_prone_annotations-2.5.1.jar
Error: Orphaned: %%DATADIR%%/lib/ipaddress-5.3.3.jar
Error: Orphaned: %%DATADIR%%/lib/jackson-datatype-jsr310-2.13.2.jar
Error: Orphaned: %%DATADIR%%/lib/jna-5.9.0.jar
Error: Orphaned: %%DATADIR%%/lib/jsr305-2.0.2.jar
Error: Orphaned: %%DATADIR%%/lib/pure_sasl-0.6.2-py2-none-any.zip
Error: Orphaned: %%DATADIR%%/pylib/cqlshlib/authproviderhandling.py
Error: Orphaned: %%DATADIR%%/tools/bin/hash_password
===> Checking for items in pkg-plist which are not in STAGEDIR
Error: Missing: %%DATADIR%%/lib/asm-7.1.jar
Error: Missing: %%DATADIR%%/lib/caffeine-2.5.6.jar
Error: Missing: %%DATADIR%%/lib/jna-5.6.0.jar
Error: Missing: %%DATADIR%%/pylib/cqlshlib/setup.cfg
Error: Missing: %%DATADIR%%/tools/bin/cassandra-stressd
===> Error: Plist issues found.

lapo-luchini · 2023-01-28T13:57:10Z

@polo-language current approach is working only if executed before poudriere is launched, for some reason the pre-fetch inside poudriere seems to work, but then the just-created file isn't found anymore.

BUILD SUCCESSFUL
Total time: 10 seconds
cd /wrkdirs/usr/ports/databases/cassandra4/work/repository && /usr/bin/find . -type f -name "*.repositorie
s" -a -exec /usr/bin/sed -i '' -e '2s,.*,Mon Aug 08 20:40:04 CEST 2022,' {} +
cd /wrkdirs/usr/ports/databases/cassandra4/work/repository && /usr/bin/find . -print0 | xargs -0 touch -d
2022-08-08T20:40:04Z
cd /wrkdirs/usr/ports/databases/cassandra4/work && /usr/bin/find -s repository/ -print >list.txt &&  /usr/
bin/tar cJf /portdistfiles/apache-cassandra-4.1.0-repo.tar.xz -nT list.txt
/bin/rm -f -r /wrkdirs/usr/ports/databases/cassandra4/work
=> apache-cassandra-4.1.0-repo.tar.xz doesn't seem to exist in /portdistfiles/.

lapo-luchini · 2023-01-29T18:09:18Z

Currently fails with:

BUILD FAILED
/wrkdirs/usr/ports/databases/cassandra4/work/apache-cassandra-4.1.0-src/.build/build-resolver.xml:172: The following error occurred while executing this line:
/wrkdirs/usr/ports/databases/cassandra4/work/apache-cassandra-4.1.0-src/.build/build-resolver.xml:74: Task [resolver:resolve] failed after [3] attempts; giving up.
Error messages:
Could not collect dependencies: Failed to collect dependencies at org.xerial.snappy:snappy-java:jar:1.1.8.4

lapo-luchini · 2023-01-29T18:43:02Z

OK, now fails with:

[resolver:resolve] Resolving artifacts                                                                                  
    [mkdir] Created dir: /wrkdirs/usr/ports/databases/cassandra4/work/apache-cassandra-4.1.0-src/lib/sigar-bin          
                                                                                                                        
BUILD FAILED                                                                                                            
/wrkdirs/usr/ports/databases/cassandra4/work/apache-cassandra-4.1.0-src/.build/build-resolver.xml:204: java.net.UnknownHostException: files.pythonhosted.org

lapo-luchini · 2023-07-02T13:52:49Z

I've noticed you're working on this in a5b269c, yay! 🎉
Holler up if you need a hand.

polo-language · 2023-07-09T16:13:54Z

FYI: Trying to get a resolution to the problem of always downloading the maven dependencies:
https://forums.freebsd.org/threads/if-exists-never-sees-file.89637/

polo-language · 2023-07-10T15:59:24Z

Regarding the "reproducible repo", in my 4.1.2 branch I've fixed the problem of make never thinking the maven cache file exists! But... the regular ant build run by either of the do-build-DOCS-* targets performs the download every time nonetheless. Can you take a look into this?

=> SHA256 Checksum OK for cassandra/apache-cassandra-4.1.2-repo.tar.xz.

[...]

===> Building for cassandra4-4.1.2
cd /usr/ports/databases/cassandra4/work/apache-cassandra-4.1.2-src && /usr/local/bin/ant -Dmaven.repo.local=/usr/ports/databases/cassandra4/work/repository -Dlocal.repository=/usr/ports/databases/cassandra4/work/repository -Duse.jdk11=true -Drat.skip=true freebsd-stage
Buildfile: /usr/ports/databases/cassandra4/work/apache-cassandra-4.1.2-src/build.xml

[...]

maven-ant-tasks-init:

maven-declare-dependencies:

_write-poms:

resolver-retrieve-build:
[resolver:resolve] Downloading https://repo1.maven.org/maven2/org/xerial/snappy/snappy-java/1.1.8.4/snappy-java-1.1.8.4.pom
[resolver:resolve] Downloaded https://repo1.maven.org/maven2/org/xerial/snappy/snappy-java/1.1.8.4/snappy-java-1.1.8.4.pom (4 KB at 2.3 KB/sec)
[resolver:resolve] Downloading https://repo1.maven.org/maven2/com/fasterxml/jackson/datatype/jackson-datatype-jsr310/2.13.2/jackson-datatype-jsr310-2.13.2.pom
[resolver:resolve] Downloaded https://repo1.maven.org/maven2/com/fasterxml/jackson/datatype/jackson-datatype-jsr310/2.13.2/jackson-datatype-jsr310-2.13.2.pom (5 KB at 31.7 KB/sec)
[... and so on]

lapo-luchini · 2023-07-10T16:07:15Z

Regarding the "reproducible repo", in my 4.1.2 branch I've fixed the problem of make never thinking the maven cache file exists!

Ohhh! what was it?
Is it simply moving it after .include <bsd.port.pre.mk> (as I've seen in the diff)?

the regular ant build run by either of the do-build-DOCS-* targets performs the download every time nonetheless. Can you take a look into this?

Okay, will do!

polo-language · 2023-08-23T17:57:20Z

Is it simply moving it after .include <bsd.port.pre.mk> (as I've seen in the diff)?

Yes, that was it. The problem had to do with the timing of when build variables are set. If the DISTDIR variable isn't set yet, the directory it will eventually point to appears not to exist.

Have you had any luck with preventing the maven retrieval when the repo file is already present in distfiles?

lapo-luchini · 2023-09-19T22:35:28Z

Unfortunately, not yet. :(
Still intend to do that, as soon as I have some time.

polo-language and others added 10 commits August 8, 2022 21:21

Download maven-managed dependencies in fetch

f88acca

Clean up unused master site

11c6bc2

Upgrade version number: 4.0.5 → 4.1.0

9a4de5a

Update some patches.

94ba2ec

Update some patches.

c8e7363

Update some patches.

9a52391

Update remaining patches.

70ea72b

Patch to remove unused imports.

dfe4f88

Remove .example to have .sample in the end.

13b70c2

lapo-luchini added 3 commits January 28, 2023 00:41

Partial plist fix.

d6c4740

Fix pre-fetch in poudriere.

1611ddf

Only pre-fetch if repo missing.

27a3340

lapo-luchini added 2 commits January 28, 2023 16:35

Copy some tricks from net-mgmt/chronograf.

7aa2399

portlint.

cdc0998

Update Java deps to avoid "snappy" build errors.

ba153c7

Fix plist.

200cc8e

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Upgrade to 4.1 #3

Upgrade to 4.1 #3

lapo-luchini commented Jan 26, 2023

lapo-luchini commented Jan 27, 2023

lapo-luchini commented Jan 28, 2023

lapo-luchini commented Jan 29, 2023

lapo-luchini commented Jan 29, 2023

lapo-luchini commented Jul 2, 2023

polo-language commented Jul 9, 2023

polo-language commented Jul 10, 2023 •

edited

Loading

lapo-luchini commented Jul 10, 2023

polo-language commented Aug 23, 2023

lapo-luchini commented Sep 19, 2023

Upgrade to 4.1 #3

Are you sure you want to change the base?

Upgrade to 4.1 #3

Conversation

lapo-luchini commented Jan 26, 2023

lapo-luchini commented Jan 27, 2023

lapo-luchini commented Jan 28, 2023

lapo-luchini commented Jan 29, 2023

lapo-luchini commented Jan 29, 2023

lapo-luchini commented Jul 2, 2023

polo-language commented Jul 9, 2023

polo-language commented Jul 10, 2023 • edited Loading

lapo-luchini commented Jul 10, 2023

polo-language commented Aug 23, 2023

lapo-luchini commented Sep 19, 2023

polo-language commented Jul 10, 2023 •

edited

Loading