Download Pkg from GitHub releases. #29615
Conversation
|
I think the errors are related to a broken registry? Why would it be complaining about JSON? |
|
Because the release of Pkg in this PR does not have everything needed for the tests to pass. We need to make a new Pkg release. |
|
Thanks for working on this! In #25714 I separated the stdlib from the deps folder and used Are you also thinking about after bootstrapping Pkg to use it to resolve a Manifest file for the stdlib? I personally think that would be neat, but it might require some gymnastics. |
Ok, will change to this. What is the advantage?
Yes, so given this, is it worth separating stdlib out since it might only be Pkg that is installed with the Makefile and the rest of the stdlibs using Pkg? |
| $(eval $(call git-external,Pkg,PKG,,,$(BUILDDIR))) | ||
|
|
||
| extract-pkg: $(BUILDDIR)/$(PKG_SRC_DIR)/source-extracted | ||
| Pkg: $(BUILDDIR)/$(PKG_SRC_DIR)/source-extracted |
There was a problem hiding this comment.
Shouldn't this check for Pkg/source-extracted rather than the file in the build location?
There was a problem hiding this comment.
$(call git-external,Pkg,PKG,,,$(BUILDDIR))
This tells git-external to download and extract it into $(BUILDDIR). See this line.
I would expect that buildstep using Pkg to grab new packages to also use the |
|
The only thing in ext is a TOML parser, we can move it to src if it matters... Also, building things like Compat into the sysimg is a bad idea. |
|
Compat isn't used by Pkg anymore so no need. |
|
I was just explaining my reasoning back then :) TOML itself would be useful to have outside Pkg as its own package. |
|
The TOML business is a bit complicated, let's deal with that later and leave it in |
fredrikekre
force-pushed
the
fe/Pkg
branch
2 times, most recently
from
0958fde
to
1ae7a73
Compare
|
This seem to work now (at least on my machines, lets see what CI says...), so please review! Note that the only thing to review is the second commit, the first one just removes |
|
Why would we backport this to 1.0? You're kind of liberal with that label, @fredrikekre. |
|
Because we need the bugfixes from Pkg that this brings into 1.0.2? |
|
It seems a bit aggressive for a point release, but ok. |
Makefile
Outdated
| @@ -59,6 +59,9 @@ endif | |||
| julia-deps: | $(DIRS) $(build_datarootdir)/julia/base $(build_datarootdir)/julia/test $(build_defaultpkgdir) | |||
| @$(MAKE) $(QUIET_MAKE) -C $(BUILDROOT)/deps | |||
|
|
|||
| julia-stdlib: | |||
There was a problem hiding this comment.
| julia-stdlib: | |
| julia-stdlib: | $(DIRS) |
Also need to be added in line 12
BUILDDIRS := $(BUILDROOT) $(addprefix $(BUILDROOT)/,base src ui doc deps stdlib test test/embedding test/llvmpasses)
Is the culprit here, we are linking the entire |
Maybe, it could be delayed for 1.1 but then someone needs to find the time to re-backport the Pkg commits to Julia (or we just wait with them until Julia 1.1). |
Right, so the idea was for this PR to be the equivalent to the "Bump Pkg" PRs we have made earlier for Julia v1.0.2. The manual syncing of the repos have not worked very well, and some important things have been missed here and there; for example JuliaLang/Pkg.jl#470 was missed for Julia 1.0 which has caused ~10 threads on discourse, and JuliaLang/Pkg.jl#634 missed Julia v1.0.1 which have resulted in even more threads/issues. |
|
Ok, carry on but please do be careful with point releases. |
|
This works for me locally with an out-of-source build now. |
|
I have made a v1.0.2 release of Pkg and updated the tree-sha in this PR so this is good to go from my POV. |
|
🎉 |
* Remove Pkg from the JuliaLang/julia repo. * Use git-external to install Pkg during build. (cherry picked from commit 1717adb)
* Remove Pkg from the JuliaLang/julia repo. * Use git-external to install Pkg during build. (cherry picked from commit 1717adb)
* Remove Pkg from the JuliaLang/julia repo. * Use git-external to install Pkg during build. (cherry picked from commit 1717adb)
| @@ -527,10 +527,11 @@ source-dist: | |||
| full-source-dist: light-source-dist.tmp | |||
| # Get all the dependencies downloaded | |||
| @$(MAKE) -C deps getall NO_GIT=1 | |||
| @$(MAKE) -C stdlib getall | |||
There was a problem hiding this comment.
I guess this was supposed to include Pkg in the source tarballs? Apparently it didn't work: the 1.0.2 tarball doesn't contain it. That's annoying for building distribution packages without network access.
There was a problem hiding this comment.
Hmm, I'm 90% sure I tried this, but maybe I didn't do it from a clean repo. We should be able to re-upload the source tarballs with this included, @ararslan? I'll have a look at it.
There was a problem hiding this comment.
Yes we can re-create and re-upload the source tarballs, it's just a bit annoying since we can't go through the full-source-dist Make target; we'll have to do everything manually. Apologies for not catching this during the release preparation.
There was a problem hiding this comment.
I was talking about light-source-dist, but maybe full-source-dist has the same problem.
There was a problem hiding this comment.
Should it be included there? I interpreted https://github.com/JuliaLang/julia/blob/master/Makefile#L516 as only including code from this repo, and thus Pkg is just an external dependency that should only be in the full variant, which it seems to be btw?
There was a problem hiding this comment.
Well, all other external dependencies are libraries that could be installed separately from Julia, and which generally exist as distribution packages. OTC Pkg cannot be used without Julia. Also, in practical terms, if the plan is to move progressively all stdlib modules to separate repos, it would be quite painful for packagers to have to handle all of them manually (and update versions one by one for each new release).
|
For anyone who gets LoadError("sysimg.jl", 491, ArgumentError("Package Pkg not found in current path:\n- Run `import Pkg; Pkg.add(\"Pkg\")` to install the Pkg package.\n"))during building, you can fix it with make -C stdlib clean-pkgand then building again. |
changes between Julia 1.0 and 1.1, including: - Custom .css-style for compat admonitions. - Information about compat annotations to CONTRIBUTING.md. - NEWS.md entry for PRs #30090, #30035, #30022, #29978, #29969, #29858, #29845, #29754, #29638, #29636, #29615, #29600, #29506, #29469, #29316, #29259, #29178, #29153, #29033, #28902, #28761, #28745, #28708, #28696, #29997, #28790, #29092, #29108, #29782 - Compat annotation for PRs #30090, #30013, #29978, #29890, #29858, #29827, #29754, #29679, #29636, #29623, #29600, #29440, #29316, #29259, #29178, #29157, #29153, #29033, #28902, #28878, #28761, #28708, #28156, #29733, #29670, #29997, #28790, #29092, #29108, #29782, #25278 - Documentation for broadcasting CartesianIndices (#30230). - Documentation for Base.julia_cmd(). - Documentation for colon constructor of CartesianIndices (#29440). - Documentation for ^(::Matrix, ::Number) and ^(::Number, ::Matrix). - Run NEWS-update.jl. Co-authored-by: Morten Piibeleht <morten.piibeleht@gmail.com> Co-authored-by: Fredrik Ekre <ekrefredrik@gmail.com>
changes between Julia 1.0 and 1.1, including: - Custom .css-style for compat admonitions. - Information about compat annotations to CONTRIBUTING.md. - NEWS.md entry for PRs #30090, #30035, #30022, #29978, #29969, #29858, #29845, #29754, #29638, #29636, #29615, #29600, #29506, #29469, #29316, #29259, #29178, #29153, #29033, #28902, #28761, #28745, #28708, #28696, #29997, #28790, #29092, #29108, #29782 - Compat annotation for PRs #30090, #30013, #29978, #29890, #29858, #29827, #29754, #29679, #29636, #29623, #29600, #29440, #29316, #29259, #29178, #29157, #29153, #29033, #28902, #28878, #28761, #28708, #28156, #29733, #29670, #29997, #28790, #29092, #29108, #29782, #25278 - Documentation for broadcasting CartesianIndices (#30230). - Documentation for Base.julia_cmd(). - Documentation for colon constructor of CartesianIndices (#29440). - Documentation for ^(::Matrix, ::Number) and ^(::Number, ::Matrix). - Run NEWS-update.jl. Co-authored-by: Morten Piibeleht <morten.piibeleht@gmail.com> Co-authored-by: Fredrik Ekre <ekrefredrik@gmail.com>
Addition of NEWS and compat admonitions for important changes between Julia 1.0 and 1.1, including: - Custom .css-style for compat admonitions. - Information about compat annotations to CONTRIBUTING.md. - NEWS.md entry for PRs #30090, #30035, #30022, #29978, #29969, #29858, #29845, #29754, #29638, #29636, #29615, #29600, #29506, #29469, #29316, #29259, #29178, #29153, #29033, #28902, #28761, #28745, #28708, #28696, #29997, #28790, #29092, #29108, #29782 - Compat annotation for PRs #30090, #30013, #29978, #29890, #29858, #29827, #29754, #29679, #29636, #29623, #29600, #29440, #29316, #29259, #29178, #29157, #29153, #29033, #28902, #28878, #28761, #28708, #28156, #29733, #29670, #29997, #28790, #29092, #29108, #29782, #25278 - Documentation for broadcasting CartesianIndices (#30230). - Documentation for Base.julia_cmd(). - Documentation for colon constructor of CartesianIndices (#29440). - Documentation for ^(::Matrix, ::Number) and ^(::Number, ::Matrix). - Run NEWS-update.jl. Co-authored-by: Morten Piibeleht <morten.piibeleht@gmail.com> Co-authored-by: Fredrik Ekre <ekrefredrik@gmail.com>
* Remove Pkg from the JuliaLang/julia repo. * Use git-external to install Pkg during build. (cherry picked from commit 1717adb)
* Remove Pkg from the JuliaLang/julia repo. * Use git-external to install Pkg during build. (cherry picked from commit 1717adb)
Hacking my way through Makefile land... at least this seems to work so
Edit: Linux only apparently...