Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Normalize the registry files #65997

Merged
merged 2 commits into from
Aug 10, 2022
Merged

Normalize the registry files #65997

merged 2 commits into from
Aug 10, 2022

Conversation

KristofferC
Copy link
Member

@KristofferC KristofferC commented Aug 10, 2022

Various manual changes and tools have made the registry a bit un-normalized in how whitespace etc is used. This normalizes it by loading all files and resaving them.
Perhaps this should be made into RegistryCI to check that doing this is a no-op.

We should hold off on merging this for a little bit until we verify that things look good.

The code for this is:

using RegistryTools
using Pkg
using Pkg: Versions
using RegistryInstances
using UUIDs

const JULIA_UUID = UUID("1222c4b2-2114-5bfd-aeef-88e4692bbb3e")

const ANY_VERSION = Pkg.Versions.VersionSpec("*")

function versionrange(lo::Versions.VersionBound, hi::Versions.VersionBound)
    lo.t == hi.t && (lo = hi)
    return Versions.VersionRange(lo, hi)
end

function convert_to_registrator_format_compat(compat_dict::Dict, reg)
    new_compat = Dict{VersionNumber, Dict{Any, Any}}()
    for (v, compat_info) in compat_dict
        d = Dict()
        for (uuid, spec) in compat_info
            if spec != ANY_VERSION
                pkg_name = if haskey(Pkg.Types.stdlibs(), uuid)
                    first(Pkg.Types.stdlibs()[uuid])
                else
                    reg[uuid].name
                end

                # Taken from https://github.com/JuliaRegistries/RegistryTools.jl/blob/841a56d8274e2857e3fd5ea993ba698cdbf51849/src/register.jl#L532
                ranges = map(r->versionrange(r.lower, r.upper), spec.ranges)
                ranges = Versions.VersionSpec(ranges).ranges # this combines joinable ranges
                d[pkg_name] = length(ranges) == 1 ? string(ranges[1]) : map(string, ranges)
            end
        end
        new_compat[v] = d
    end
    return new_compat
end

function convert_to_registrator_format_deps(compat_dict::Dict, reg)
    new_deps = Dict{VersionNumber, Dict{Any, Any}}()
    for (v, compat_info) in compat_dict
        d = Dict()
        for (uuid, _) in compat_info
            pkg_name = if haskey(Pkg.Types.stdlibs(), uuid)
                first(Pkg.Types.stdlibs()[uuid])
            else
                reg[uuid].name
            end
            pkg_name == "julia" && continue
            d[pkg_name] = uuid
        end
        new_deps[v] = d
    end
    return new_deps
end

function resave(registry_path=joinpath(DEPOT_PATH[1], "registries", "General"))
    if !isdir(joinpath(registry_path, ".git"))
        error("registry needs to be a git repo")
    end
    reg = RegistryInstances.RegistryInstance(registry_path)
    for (uuid, entry) in reg
        entry.name == "julia" && continue
        info = RegistryInstances.registry_info(entry)
        compat_info = RegistryInstances.compat_info(info)
        reg_compat_dict = convert_to_registrator_format_compat(compat_info, reg)
        RegistryTools.Compress.save(joinpath(reg.path, entry.path, "Compat.toml"), reg_compat_dict)

        reg_deps_dict = convert_to_registrator_format_deps(compat_info, reg)
        if !isempty(reg_deps_dict)
            RegistryTools.Compress.save(joinpath(reg.path, entry.path, "Deps.toml"), reg_deps_dict)
        end
    end
end

@KristofferC KristofferC temporarily deployed to stopwatch August 10, 2022 07:55 Inactive
@KristofferC KristofferC temporarily deployed to stopwatch August 10, 2022 08:18 Inactive
@KristofferC
Copy link
Member Author

KristofferC commented Aug 10, 2022

I'll create a whitespace only change first so that the diff here will be easier to see.

Edit: Done.

@KristofferC
Copy link
Member Author

KristofferC commented Aug 10, 2022

I've verified the compat read from this PR is the same as the other one by running the following function on this branch and the commit before it:

using Pkg

function compare(regpath1, regpath2)
    reg1 = Pkg.Registry.RegistryInstance(regpath1)
    reg2 = Pkg.Registry.RegistryInstance(regpath2)

    for (uuid, _) in reg1
        entry_1 = reg1[uuid]
        entry_2 = reg2[uuid]
        info_1 = Pkg.Registry.registry_info(entry_1)
        info_2 = Pkg.Registry.registry_info(entry_2)
        compat_info_1 = Pkg.Registry.compat_info(info_1)
        compat_info_2 = Pkg.Registry.compat_info(info_2)
        if compat_info_1 != compat_info_2
            error("mismatch in pkg $(entry_1.name)")
        end
    end
end

We have also run a PkgEval run with this branch with no registry surprises (JuliaLang/julia#45540 (comment)). I would therefore claim this is good to merge.

@DilumAluthge
Copy link
Member

Perhaps this should be made into RegistryCI to check that doing this is a no-op.

JuliaRegistries/RegistryCI.jl#459

bors bot added a commit to JuliaRegistries/RegistryCI.jl that referenced this pull request Aug 14, 2022
458: Registry consistency tests: the compress roundtrip check should compare `VersionRange`s (not `String`s) r=DilumAluthge a=DilumAluthge

Fixes #457

Will likely fail until JuliaRegistries/General#65997 is merged.

Co-authored-by: Dilum Aluthge <dilum@aluthge.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants