-
Notifications
You must be signed in to change notification settings - Fork 27
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
refactor: deb_import #96
base: main
Are you sure you want to change the base?
refactor: deb_import #96
Commits on Sep 22, 2024
-
fix: MODULE.bazel.lock conflicts
See bazelbuild/bazel#20369 While working on refactoring, I kept hitting rebase conflicts due to issues with the MODULE lock. I guess it's because the Bazel version in e2e tests is much lower than the current one with the fix (7.2) but still, I don't think it adds much to have the lock in e2e testing.
Configuration menu - View commit details
-
Copy full SHA for 3675db4 - Browse repository at this point
Copy the full SHA 3675db4View commit details -
fix: repo name in copy.sh script
PR GoogleContainerTools#73 added the `_resolve` and this breaks the buildozer fix / autofix
Configuration menu - View commit details
-
Copy full SHA for bf9b27a - Browse repository at this point
Copy the full SHA bf9b27aView commit details -
It seemed like 75afff9 in GoogleContainerTools#47 added the new locks but as new files, that is, the old ones were left behind.
Configuration menu - View commit details
-
Copy full SHA for 22ba9e5 - Browse repository at this point
Copy the full SHA 22ba9e5View commit details -
feat: add support for MODULE.bazel to the lock copy.sh script
Add support for MODULE.bazel to the lock script and avoid printing an unnecessary (and annoying 😅) error when building in a "modern repo".
Configuration menu - View commit details
-
Copy full SHA for eff7b9f - Browse repository at this point
Copy the full SHA eff7b9fView commit details -
feat: avoid DEBUG messages for lockless repos
Add a `nolock` attribute to avoid getting annoying DEBUG messages for repos that we explicitly want to run without a lock.
Configuration menu - View commit details
-
Copy full SHA for 540a7cb - Browse repository at this point
Copy the full SHA 540a7cbView commit details -
* make apt/tests more readable by factoring out the parameters * add a "test suite macro" in each test file that group all of the unit tests in the file and prepends a "test suite prefix". IMHO this is better than using `unittest.suite` because we provide better naming than the automated `_test<NUMBER>` plus these better names are actual targets that can be executed one-by-one by name.
Configuration menu - View commit details
-
Copy full SHA for c789aad - Browse repository at this point
Copy the full SHA c789aadView commit details -
test: add a bullseye_nolock package to the tests
Add other nolock tests to exercise the package repos (the templates, etc).
Configuration menu - View commit details
-
Copy full SHA for c038d26 - Browse repository at this point
Copy the full SHA c038d26View commit details -
refactor: improve the lock copy.sh script
* separate the script into a template file so it's easier to shellcheck and syntax highlight in editors. * shellcheck the script and remove all SC2086 warnings ("Double quote to prevent globbing and word splitting") * improve the buildozer help messages in the copy.sh script: * reduce duplication of buildozer command * add a more clear autofix bazel run command that can be easily copy-pasted * change some of the variable names in the copy.sh template for longer, easier to understand names (repo_name >> name, lock_label >> label) * move repo_name and workspace_relative_path into variables to reduce line length and improve readability
Configuration menu - View commit details
-
Copy full SHA for 0afb5dc - Browse repository at this point
Copy the full SHA 0afb5dcView commit details -
refactor: package_index _fetch_package_index
While working with some flaky mirrors and trying to figure out why they were failing I found the _fetch_package_index code a bit hard to follow so here's my attempt at streamlining it a bit: * Change the general flow of the for-loop so that we can directly set the reasons for failure in failed_attempts * Remove both integrity as an argument and as a return value since neither is ever used. * return the content of the Packages index instead of the path, since we already have the repository_context. This way we don't need rctx anywhere else. * Reword failure messages adding more context and debug information * Shorter lines and templated strings, trying to make the code easier to read and follow.
Configuration menu - View commit details
-
Copy full SHA for d0bd4eb - Browse repository at this point
Copy the full SHA d0bd4ebView commit details -
refactor: package_index struct
* remove the `state` "intermediary `struct`" in `package_resolution.bzl` since it wasn't used / needed. * refactor and move _set_dict from `util.bzl` to a `_package_set` method * use `dict .get()` with default values instead of the "nested `if`s" * renamed `package()` to `package_get` and make it return all package versions when the version is not specified so we can remove `_package_versions` * reordeder `(name, version, arch)` args to match the order of the index keys `(arch, name, version)`
Configuration menu - View commit details
-
Copy full SHA for d5b7bda - Browse repository at this point
Copy the full SHA d5b7bdaView commit details -
Add testing for package_index mocking the external / side effects (downloads, decompression, etc).
Configuration menu - View commit details
-
Copy full SHA for 36258ed - Browse repository at this point
Copy the full SHA 36258edView commit details -
refactor: centralize and improve package_resolution logging
The package resolution debugging that e.g. checks the package dependencies should all be within package_resolution.bzl _resolve_all() and not "leak out" returning the information that's only needed for debugging / logging. Also: * reduce the verbosity of the optional dependencies warning by just printing one message per root package instead of one per package. * break up the long lines to build the error messages and remove the "# buildifier: disable=print".
Configuration menu - View commit details
-
Copy full SHA for 1e42af0 - Browse repository at this point
Copy the full SHA 1e42af0View commit details -
* move version constraint parsing from package_resolution to its own _parse_version_and_constraint method in version.bzl * refactor _version_relop into a compare method in version.bzl plus a VERSION_OPERATORS dict so that (1) we use the operator strings everywhere and (2) we can use the keys to validate the operators.
Configuration menu - View commit details
-
Copy full SHA for 628d989 - Browse repository at this point
Copy the full SHA 628d989View commit details -
refactor: combine package_resolution and package_index
Previously we had: ```starlark pkgindex = package_index.new(rctx, sources = sources, archs = manifest["archs"]) pkgresolution = package_resolution.new(index = pkgindex) ``` And none of the code of package_resolution was used anywhere but in resolve.bzl and after initializing the `pkgindex`. Also, it makes sense since we are building the index from the manifest and once we have the index we use it to resolve the packages and populate the lock.
Configuration menu - View commit details
-
Copy full SHA for 88623f1 - Browse repository at this point
Copy the full SHA 88623f1View commit details -
refactor: move all manifest parsing and validation to manifest.bzl
Cleanup resolve.bzl and package_index.bzl by moving all of the manifest functionality to a separate manifest.bzl file where we now do all of the work to generate the lock: manifest parsing, validation and the package index and resolution. IMHO this is how it should be because the lock is the "frozen state" of the manifest. * _parse() parses the YAML * _from_dict validates the manifest dict and does the rest of the changes that we need to produce a manifest struct * add extra validation for e.g. duplicated architectures * _lock is the only method that's exposed to the outside and it encapsulates all of the other parts, calling _from_dict and all of the package index and resolution, to produce the lock file. * move get_dupes to util.bzl * refactor the "source" struct into the new manifest where we can now centralize a lot of the structure and logic spread across multiple parts of the code. * remove yq_toolchain_prefix since it's always "yq" and, looking at GH code search, this seems to be a copy-paste leftover from rules_js (or the other way around)... the code is always the same and it never receives a string different from "yq".
Configuration menu - View commit details
-
Copy full SHA for dd7b1a7 - Browse repository at this point
Copy the full SHA dd7b1a7View commit details -
refactor: lockfile, pkg, and lock v1-to-v2 auto-migration
* move all of the "package logic" to pkg.bzl * The v2 lockfile format: * doesn't need the fast_package_lookup dict because it's already using a dict to store the packages. * has the dependencies sorted so the lockfile now has stable serialization and the diffs of the lock are actually usable and useful to compare with the changes to the manifest. * removes the package and dependency key from the lockfile, now it's done via an external function (make_deb_import_key in deb_import.bzl) * Remove add_package_dependency from the lockfile API. Now, the package dependencies are passed as an argument to add_package. This way, the lockfile functionality is fully contained in lockfile.bzl and e.g. we can remove the "consistency checks" that were only needed because users could forget to add the dependency as a package to the lockfile. * Ensure backwards-compatibility by internally converting lock v1 to v2. Also, when a lock is set and it's in v1 format, there's a reminder that encourages the users to run @repo//:lock to update the lockfile format.
Configuration menu - View commit details
-
Copy full SHA for 6480ba3 - Browse repository at this point
Copy the full SHA 6480ba3View commit details -
chore: migrate repo locks to v2
By separating the migration from the previous commit we get to 1. in the previous commit, run all tests with the new code while locks are still v1 2. update the locks n this commit to V2 so we can then re-run all tests in the final state.
Configuration menu - View commit details
-
Copy full SHA for ab87a69 - Browse repository at this point
Copy the full SHA ab87a69View commit details -
refactor: centralize all of the deb_import functionality in deb_impor…
…t.bzl Refactor the package repo templates into their own methods and massively cleanup the `for`-loop in `_deb_package_index_impl`. IMHO overall now there's a much better and clear separation of concerns between the "index repo" (`apt/private/index.bzl`) and the "package repos" (`apt/private/deb_import.bzl`).
Configuration menu - View commit details
-
Copy full SHA for 8c8fe9e - Browse repository at this point
Copy the full SHA 8c8fe9eView commit details