Feature/update-celery-config #578
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
bgunnar5
added a commit
that referenced
this pull request
Sep 17, 2025
* Release/1.13.0 (#549) * Refactor/distributed-tests (#493) * remove a merge conflict statement that was missed * add pytest coverage library and add sample_index coverage * run fix style and add module header * add tests for encryption modules * add unit tests for util_sampling * run fix-style and fix typo * create directory for context managers and fix issue with an encryption test * add a context manager for spinning up/down the redis server * fix issue with path in one test * rework CONFIG functionality for testing * refactor config fixture so it doesn't depend on redis server to be started * split CONFIG fixtures into rabbit and redis configs, run fix-style * add unit tests for broker.py * add unit tests for the Config object * update CHANGELOG * make CONFIG fixtures more flexible for tests * add tests for results_backend.py * fix lint issues for most recent changes * fix filename issue in setup.cfg and move celeryadapter tests to integration suite * add ssl filepaths to mysql config object * add unit tests for configfile.py * add tests for the utils.py file in config/ * create utilities file and constants file * move create_dir function to utils.py * add tests for merlin/examples/generator.py * run fix-style and update changelog * add a 'pip freeze' call in github workflow to view reqs versions * re-delete the old config test files * fix tests/bugs introduced by merging in develop * add a unit test file for the dumper module * begin work on server tests and modular fixtures * start work on tests for RedisConfig * add tests for RedisConfig object * add tests for RedisUsers class * change server fixtures to use redis config files * add tests for AppYaml class * final cleanup of server_utils * fix lint issues * parametrize setup examples tests * sort example output * ensure directory is changed back on no outdir test * sort the specs in examples output * fix lint issues * start writing tests for server config * add pytest coverage library and add sample_index coverage * run fix style and add module header * add tests for encryption modules * add unit tests for util_sampling * run fix-style and fix typo * create directory for context managers and fix issue with an encryption test * add a context manager for spinning up/down the redis server * fix issue with path in one test * rework CONFIG functionality for testing * refactor config fixture so it doesn't depend on redis server to be started * split CONFIG fixtures into rabbit and redis configs, run fix-style * add unit tests for broker.py * add unit tests for the Config object * update CHANGELOG * make CONFIG fixtures more flexible for tests * add tests for results_backend.py * fix lint issues for most recent changes * fix filename issue in setup.cfg and move celeryadapter tests to integration suite * add ssl filepaths to mysql config object * add unit tests for configfile.py * add tests for the utils.py file in config/ * create utilities file and constants file * move create_dir function to utils.py * add tests for merlin/examples/generator.py * run fix-style and update changelog * fix tests/bugs introduced by merging in develop * add a unit test file for the dumper module * begin work on server tests and modular fixtures * start work on tests for RedisConfig * add tests for RedisConfig object * add tests for RedisUsers class * change server fixtures to use redis config files * add tests for AppYaml class * final cleanup of server_utils * fix lint issues * parametrize setup examples tests * sort example output * ensure directory is changed back on no outdir test * sort the specs in examples output * fix lint issues * start writing tests for server config * bake in LC_ALL env variable setting for server cmds * add tests for parse_redis_output * fix issue with scope of fixture after rebase * run fix-style * Include celerymanager and update celeryadapter to check the status of celery workers. * Fixed issue where the update status was outside of if statement for checking workers * Include worker status stop and add template for merlin restart * Added comment to the CeleryManager init * Increment db_num instead of being fixed * Added other subprocess parameters and created a linking system for redis to store env dict * Implemented stopping of celery workers and restarting workers properly * Update stopped to stalled for when the worker doesn't respond to restart * Working merlin manager run but start and stop not working properly * Made fix for subprocess to start new shell and fixed manager start and stop * Added comments and update changelog * Include style fixes * Fix style for black * Revert launch_job script that was edited when doing automated lint * Move importing of CONFIG to be within redis_connection due to error of config not being created yet * Added space to fix style * Revert launch_jobs.py: * Update import of all merlin.config to be in the function * suggested changes plus beginning work on monitor/manager collab * move managers to their own folder and fix ssl problems * final PR touch ups * Fix lint style changes * Fixed issue with context manager * Reset file that was incorrect changed * Check for ssl cert before applying to Redis connection * Comment out Active tests for celerymanager * split up create_server_config and write tests for it * add tests for config_merlin_server function * Fix lint issue with unused import after commenting out Active celery tests * Fixed style for import * add tests for pull_server_config * add tests for pull_server_image * finish writing tests for server_config.py * Fixed kwargs being modified when making a copy for saving to redis worker args. * add tests for server_commands.py * run fix-style * update README for testing directory * update the temp_output_directory to include python version * mock the open.write to try to fix github CI * ensure config dir is created * update CHANGELOG * add print of exception to OSError catch in pull_server_image * change name of config_file in test that's failing * Added password check and omit if a password doesn't exist * update CHANGELOG * change testing log level to debug * add debug statement for redis_connection * change debug log to info so github ci will display it * attempt to fix password missing from Namespace error * run checks for all necessary configurations * convert stop-workers tests to pytest format * update github wf and comment out stop-workers tests in definitions.py * add missing key to GH wf file * fix invalid syntax in definitions.py * comment out stop_workers tests * playing with new caches for workflow CI * fix yaml syntax error * fix typo for getting runner os * fix test and add python version to CI cache * add in common-setup step again with caches this time * run fix-style * update CHANGELOG * fix remaining style issues * run without caches to compare execution time of test suite * allow redis config to not use ssl * remove stop-workers and query-workers tests from definitions.py * create helper_funcs file with common testing functions * move query-workers to pytest and add base class w/ stop-workers tests * update CHANGELOG * final changes for the stop-workers & query-workers tests * run fix-style * move stop and query workers tests to the same file * run fix-style * go back to original cache setup * try new cache for singularity install * fix syntax issue in github workflow * attempt to fix singularity cache * remove ls statement that breaks workflow * revert back to no common setup * remove unnecessary dependency * update github actions versions to use latest * update action versions that didn't save * run fix-style * move distributed test suite actions back to v2 * add 'merlin run' tests and port existing ones to pytest * update CHANGELOG * add aliased fixture types for typehinting * add tests for the purge command * update CHANGELOG * update run command tests to use conditions when appropriate * start work on adding workflow tests * create function and class scoped config fixtures * add Tuple fixture type * get e2e test of feature_demo workflow running * add check for proper variable substitution in e2e test * generalize functionality to run workflows * add create_testing_dir fixture * port chord error workflow to pytest * create dataclasses to house common fixtures and reduce fixture import requirements * fix lint issues * remove hard requirement of Annotated type for python 3.7 and 3.8 * remove distributed test CI and add unit test CI * fix typo in fixture_types and fix lint issues * run fix-style * add check for python2 before adding that condition check * convert local run test to use StepFinishedFilesCount condition * update CHANGELOG.md * fix problem created by merge conflict when mergin develop * remove manager functionality from this PR * update README for test suite * change SIGTERM to SIGKILL * update Makefile to include new changes to test suite --------- Co-authored-by: Ryan Lee <lee1098@llnl.gov> Co-authored-by: Ryan Lee <44886374+ryannova@users.noreply.github.com> * Drop Python 3.7 and Add Python 3.12 & 3.13 (#495) * drop support for py 3.7, add support for py 3.12 and 3.13 * fix docs build issue * remove py 3.7 and add py 3.12/3.13 to integration tests * update Makefile to use target_version variable and update spellbook requirements in examples * Requirements fixes (#501) * add fix for broken deepdiff dependency in py38 and remove py37 specific requirements that were missed * remove try/except imports that are no longer necessary * update CHANGELOG * docs/api_docs (#496) * add/update docstrings for top-level files and enable API docs * remove Generator type hint * fix invalid escape sequences * add docstrings for the common/ directory * fix styling and add in cross-references to portions of Merlin codebase * give code blocks types for formatting * add api docs for half of the study directory * add API docs for study.py * ignore the data path for API docs * finish API docs for study directory * finish api docs for spec/ directory * update CHANGELOG * final cleanup of API docs for common, spec, and study folders * finish API docs for examples folder * began work on API docs for config folder * write API docs for server utils * finish api docs for the server directory * finish API docs for config directory * final cleanup * fix doc build issues * add section explaining API docs to the docs' README file * run fix-style * update readthedocs to build with latest python * fix too few arguments to generator issue * rename in MerlinStepRecord so that it's hidden again * add most of Charles' suggestions * remove unused openfilelist.py, opennpylib.py, and merlin_templates.py files, and add remainder of Charles suggestions * run fix-style * rename load_default_user_names to set_username_and_vhost * Update README.md to remove lgtm banner (#488) * Update README.md to remove lgtm banner * Update CHANGELOG.md * Delete .lgtm.yml * move changelog update to the unreleased section * remove lgtm.yml --------- Co-authored-by: Brian Gunnarson <brianfunnarson14@gmail.com> * Refactor/simplified-ci (#504) * split python and singularity setup into individual actions * add shell to new actions * try to fix shell issue * remove install-deps * reorder cache step and check pip version in the output * update CHANGELOG and make path to definitions.py relative * fix style issues * move cache check to setup-python action * fix an issue with docs build on python 3.9 (#503) * fix an issue with docs build on python 3.9 * adding ChatGPT suggestion to use get-pip.py script * add python version check to reinstall pip CI step * add reinstall pip step to all jobs that need it * save get-pip.py to tmp folder instead of directly to the repo * update CHANGELOG * fix .wci.yml links (#505) * Refactor/config (#498) * update config to use launchit as default and add ability to update from cli * add tests for config broker/backend and update github action * update CHANGELOG and run fix-style * update docs for the config broker/backend update * add new debug step to local-test-suite * Refactor/config-defaults (#497) * update config to use launchit as default and add ability to update from cli * add tests for config broker/backend and update github action * update CHANGELOG and run fix-style * update docs for the config broker/backend update * add new debug step to local-test-suite * remove debug step from github ci * add user permission check to local test suite * create a CI workflow for push and pull_request_target so secrets can be accessed * add link to article where I found this solution * add a merlin info test so we can see what server CI is connecting to * add unit tests for new config commands * add MerlinConfigManager class to replace config-related functions * add unit tests for MerlinConfigManager and update integration tests * comment out the pr-target workflow since it's not working * run fix-style * rewrite tests for find_config_file * remove uses of CONFIGFILE_DIR and chdir from configfile unit tests * run fix-style * re-order the find_config_file function to be local, config_path, merlin home * remove pr-target workflow, add missing docstrings, and run fix-style * add create and use commands, update MerlinConfigManager as necessary * update test suite for config command * update all documentation where 'merlin config' is mentioned * change merlin config to merlin config create * update CHANGELOG * run fix-style * update GitHub workflow to use 'merlin config create' * try to fix f-string issue * add Charles' suggestions * fix broken tests and security issue with new log statement * remove app.yaml file that was accidentally pushed * fix config file validation as Charles suggested * Refactor/server-config (#506) * have server init create an app.yaml in merlin_server folder * add str and repr methods to server config classes and fix bug with server start * update docs for new server refactor * update CHANGELOG * run fix-style * change print to log statement * bugfix/local-config (#507) * add local mode config initialization * have server init create an app.yaml in merlin_server folder * add str and repr methods to server config classes and fix bug with server start * update docs for new server refactor * update CHANGELOG * run fix-style * add unit tests for new functions in configfile.py * run fix-style and update CHANGELOG * change defaults to not use getpass * run fix-style * add changes that Charles suggested * Update copyright (#510) * add COPYRIGHT file, update LICENSE, and change copyright header in all files * update Makefile commands related to copyright/copyright headers * Update CHANGELOG * add a check for the copyright header in 'make check-style' * update CHANGELOG * Potential fix for code scanning alert no. 23: Workflow does not contain permissions Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> * codeql-fixes (#513) * fix logging security vulnerabilities * update CHANGELOG * refactor/main-module (#515) * move each command's entry point logic to their own files * add missing docstrings * add copyright to the top of all new files * fix lint issues * rename backend utils test file * begin work on adding unit tests to the new cli folder * add common create_parser fixture * add tests for example and info * fix docstrings and move conftest file up a directory * add unit tests for 3 more commands * add tests for queue-info, restart, and run-workers * add tests for the remaining commands * remove parser fixtures when only one test uses it * run fix-style * add the no_restart functionality that got wiped from the merge * update CHANGELOG * docs/feature-demo-example (#519) * add a page for the feature_demo example to the docs * update CHANGELOG * feature/tests-spec-folder (#521) * add tests for expansion.py and fix bug * add docstrings and remove comments * add tests for override.py * added merlin and user attributes to init * add initial tests for MerlinSpec * finish writing unit tests for MerlinSpec * run fix-style * update CHANGELOG * run fix-style * Feature/allow-none-in-batch (#552) * allow None and empty strings to be in batch block * update CHANGELOG * Bugfix/abspath-required (#550) * make broker and backend use same function for reading pass file * fix tests * update changelog * run fix-style * change test log level to debug so I can see more output on gh ci * standardize password resolution across broker and backend funcs * fix tests * run fix-style * fix test and update another test * add check for local mode in celery.py file * fix style * Potential fix for code scanning alert no. 37: Clear-text logging of sensitive information Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> * Bugfix/caps-required (#551) * normalize for uppercase in detailed-status filters * update CHANGELOG * run fix-style * update version to 1.13.0 --------- Co-authored-by: Ryan Lee <lee1098@llnl.gov> Co-authored-by: Ryan Lee <44886374+ryannova@users.noreply.github.com> Co-authored-by: Luc Peterson <peterson76@llnl.gov> Co-authored-by: Wout De Nolf <wout.de_nolf@esrf.eu> Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com> * bugfix/retry-on-store-error (#577) * add in BackendStoreError to be auto-retried * run fix style and update CHANGELOG * fix another styling issue * Feature/update-celery-config (#578) * update Celery settings to be more resilient * update CHANGELOG * Bugfix/workspace-ordering (#579) * sort potential studies alphabetically * update CHANGELOG * run fix-style --------- Co-authored-by: Ryan Lee <lee1098@llnl.gov> Co-authored-by: Ryan Lee <44886374+ryannova@users.noreply.github.com> Co-authored-by: Luc Peterson <peterson76@llnl.gov> Co-authored-by: Wout De Nolf <wout.de_nolf@esrf.eu> Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is already in the 2.0 branch but I forgot to leave it in 1.13.0. Going to merge "without approval" since this was already reviewed in the past.