Work around weird scoping issue #76
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
*Overview* - Migrate "manage dependencies" UI to APIs and VueJS. - Add new `summarize_toolbox` API endpoint to mimic information produced by former grid. - The tool and requirement views are in the same grid now - depending on what you are viewing there is a "collapse tools" / "expand tools" button to swap to the other view. - Unused environments grid migrated as its own component. - By default information displayed information is more compact but every row can be expanded and many elements display a lot more information on mouse over. - Implement fallback container resolvers (describe below). - Implement RESTful API around container resolvers - describe both how resolvers are configured and the resolution of particular tools (or all tools). Add API endpoint for building containers as well. - Implement new VueJS table describing container resolution independent of dependencies. Allow mixed mode (show whatever container first resolved) as well as filtering by Docker/Singularity. - This information about containers is much more correct than what the manage dependencies grid would display - which was independent of tool IDs and based only on requirements, would not query quay.io for information, etc.. - Since the grid is restricted to containers, embedded information is richer - displays information about the container resolver, a link to the container, correct table header, shell used by the container, etc.. *Why Fallback Container Resolvers?* tl;dr - we can write better config files and more useful UIs if we specify default containers this way. Default containers can be specified in the job_conf but Galaxy doesn't really have the ability to dispatch or reason about such configuration until it is time to run the job. galaxy.tool_util has no clue about job configuration structure and job configuration structure can be (and often is) completely dynamic. As a result, such configuration should be thought of procedurally specified - configuration via container resolvers is effectively more explicitly annotated. This has two upshots for my current cloud dev push: - Containers specified this way can reasoned about in terms of the dependency magnagement API so we can provide an interface for seeing what tools will map to job (by default anyway - same stipulation applies to mulled containers which can also be overridden). I will utilize to make galaxyproject#8741. - While I think job_conf should ultimately allow overriding the container, I think the way this works out in most setups - including the CloudVE setup (https://github.com/galaxyproject/galaxy-helm/blob/master/galaxy/values.yaml#L201) the container is more realistically a global configuration setting. As such it is not naturally specified job_conf, awkardness results, and it would be much more cleanly specified in the container configuration - a global configuration. Handle empty requirements list in ResolutionDetails.vue. Error reported here galaxyproject#8741 (comment).
... when resolving across a toolbox. Start with caching the result of 'docker images'.
…os per namespace.
Profiling before:
```
*** PROFILER RESULTS ***
summarize_toolbox (/Users/mvandenb/src/galaxy/lib/galaxy/web/framework/decorators.py:228)
function called 1 times
42850785 function calls (40691205 primitive calls) in 63.276 seconds
Ordered by: cumulative time, internal time, call count
List reduced from 279 to 40 due to restriction <40>
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.000 0.000 63.276 63.276 decorators.py:228(decorator)
1 0.000 0.000 63.272 63.272 decorators.py:87(decorator)
1 0.000 0.000 63.272 63.272 tool_dependencies.py:189(summarize_toolbox)
1 0.056 0.056 63.272 63.272 views.py:228(summarize_requirements)
277 7.137 0.026 61.665 0.223 views.py:211(tool_ids_by_requirements)
716598 2.004 0.000 26.227 0.000 requirements.py:127(__eq__)
1586883 3.859 0.000 24.838 0.000 oset.py:8(__init__)
1428437 5.199 0.000 20.979 0.000 _collections_abc.py:604(__ior__)
869226 1.421 0.000 20.939 0.000 __init__.py:1664(tool_requirements)
716598 1.772 0.000 20.415 0.000 _collections_abc.py:474(__and__)
869226 3.086 0.000 19.518 0.000 requirements.py:115(packages)
716598 1.240 0.000 16.914 0.000 _collections_abc.py:465(_from_iterable)
870285 2.999 0.000 12.723 0.000 requirements.py:99(__init__)
1677185 5.694 0.000 8.816 0.000 oset.py:21(add)
869778 1.800 0.000 7.336 0.000 requirements.py:143(__hash__)
1413485 3.166 0.000 6.964 0.000 _collections_abc.py:477(<genexpr>)
869778 3.816 0.000 5.332 0.000 requirements.py:144(<listcomp>)
5052459 4.203 0.000 4.730 0.000 requirements.py:51(__hash__)
4321842/2163046 3.081 0.000 3.828 0.000 {built-in method builtins.len}
869226 2.847 0.000 3.734 0.000 requirements.py:117(<listcomp>)
696887 2.075 0.000 3.291 0.000 oset.py:18(__contains__)
5151455 2.050 0.000 2.050 0.000 oset.py:33(__iter__)
2456292 1.339 0.000 1.918 0.000 {built-in method builtins.isinstance}
1 0.008 0.008 1.531 1.531 views.py:241(<dictcomp>)
276 0.023 0.000 1.496 0.005 views.py:282(get_requirements_status)
276 0.007 0.000 1.436 0.005 views.py:52(show_dependencies)
276 0.007 0.000 1.428 0.005 views.py:62(<dictcomp>)
276 0.002 0.000 1.420 0.005 __init__.py:185(requirements_to_dependencies)
276 0.083 0.000 1.418 0.005 __init__.py:198(_requirements_to_dependencies_dict)
2154295 0.745 0.000 0.927 0.000 oset.py:15(__len__)
2457 0.016 0.000 0.736 0.000 __init__.py:1356(debug)
2457 0.022 0.000 0.719 0.000 __init__.py:1491(_log)
723361 0.247 0.000 0.579 0.000 abc.py:137(__instancecheck__)
2457 0.011 0.000 0.577 0.000 __init__.py:1516(handle)
2457 0.011 0.000 0.565 0.000 __init__.py:1570(callHandlers)
2457 0.018 0.000 0.554 0.000 __init__.py:881(handle)
5052459 0.527 0.000 0.527 0.000 {built-in method builtins.hash}
2457 0.017 0.000 0.523 0.000 __init__.py:1013(emit)
696378 0.495 0.000 0.495 0.000 requirements.py:45(__eq__)
711839 0.398 0.000 0.481 0.000 requirements.py:103(<listcomp>)
```
Profiling after:
```
*** PROFILER RESULTS ***
summarize_toolbox (/Users/mvandenb/src/galaxy/lib/galaxy/web/framework/decorators.py:228)
function called 1 times
725248 function calls (707704 primitive calls) in 1.945 seconds
Ordered by: cumulative time, internal time, call count
List reduced from 278 to 40 due to restriction <40>
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.000 0.000 1.945 1.945 decorators.py:228(decorator)
1 0.000 0.000 1.941 1.941 decorators.py:87(decorator)
1 0.000 0.000 1.941 1.941 tool_dependencies.py:189(summarize_toolbox)
1 0.005 0.005 1.941 1.941 views.py:228(summarize_requirements)
1 0.008 0.008 1.708 1.708 views.py:242(<dictcomp>)
276 0.025 0.000 1.672 0.006 views.py:283(get_requirements_status)
276 0.008 0.000 1.607 0.006 views.py:52(show_dependencies)
276 0.008 0.000 1.599 0.006 views.py:62(<dictcomp>)
276 0.002 0.000 1.590 0.006 __init__.py:185(requirements_to_dependencies)
276 0.089 0.000 1.589 0.006 __init__.py:198(_requirements_to_dependencies_dict)
2457 0.018 0.000 0.818 0.000 __init__.py:1356(debug)
2457 0.025 0.000 0.798 0.000 __init__.py:1491(_log)
2457 0.012 0.000 0.643 0.000 __init__.py:1516(handle)
2457 0.011 0.000 0.630 0.000 __init__.py:1570(callHandlers)
2457 0.019 0.000 0.619 0.000 __init__.py:881(handle)
2457 0.018 0.000 0.586 0.000 __init__.py:1013(emit)
2457 0.021 0.000 0.456 0.000 pydevd_io.py:30(write)
4198 0.265 0.000 0.265 0.000 {built-in method posix.stat}
2457 0.024 0.000 0.243 0.000 pydevd.py:2044(write)
1 0.026 0.026 0.220 0.220 views.py:211(tool_ids_by_requirements)
1483 0.008 0.000 0.200 0.000 galaxy_packages.py:68(resolve)
961 0.023 0.000 0.192 0.000 conda.py:256(resolve)
2457 0.190 0.000 0.190 0.000 {method 'write' of '_io.TextIOWrapper' objects}
2936 0.006 0.000 0.175 0.000 genericpath.py:39(isdir)
507 0.016 0.000 0.171 0.000 conda.py:184(resolve_all)
2457 0.029 0.000 0.163 0.000 pydevd_net_command_factory_json.py:248(make_io_message)
974 0.006 0.000 0.149 0.000 galaxy_packages.py:112(resolve)
631 0.010 0.000 0.117 0.000 galaxy_packages.py:96(_galaxy_package_dep)
2457 0.035 0.000 0.114 0.000 pydevd_net_command.py:36(__init__)
6783 0.018 0.000 0.108 0.000 oset.py:8(__init__)
2457 0.012 0.000 0.106 0.000 __init__.py:1476(makeRecord)
1262 0.003 0.000 0.099 0.000 genericpath.py:16(exists)
2586 0.007 0.000 0.094 0.000 requirements.py:127(__eq__)
2457 0.056 0.000 0.094 0.000 __init__.py:282(__init__)
1468 0.006 0.000 0.092 0.000 conda_util.py:318(has_env)
6209 0.023 0.000 0.090 0.000 _collections_abc.py:604(__ior__)
503 0.003 0.000 0.088 0.000 galaxy_packages.py:81(_find_dep_versioned)
2457 0.006 0.000 0.077 0.000 __init__.py:858(format)
3138 0.005 0.000 0.074 0.000 __init__.py:1664(tool_requirements)
2586 0.006 0.000 0.073 0.000 _collections_abc.py:474(__and__)
```
Co-Authored-By: Marius van den Beek <m.vandenbeek@gmail.com>
... easier to cache.
For some reason `cached_name` didn't work even though Ijust added an entry to mulled_resolution_cache. Brings down the time for checking the container resolution from 90 seconds to 31. `
jmchilton
force-pushed
the
manage_dependencies
branch
2 times, most recently
from
ae5dd31
to
fa67bf0
Compare
jmchilton
force-pushed
the
manage_dependencies
branch
2 times, most recently
from
b6b67f1
to
1f0ec82
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
For some reason
cached_namedidn't work even though I justadded an entry to mulled_resolution_cache.
Brings down the time for checking the container resolution
from 90 seconds to 31.