Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

initial runner api #1732

Merged
merged 55 commits into from
May 23, 2024
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
55 commits
Select commit Hold shift + click to select a range
90e1aea
initial runner map api
madhur-ob Feb 15, 2024
a8cd6ab
return Run objects
madhur-ob Feb 21, 2024
89cb6eb
WIP: click to method mapping
madhur-ob Mar 2, 2024
723e835
WIP: click to method, adding groups
madhur-ob Mar 2, 2024
1d429de
remove comment
madhur-ob Mar 2, 2024
ef279d1
pass in args and opts correctly
madhur-ob Mar 2, 2024
682f6a0
fix validation for top level params
madhur-ob Mar 2, 2024
37ba90d
restore metaflow_runner.py
madhur-ob Mar 2, 2024
d6fe8ac
handle list in options
madhur-ob Mar 3, 2024
37bb394
fix nargs
madhur-ob Mar 3, 2024
5c255ef
fix issue with validating missing args
madhur-ob Mar 3, 2024
d47cd75
inject flow parameters as well
madhur-ob Mar 3, 2024
7812e58
handle include file
madhur-ob Mar 3, 2024
f958c68
internal click to chaining methods functionality
madhur-ob Mar 4, 2024
86968e6
subprocess manager
madhur-ob Mar 12, 2024
1c3c5bd
simpler subprocess manager
madhur-ob Mar 28, 2024
61623cd
handle SIGINT
madhur-ob Mar 28, 2024
216c366
wait() handles timeout + logs both
madhur-ob Mar 28, 2024
847822d
improved log streaming with ability to resume + stream multiple times
madhur-ob Apr 9, 2024
be5512e
cleanup in context manager, otherwise user is responsible
madhur-ob Apr 9, 2024
4158db0
remove aiofiles
madhur-ob Apr 9, 2024
b7d1083
split into CommandManager and SubprocessManager
madhur-ob Apr 9, 2024
bf04383
return components array instead
madhur-ob Apr 9, 2024
efa8750
remove assert
madhur-ob Apr 9, 2024
cc6d602
add type hints
madhur-ob Apr 9, 2024
9766d0f
return ExecutingRun encapsulating CommandManager and metaflow.Run
madhur-ob Apr 9, 2024
080fbb2
print as string
madhur-ob Apr 16, 2024
8a5a14f
suggested improvements
madhur-ob Apr 16, 2024
04b8811
use correct variable name
madhur-ob Apr 16, 2024
0cfc979
update usage of subprocess manager
madhur-ob Apr 16, 2024
b8ed331
fix flags such as --pylint/--no-pylint
madhur-ob Apr 28, 2024
73935b1
cache loaded modules
madhur-ob Apr 28, 2024
d3f53aa
fix boolean flags
madhur-ob Apr 28, 2024
79aeb4e
remove aiofiles
madhur-ob Apr 28, 2024
b8a47ac
no f-strings
madhur-ob Apr 28, 2024
25f8ace
get run object explicitly
madhur-ob May 1, 2024
33c08c3
new higher level UX
madhur-ob May 2, 2024
bafd29b
typeguard warning for 3.5/3.6 + add profile argument
madhur-ob May 2, 2024
2128a22
default None for profile
madhur-ob May 2, 2024
95e7076
detect early issues
madhur-ob May 6, 2024
1f20019
fix user defined params not being recognised
madhur-ob May 6, 2024
6b67548
kill all descendants
madhur-ob May 6, 2024
a77ba01
do not use f-strings
madhur-ob May 6, 2024
5c8a9aa
suggested changes
madhur-ob May 13, 2024
9a43517
use pkill instead of psutil
madhur-ob May 13, 2024
93dcd5c
minor fix
madhur-ob May 15, 2024
fc0ee63
vendor typeguard
madhur-ob May 15, 2024
471c335
update error message
madhur-ob May 21, 2024
706c00c
remove comma and check in init
madhur-ob May 21, 2024
c387d84
Added docstrings; fixed stubs; moved stuff around
romain-intel May 23, 2024
737d2da
change dir name to runner
madhur-ob May 23, 2024
4363d8f
fix import in test
madhur-ob May 23, 2024
fc1d92e
fix import
madhur-ob May 23, 2024
790e270
add status property to ExecutingRun
madhur-ob May 23, 2024
d661bbc
add docstring for status property
madhur-ob May 23, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
minor fix
  • Loading branch information
madhur-ob committed May 21, 2024
commit 93dcd5ca3bdc3018baecd9cd5152aab014d591f1
2 changes: 1 addition & 1 deletion metaflow/_vendor/vendor_any.txt
Original file line number Diff line number Diff line change
@@ -1,2 +1,2 @@
click==7.1.2
packaging==23.0
packaging==23.0
21 changes: 10 additions & 11 deletions metaflow/click_api.py
Original file line number Diff line number Diff line change
@@ -1,3 +1,12 @@
import sys

if sys.version_info < (3, 7):
raise RuntimeError(
"""
The Metaflow Programmatic API requires 'typeguard', which is not supported for python 3.5 and 3.6.
madhur-ob marked this conversation as resolved.
Show resolved Hide resolved
"""
)

import inspect
madhur-ob marked this conversation as resolved.
Show resolved Hide resolved
import importlib
import itertools
madhur-ob marked this conversation as resolved.
Show resolved Hide resolved
Expand All @@ -17,6 +26,7 @@
from metaflow._vendor import click
from metaflow.parameters import JSONTypeClass
from metaflow.includefile import FilePathClass
from metaflow._vendor.typeguard import check_type, TypeCheckError
from metaflow._vendor.click.types import (
StringParamType,
IntParamType,
Expand All @@ -30,17 +40,6 @@
File,
)

try:
from typeguard import check_type, TypeCheckError
except ImportError:
raise ImportError(
"""
The Metaflow Programmatic API requires 'typeguard', which is not installed or available.
Please try installing with `pip install typeguard`. This package and thus, the programmatic API
functionality is unavailable for python 3.5 and 3.6.
"""
)

click_to_python_types = {
StringParamType: str,
IntParamType: int,
Expand Down