Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

snowpark decorator #1899

Closed
wants to merge 17 commits into from
Prev Previous commit
Next Next commit
@parallel throws an error with @snowpark
  • Loading branch information
madhur-ob committed Aug 5, 2024
commit 9f8f99a8b6fb8c975761501d854ef0a8e616fb7a
9 changes: 9 additions & 0 deletions metaflow/plugins/snowpark/snowpark_decorator.py
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@
from metaflow.metadata.util import sync_local_metadata_to_datastore
from metaflow.sidecar import Sidecar
from metaflow.decorators import StepDecorator
from metaflow.exception import MetaflowException
from metaflow.metaflow_config import DEFAULT_CONTAINER_IMAGE, DEFAULT_CONTAINER_REGISTRY

from metaflow.metaflow_config import (
Expand Down Expand Up @@ -81,6 +82,14 @@ def step_init(self, flow, graph, step, decos, environment, flow_datastore, logge
self.step = step
self.flow_datastore = flow_datastore

if any([deco.name == "parallel" for deco in decos]):
raise MetaflowException(
"Step *{step}* contains a @parallel decorator "
"with the @snowpark decorator. @parallel is not supported with @snowpark.".format(
step=step
)
)

def package_init(self, flow, step_name, environment):
try:
# Snowflake is a soft dependency.
Expand Down
Loading