Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Zombie Tasks Runs caused by increasing concurrency limit during task run #7203

Closed
4 tasks done
j-tr opened this issue Oct 17, 2022 · 5 comments
Closed
4 tasks done

Zombie Tasks Runs caused by increasing concurrency limit during task run #7203

j-tr opened this issue Oct 17, 2022 · 5 comments
Labels
api Related the Prefect REST API bug Something isn't working cloud Related to Prefect Cloud

Comments

@j-tr
Copy link
Contributor

j-tr commented Oct 17, 2022 •
edited
Loading

First check

  • I added a descriptive title to this issue.
  • I used the GitHub search to find a similar issue and didn't find it.
  • I searched the Prefect documentation for this issue.
  • I checked that this issue is related to Prefect and not one of its dependencies.

Bug summary

when increasing the task concurrency limit while a flow is executing (this would be very useful when having long running flows and a dynamically scaleable db), the number of "Active Tasks Runs" doesn't return to zero after all flows are completed. instead a certain number of tasks is still listed as active by the prefect concurrency-limit inspect command but is in fact in status completed.

Reproduction

import time
from prefect import flow, task, tags, context, get_run_logger
from prefect.utilities.asyncutils import sync_compatible


tag = "test-limit"


@sync_compatible
async def create_limit(n):
    # set concurrency limit
    await context.get_run_context().client.create_concurrency_limit(tag=tag, concurrency_limit=n)


@sync_compatible
async def get_active_tasks():
    # list all "active" tasks for the tag
    return (await context.get_run_context().client.read_concurrency_limit_by_tag(tag=tag)).active_slots


@sync_compatible
async def get_running_tasks():
    # list all tasks ids in state "Running"
    running_tasks = []
    for future in context.get_run_context().task_run_futures:
        if (await future._get_state()).name == "Running":
            running_tasks.append(future.task_run.id)
    return running_tasks


def monitor_tasks(t):
    # print every two seconds a list of "active" and "Running" tasks for t seconds
    logger = get_run_logger()
    start = time.time()
    while time.time() - start < t:
        running_tasks = get_running_tasks()
        active_tasks = get_active_tasks()
        if running_tasks or active_tasks:
            logger.info(f"Running tasks ({len(running_tasks)}): {running_tasks}")
            logger.info(f"Active tasks ({len(active_tasks)}): {active_tasks}")
            time.sleep(2)
        else:
            break


@task
def sleep_task(t):
    # sleep for t seconds
    time.sleep(t)


@flow
def test_limits():
    logger = get_run_logger()

    # setting concurrency limit 5 for tag
    create_limit(5)

    # submit 20 parallel tagged tasks that sleep for 20s each
    with tags(tag):
        for _ in range(0, 20):
            sleep_task.submit(20)
            # time.sleep(0.2) # This leads to expected behaviour

    # monitor "Running" and "active" tasks for 30s
    # see issue 1 in output: more tasks "Running" than "active"
    monitor_tasks(30)

    # increase concurrency limit while tasks are still running
    logger.info("Updating limits")
    create_limit(10)

    # continue monitoring
    # see issue 2 in output: No tasks in "Running" but still tasks in "active"
    monitor_tasks(120)

    # After the flow exits, check limits via `prefect concurrency-limit inspect test-limit`
    # see that there are still "Active Task Runs" even though the flow "Finished in state Completed('All states completed.')"


if __name__ == "__main__":
    test_limits()

Error

17:01:35.428 | INFO    | prefect.engine - Created flow run 'cherry-orangutan' for flow 'test-limits'
17:01:37.039 | INFO    | Flow run 'cherry-orangutan' - Created task run 'sleep_task-f305a62b-0' for task 'sleep_task'
17:01:37.040 | INFO    | Flow run 'cherry-orangutan' - Submitted task run 'sleep_task-f305a62b-0' for execution.
17:01:37.113 | INFO    | Flow run 'cherry-orangutan' - Created task run 'sleep_task-f305a62b-2' for task 'sleep_task'
17:01:37.115 | INFO    | Flow run 'cherry-orangutan' - Submitted task run 'sleep_task-f305a62b-2' for execution.
17:01:37.126 | INFO    | Flow run 'cherry-orangutan' - Created task run 'sleep_task-f305a62b-8' for task 'sleep_task'
17:01:37.127 | INFO    | Flow run 'cherry-orangutan' - Submitted task run 'sleep_task-f305a62b-8' for execution.
17:01:37.138 | INFO    | Flow run 'cherry-orangutan' - Created task run 'sleep_task-f305a62b-1' for task 'sleep_task'
17:01:37.139 | INFO    | Flow run 'cherry-orangutan' - Submitted task run 'sleep_task-f305a62b-1' for execution.
17:01:37.143 | INFO    | Flow run 'cherry-orangutan' - Created task run 'sleep_task-f305a62b-4' for task 'sleep_task'
17:01:37.144 | INFO    | Flow run 'cherry-orangutan' - Submitted task run 'sleep_task-f305a62b-4' for execution.
17:01:37.154 | INFO    | Flow run 'cherry-orangutan' - Created task run 'sleep_task-f305a62b-13' for task 'sleep_task'
17:01:37.155 | INFO    | Flow run 'cherry-orangutan' - Submitted task run 'sleep_task-f305a62b-13' for execution.
17:01:37.160 | INFO    | Flow run 'cherry-orangutan' - Created task run 'sleep_task-f305a62b-11' for task 'sleep_task'
17:01:37.161 | INFO    | Flow run 'cherry-orangutan' - Submitted task run 'sleep_task-f305a62b-11' for execution.
17:01:37.164 | INFO    | Flow run 'cherry-orangutan' - Created task run 'sleep_task-f305a62b-10' for task 'sleep_task'
17:01:37.165 | INFO    | Flow run 'cherry-orangutan' - Submitted task run 'sleep_task-f305a62b-10' for execution.
17:01:37.166 | INFO    | Flow run 'cherry-orangutan' - Created task run 'sleep_task-f305a62b-7' for task 'sleep_task'
17:01:37.167 | INFO    | Flow run 'cherry-orangutan' - Submitted task run 'sleep_task-f305a62b-7' for execution.
17:01:37.169 | INFO    | Flow run 'cherry-orangutan' - Created task run 'sleep_task-f305a62b-3' for task 'sleep_task'
17:01:37.170 | INFO    | Flow run 'cherry-orangutan' - Submitted task run 'sleep_task-f305a62b-3' for execution.
17:01:37.174 | INFO    | Flow run 'cherry-orangutan' - Created task run 'sleep_task-f305a62b-19' for task 'sleep_task'
17:01:37.174 | INFO    | Flow run 'cherry-orangutan' - Submitted task run 'sleep_task-f305a62b-19' for execution.
17:01:37.179 | INFO    | Flow run 'cherry-orangutan' - Created task run 'sleep_task-f305a62b-6' for task 'sleep_task'
17:01:37.180 | INFO    | Flow run 'cherry-orangutan' - Submitted task run 'sleep_task-f305a62b-6' for execution.
17:01:37.182 | INFO    | Flow run 'cherry-orangutan' - Created task run 'sleep_task-f305a62b-16' for task 'sleep_task'
17:01:37.183 | INFO    | Flow run 'cherry-orangutan' - Submitted task run 'sleep_task-f305a62b-16' for execution.
17:01:37.186 | INFO    | Flow run 'cherry-orangutan' - Created task run 'sleep_task-f305a62b-9' for task 'sleep_task'
17:01:37.186 | INFO    | Flow run 'cherry-orangutan' - Submitted task run 'sleep_task-f305a62b-9' for execution.
17:01:37.192 | INFO    | Flow run 'cherry-orangutan' - Created task run 'sleep_task-f305a62b-17' for task 'sleep_task'
17:01:37.193 | INFO    | Flow run 'cherry-orangutan' - Submitted task run 'sleep_task-f305a62b-17' for execution.
17:01:37.199 | INFO    | Flow run 'cherry-orangutan' - Created task run 'sleep_task-f305a62b-15' for task 'sleep_task'
17:01:37.199 | INFO    | Flow run 'cherry-orangutan' - Submitted task run 'sleep_task-f305a62b-15' for execution.
17:01:37.201 | INFO    | Flow run 'cherry-orangutan' - Created task run 'sleep_task-f305a62b-12' for task 'sleep_task'
17:01:37.202 | INFO    | Flow run 'cherry-orangutan' - Submitted task run 'sleep_task-f305a62b-12' for execution.
17:01:37.206 | INFO    | Flow run 'cherry-orangutan' - Created task run 'sleep_task-f305a62b-18' for task 'sleep_task'
17:01:37.206 | INFO    | Flow run 'cherry-orangutan' - Submitted task run 'sleep_task-f305a62b-18' for execution.
17:01:37.456 | INFO    | Flow run 'cherry-orangutan' - Created task run 'sleep_task-f305a62b-14' for task 'sleep_task'
17:01:37.456 | INFO    | Flow run 'cherry-orangutan' - Submitted task run 'sleep_task-f305a62b-14' for execution.
17:01:37.460 | INFO    | Flow run 'cherry-orangutan' - Created task run 'sleep_task-f305a62b-5' for task 'sleep_task'
17:01:37.461 | INFO    | Flow run 'cherry-orangutan' - Submitted task run 'sleep_task-f305a62b-5' for execution.
17:01:41.672 | INFO    | Flow run 'cherry-orangutan' - Running tasks (17): [UUID('34d0c34b-cbc2-4f9f-adb9-1343d9022357'), UUID('6687156a-4b72-461a-8c1f-da2f30a4dee3'), UUID('d334dfe6-aa5e-460d-a3bc-a3fa882a5318'), UUID('be5ead11-4aa3-45b7-be37-3d022b1f6593'), UUID('30537a8f-015b-4144-b76f-8d3934397b24'), UUID('b297665c-610e-40c5-acbc-d4f627a44cca'), UUID('c0b9d5a7-6121-4c7a-9cf3-eef565cb768b'), UUID('a123de8e-6149-4357-9e21-fa45cc6578f3'), UUID('f05bd772-6977-43fc-8fea-20ddfb5f82ed'), UUID('d0322073-d5df-406c-b762-34b6f88cd735'), UUID('9d70db99-c93f-49ab-8214-b2789f71f4d7'), UUID('b507d71f-3bed-4423-96ec-abc168fff745'), UUID('3bac9d81-7184-42ac-8c82-6384ca169259'), UUID('6ddb12ca-b74d-4304-91e5-7e0394dbae9d'), UUID('94322a0f-3e25-4222-862c-f6cd46ad3a80'), UUID('41a245ca-4c0b-47e2-ac80-7e6cb14cc9c2'), UUID('578527a4-2800-4bd6-82ab-fe929312aa3f')]
17:01:41.672 | INFO    | Flow run 'cherry-orangutan' - Active tasks (5): [UUID('578527a4-2800-4bd6-82ab-fe929312aa3f'), UUID('63061c8f-a670-410d-b95e-6e7f160639f1'), UUID('a123de8e-6149-4357-9e21-fa45cc6578f3'), UUID('9d70db99-c93f-49ab-8214-b2789f71f4d7'), UUID('c0b9d5a7-6121-4c7a-9cf3-eef565cb768b')]
17:01:48.066 | INFO    | Flow run 'cherry-orangutan' - Running tasks (18): [UUID('63061c8f-a670-410d-b95e-6e7f160639f1'), UUID('34d0c34b-cbc2-4f9f-adb9-1343d9022357'), UUID('6687156a-4b72-461a-8c1f-da2f30a4dee3'), UUID('d334dfe6-aa5e-460d-a3bc-a3fa882a5318'), UUID('be5ead11-4aa3-45b7-be37-3d022b1f6593'), UUID('30537a8f-015b-4144-b76f-8d3934397b24'), UUID('b297665c-610e-40c5-acbc-d4f627a44cca'), UUID('c0b9d5a7-6121-4c7a-9cf3-eef565cb768b'), UUID('a123de8e-6149-4357-9e21-fa45cc6578f3'), UUID('f05bd772-6977-43fc-8fea-20ddfb5f82ed'), UUID('d0322073-d5df-406c-b762-34b6f88cd735'), UUID('9d70db99-c93f-49ab-8214-b2789f71f4d7'), UUID('b507d71f-3bed-4423-96ec-abc168fff745'), UUID('3bac9d81-7184-42ac-8c82-6384ca169259'), UUID('6ddb12ca-b74d-4304-91e5-7e0394dbae9d'), UUID('94322a0f-3e25-4222-862c-f6cd46ad3a80'), UUID('41a245ca-4c0b-47e2-ac80-7e6cb14cc9c2'), UUID('578527a4-2800-4bd6-82ab-fe929312aa3f')]
17:01:48.067 | INFO    | Flow run 'cherry-orangutan' - Active tasks (5): [UUID('578527a4-2800-4bd6-82ab-fe929312aa3f'), UUID('63061c8f-a670-410d-b95e-6e7f160639f1'), UUID('a123de8e-6149-4357-9e21-fa45cc6578f3'), UUID('9d70db99-c93f-49ab-8214-b2789f71f4d7'), UUID('c0b9d5a7-6121-4c7a-9cf3-eef565cb768b')]
17:01:54.367 | INFO    | Flow run 'cherry-orangutan' - Running tasks (18): [UUID('63061c8f-a670-410d-b95e-6e7f160639f1'), UUID('34d0c34b-cbc2-4f9f-adb9-1343d9022357'), UUID('6687156a-4b72-461a-8c1f-da2f30a4dee3'), UUID('d334dfe6-aa5e-460d-a3bc-a3fa882a5318'), UUID('be5ead11-4aa3-45b7-be37-3d022b1f6593'), UUID('30537a8f-015b-4144-b76f-8d3934397b24'), UUID('b297665c-610e-40c5-acbc-d4f627a44cca'), UUID('c0b9d5a7-6121-4c7a-9cf3-eef565cb768b'), UUID('a123de8e-6149-4357-9e21-fa45cc6578f3'), UUID('f05bd772-6977-43fc-8fea-20ddfb5f82ed'), UUID('d0322073-d5df-406c-b762-34b6f88cd735'), UUID('9d70db99-c93f-49ab-8214-b2789f71f4d7'), UUID('b507d71f-3bed-4423-96ec-abc168fff745'), UUID('3bac9d81-7184-42ac-8c82-6384ca169259'), UUID('6ddb12ca-b74d-4304-91e5-7e0394dbae9d'), UUID('94322a0f-3e25-4222-862c-f6cd46ad3a80'), UUID('41a245ca-4c0b-47e2-ac80-7e6cb14cc9c2'), UUID('578527a4-2800-4bd6-82ab-fe929312aa3f')]
17:01:54.368 | INFO    | Flow run 'cherry-orangutan' - Active tasks (5): [UUID('578527a4-2800-4bd6-82ab-fe929312aa3f'), UUID('63061c8f-a670-410d-b95e-6e7f160639f1'), UUID('a123de8e-6149-4357-9e21-fa45cc6578f3'), UUID('9d70db99-c93f-49ab-8214-b2789f71f4d7'), UUID('c0b9d5a7-6121-4c7a-9cf3-eef565cb768b')]
17:01:57.844 | INFO    | Task run 'sleep_task-f305a62b-0' - Finished in state Completed()
17:01:57.959 | INFO    | Task run 'sleep_task-f305a62b-8' - Finished in state Completed()
17:01:57.963 | INFO    | Task run 'sleep_task-f305a62b-1' - Finished in state Completed()
17:01:57.968 | INFO    | Task run 'sleep_task-f305a62b-2' - Finished in state Completed()
17:01:57.975 | INFO    | Task run 'sleep_task-f305a62b-11' - Finished in state Completed()
17:01:58.007 | INFO    | Task run 'sleep_task-f305a62b-16' - Finished in state Completed()
17:01:58.023 | INFO    | Task run 'sleep_task-f305a62b-10' - Finished in state Completed()
17:01:58.025 | INFO    | Task run 'sleep_task-f305a62b-15' - Finished in state Completed()
17:01:58.030 | INFO    | Task run 'sleep_task-f305a62b-9' - Finished in state Completed()
17:01:58.032 | INFO    | Task run 'sleep_task-f305a62b-7' - Finished in state Completed()
17:01:58.044 | INFO    | Task run 'sleep_task-f305a62b-13' - Finished in state Completed()
17:01:58.056 | INFO    | Task run 'sleep_task-f305a62b-18' - Finished in state Completed()
17:01:58.059 | INFO    | Task run 'sleep_task-f305a62b-6' - Finished in state Completed()
17:01:58.110 | INFO    | Task run 'sleep_task-f305a62b-19' - Finished in state Completed()
17:01:58.112 | INFO    | Task run 'sleep_task-f305a62b-4' - Finished in state Completed()
17:01:58.132 | INFO    | Task run 'sleep_task-f305a62b-3' - Finished in state Completed()
17:01:58.387 | INFO    | Task run 'sleep_task-f305a62b-12' - Finished in state Completed()
17:01:58.405 | INFO    | Task run 'sleep_task-f305a62b-17' - Finished in state Completed()
17:02:00.350 | INFO    | Flow run 'cherry-orangutan' - Running tasks (7): [UUID('63061c8f-a670-410d-b95e-6e7f160639f1'), UUID('34d0c34b-cbc2-4f9f-adb9-1343d9022357'), UUID('6687156a-4b72-461a-8c1f-da2f30a4dee3'), UUID('d334dfe6-aa5e-460d-a3bc-a3fa882a5318'), UUID('be5ead11-4aa3-45b7-be37-3d022b1f6593'), UUID('30537a8f-015b-4144-b76f-8d3934397b24'), UUID('b297665c-610e-40c5-acbc-d4f627a44cca')]
17:02:00.351 | INFO    | Flow run 'cherry-orangutan' - Active tasks (1): [UUID('c0b9d5a7-6121-4c7a-9cf3-eef565cb768b')]
17:02:06.653 | INFO    | Flow run 'cherry-orangutan' - Running tasks (0): []
17:02:06.654 | INFO    | Flow run 'cherry-orangutan' - Active tasks (1): [UUID('c0b9d5a7-6121-4c7a-9cf3-eef565cb768b')]
17:02:08.657 | INFO    | Flow run 'cherry-orangutan' - Updating limits
17:02:12.907 | INFO    | Flow run 'cherry-orangutan' - Running tasks (2): [UUID('201df0a6-f638-4a43-b693-8aa32ff567d1'), UUID('c13cc965-beba-4d61-a1f2-ce38eeccc96c')]
17:02:12.908 | INFO    | Flow run 'cherry-orangutan' - Active tasks (3): [UUID('c13cc965-beba-4d61-a1f2-ce38eeccc96c'), UUID('c0b9d5a7-6121-4c7a-9cf3-eef565cb768b'), UUID('201df0a6-f638-4a43-b693-8aa32ff567d1')]
17:02:18.628 | INFO    | Flow run 'cherry-orangutan' - Running tasks (2): [UUID('201df0a6-f638-4a43-b693-8aa32ff567d1'), UUID('c13cc965-beba-4d61-a1f2-ce38eeccc96c')]
17:02:18.629 | INFO    | Flow run 'cherry-orangutan' - Active tasks (3): [UUID('c13cc965-beba-4d61-a1f2-ce38eeccc96c'), UUID('c0b9d5a7-6121-4c7a-9cf3-eef565cb768b'), UUID('201df0a6-f638-4a43-b693-8aa32ff567d1')]
17:02:25.434 | INFO    | Flow run 'cherry-orangutan' - Running tasks (2): [UUID('201df0a6-f638-4a43-b693-8aa32ff567d1'), UUID('c13cc965-beba-4d61-a1f2-ce38eeccc96c')]
17:02:25.434 | INFO    | Flow run 'cherry-orangutan' - Active tasks (3): [UUID('c13cc965-beba-4d61-a1f2-ce38eeccc96c'), UUID('c0b9d5a7-6121-4c7a-9cf3-eef565cb768b'), UUID('201df0a6-f638-4a43-b693-8aa32ff567d1')]
17:02:28.555 | INFO    | Task run 'sleep_task-f305a62b-14' - Finished in state Completed()
17:02:28.787 | INFO    | Task run 'sleep_task-f305a62b-5' - Finished in state Completed()
17:02:31.648 | INFO    | Flow run 'cherry-orangutan' - Running tasks (1): [UUID('201df0a6-f638-4a43-b693-8aa32ff567d1')]
17:02:31.649 | INFO    | Flow run 'cherry-orangutan' - Active tasks (1): [UUID('c0b9d5a7-6121-4c7a-9cf3-eef565cb768b')]
17:02:38.202 | INFO    | Flow run 'cherry-orangutan' - Running tasks (0): []
17:02:38.204 | INFO    | Flow run 'cherry-orangutan' - Active tasks (1): [UUID('c0b9d5a7-6121-4c7a-9cf3-eef565cb768b')]
17:02:44.344 | INFO    | Flow run 'cherry-orangutan' - Running tasks (0): []
17:02:44.345 | INFO    | Flow run 'cherry-orangutan' - Active tasks (1): [UUID('c0b9d5a7-6121-4c7a-9cf3-eef565cb768b')]
17:02:50.285 | INFO    | Flow run 'cherry-orangutan' - Running tasks (0): []
17:02:50.286 | INFO    | Flow run 'cherry-orangutan' - Active tasks (1): [UUID('c0b9d5a7-6121-4c7a-9cf3-eef565cb768b')]
17:02:56.429 | INFO    | Flow run 'cherry-orangutan' - Running tasks (0): []
17:02:56.430 | INFO    | Flow run 'cherry-orangutan' - Active tasks (1): [UUID('c0b9d5a7-6121-4c7a-9cf3-eef565cb768b')]
17:03:02.778 | INFO    | Flow run 'cherry-orangutan' - Running tasks (0): []
17:03:02.779 | INFO    | Flow run 'cherry-orangutan' - Active tasks (1): [UUID('c0b9d5a7-6121-4c7a-9cf3-eef565cb768b')]
17:03:09.998 | INFO    | Flow run 'cherry-orangutan' - Running tasks (0): []
17:03:09.998 | INFO    | Flow run 'cherry-orangutan' - Active tasks (1): [UUID('c0b9d5a7-6121-4c7a-9cf3-eef565cb768b')]
17:03:16.047 | INFO    | Flow run 'cherry-orangutan' - Running tasks (0): []
17:03:16.047 | INFO    | Flow run 'cherry-orangutan' - Active tasks (1): [UUID('c0b9d5a7-6121-4c7a-9cf3-eef565cb768b')]
17:03:22.320 | INFO    | Flow run 'cherry-orangutan' - Running tasks (0): []
17:03:22.321 | INFO    | Flow run 'cherry-orangutan' - Active tasks (1): [UUID('c0b9d5a7-6121-4c7a-9cf3-eef565cb768b')]
17:03:28.480 | INFO    | Flow run 'cherry-orangutan' - Running tasks (0): []
17:03:28.482 | INFO    | Flow run 'cherry-orangutan' - Active tasks (1): [UUID('c0b9d5a7-6121-4c7a-9cf3-eef565cb768b')]
17:03:34.559 | INFO    | Flow run 'cherry-orangutan' - Running tasks (0): []
17:03:34.560 | INFO    | Flow run 'cherry-orangutan' - Active tasks (1): [UUID('c0b9d5a7-6121-4c7a-9cf3-eef565cb768b')]
17:03:40.872 | INFO    | Flow run 'cherry-orangutan' - Running tasks (0): []
17:03:40.874 | INFO    | Flow run 'cherry-orangutan' - Active tasks (1): [UUID('c0b9d5a7-6121-4c7a-9cf3-eef565cb768b')]
17:03:47.528 | INFO    | Flow run 'cherry-orangutan' - Running tasks (0): []
17:03:47.529 | INFO    | Flow run 'cherry-orangutan' - Active tasks (1): [UUID('c0b9d5a7-6121-4c7a-9cf3-eef565cb768b')]
17:03:53.361 | INFO    | Flow run 'cherry-orangutan' - Running tasks (0): []
17:03:53.361 | INFO    | Flow run 'cherry-orangutan' - Active tasks (1): [UUID('c0b9d5a7-6121-4c7a-9cf3-eef565cb768b')]
17:04:00.123 | INFO    | Flow run 'cherry-orangutan' - Running tasks (0): []
17:04:00.124 | INFO    | Flow run 'cherry-orangutan' - Active tasks (1): [UUID('c0b9d5a7-6121-4c7a-9cf3-eef565cb768b')]
17:04:06.829 | INFO    | Flow run 'cherry-orangutan' - Running tasks (0): []
17:04:06.830 | INFO    | Flow run 'cherry-orangutan' - Active tasks (1): [UUID('c0b9d5a7-6121-4c7a-9cf3-eef565cb768b')]
17:04:13.025 | INFO    | Flow run 'cherry-orangutan' - Running tasks (0): []
17:04:13.026 | INFO    | Flow run 'cherry-orangutan' - Active tasks (1): [UUID('c0b9d5a7-6121-4c7a-9cf3-eef565cb768b')]
17:04:15.278 | INFO    | Flow run 'cherry-orangutan' - Finished in state Completed('All states completed.')

Versions

Version:             2.6.1
API version:         0.8.2
Python version:      3.8.10
Git commit:          d9dd4443
Built:               Fri, Oct 14, 2022 2:01 PM
OS/Arch:             linux/x86_64
Server type:         cloud

Additional context

relates to #5995
@j-tr j-tr added bug Something isn't working status:triage labels Oct 17, 2022
@zanieb zanieb added status:accepted api Related the Prefect REST API cloud Related to Prefect Cloud and removed status:triage labels Oct 17, 2022
@zanieb
Copy link
Contributor

zanieb commented Oct 17, 2022

@anticorrelator is it possible this is a regression in the new concurrency limit implementation?

@j-tr have you encountered this in the open source version?

@anticorrelator
Copy link
Contributor

That update isn't live yet, but will be soon--I think this error seems consistent with a previous issue that I believe we're fixing with the upcoming release

@discdiver discdiver reopened this Oct 18, 2022
@discdiver
Copy link
Contributor

Still seeing some runs listed as active.

Same code as above:

(base) jeffhale demos/shelling $ python test_con.py    
14:34:06.199 | INFO    | prefect.engine - Created flow run 'astute-roadrunner' for flow 'test-limits'
14:34:07.017 | INFO    | Flow run 'astute-roadrunner' - Created task run 'sleep_task-f305a62b-0' for task 'sleep_task'
14:34:07.017 | INFO    | Flow run 'astute-roadrunner' - Submitted task run 'sleep_task-f305a62b-0' for execution.
14:34:07.056 | INFO    | Flow run 'astute-roadrunner' - Created task run 'sleep_task-f305a62b-2' for task 'sleep_task'
14:34:07.056 | INFO    | Flow run 'astute-roadrunner' - Submitted task run 'sleep_task-f305a62b-2' for execution.
14:34:07.058 | INFO    | Flow run 'astute-roadrunner' - Created task run 'sleep_task-f305a62b-7' for task 'sleep_task'
14:34:07.058 | INFO    | Flow run 'astute-roadrunner' - Submitted task run 'sleep_task-f305a62b-7' for execution.
14:34:07.061 | INFO    | Flow run 'astute-roadrunner' - Created task run 'sleep_task-f305a62b-6' for task 'sleep_task'
14:34:07.061 | INFO    | Flow run 'astute-roadrunner' - Submitted task run 'sleep_task-f305a62b-6' for execution.
14:34:07.063 | INFO    | Flow run 'astute-roadrunner' - Created task run 'sleep_task-f305a62b-14' for task 'sleep_task'
14:34:07.063 | INFO    | Flow run 'astute-roadrunner' - Submitted task run 'sleep_task-f305a62b-14' for execution.
14:34:07.064 | INFO    | Flow run 'astute-roadrunner' - Created task run 'sleep_task-f305a62b-4' for task 'sleep_task'
14:34:07.064 | INFO    | Flow run 'astute-roadrunner' - Submitted task run 'sleep_task-f305a62b-4' for execution.
14:34:07.066 | INFO    | Flow run 'astute-roadrunner' - Created task run 'sleep_task-f305a62b-8' for task 'sleep_task'
14:34:07.066 | INFO    | Flow run 'astute-roadrunner' - Submitted task run 'sleep_task-f305a62b-8' for execution.
14:34:07.069 | INFO    | Flow run 'astute-roadrunner' - Created task run 'sleep_task-f305a62b-1' for task 'sleep_task'
14:34:07.069 | INFO    | Flow run 'astute-roadrunner' - Submitted task run 'sleep_task-f305a62b-1' for execution.
14:34:07.070 | INFO    | Flow run 'astute-roadrunner' - Created task run 'sleep_task-f305a62b-18' for task 'sleep_task'
14:34:07.071 | INFO    | Flow run 'astute-roadrunner' - Submitted task run 'sleep_task-f305a62b-18' for execution.
14:34:07.072 | INFO    | Flow run 'astute-roadrunner' - Created task run 'sleep_task-f305a62b-9' for task 'sleep_task'
14:34:07.072 | INFO    | Flow run 'astute-roadrunner' - Submitted task run 'sleep_task-f305a62b-9' for execution.
14:34:07.073 | INFO    | Flow run 'astute-roadrunner' - Created task run 'sleep_task-f305a62b-17' for task 'sleep_task'
14:34:07.074 | INFO    | Flow run 'astute-roadrunner' - Submitted task run 'sleep_task-f305a62b-17' for execution.
14:34:07.079 | INFO    | Flow run 'astute-roadrunner' - Created task run 'sleep_task-f305a62b-3' for task 'sleep_task'
14:34:07.079 | INFO    | Flow run 'astute-roadrunner' - Submitted task run 'sleep_task-f305a62b-3' for execution.
14:34:07.080 | INFO    | Flow run 'astute-roadrunner' - Created task run 'sleep_task-f305a62b-10' for task 'sleep_task'
14:34:07.081 | INFO    | Flow run 'astute-roadrunner' - Submitted task run 'sleep_task-f305a62b-10' for execution.
14:34:07.085 | INFO    | Flow run 'astute-roadrunner' - Created task run 'sleep_task-f305a62b-12' for task 'sleep_task'
14:34:07.085 | INFO    | Flow run 'astute-roadrunner' - Submitted task run 'sleep_task-f305a62b-12' for execution.
14:34:07.088 | INFO    | Flow run 'astute-roadrunner' - Created task run 'sleep_task-f305a62b-5' for task 'sleep_task'
14:34:07.088 | INFO    | Flow run 'astute-roadrunner' - Submitted task run 'sleep_task-f305a62b-5' for execution.
14:34:07.091 | INFO    | Flow run 'astute-roadrunner' - Created task run 'sleep_task-f305a62b-16' for task 'sleep_task'
14:34:07.091 | INFO    | Flow run 'astute-roadrunner' - Submitted task run 'sleep_task-f305a62b-16' for execution.
14:34:07.105 | INFO    | Flow run 'astute-roadrunner' - Created task run 'sleep_task-f305a62b-11' for task 'sleep_task'
14:34:07.105 | INFO    | Flow run 'astute-roadrunner' - Submitted task run 'sleep_task-f305a62b-11' for execution.
14:34:07.111 | INFO    | Flow run 'astute-roadrunner' - Created task run 'sleep_task-f305a62b-13' for task 'sleep_task'
14:34:07.111 | INFO    | Flow run 'astute-roadrunner' - Submitted task run 'sleep_task-f305a62b-13' for execution.
14:34:07.137 | INFO    | Flow run 'astute-roadrunner' - Created task run 'sleep_task-f305a62b-15' for task 'sleep_task'
14:34:07.138 | INFO    | Flow run 'astute-roadrunner' - Submitted task run 'sleep_task-f305a62b-15' for execution.
14:34:07.478 | INFO    | Flow run 'astute-roadrunner' - Created task run 'sleep_task-f305a62b-19' for task 'sleep_task'
14:34:07.478 | INFO    | Flow run 'astute-roadrunner' - Submitted task run 'sleep_task-f305a62b-19' for execution.
14:34:08.821 | INFO    | Flow run 'astute-roadrunner' - Running tasks (4): [UUID('f48813f6-8226-4557-8b1a-1a072380961a'), UUID('6aa2e4f7-8332-41e1-bca3-673cda208ac7'), UUID('aa9b031f-4c4b-4bfd-be18-2389053eb10b'), UUID('ab33ad84-9d2f-4e7f-aea6-6604bcb5df49')]
14:34:08.822 | INFO    | Flow run 'astute-roadrunner' - Active tasks (5): [UUID('63bf234c-a4de-49c8-8133-ea19d847a193'), UUID('cb910ad0-126e-4571-af30-fb6a72006454'), UUID('ab33ad84-9d2f-4e7f-aea6-6604bcb5df49'), UUID('e32434f1-e056-4698-b51b-acc497dc4a08'), UUID('1ee75ef1-2b5f-4a0a-90bf-6b75e2902715')]
14:34:12.412 | INFO    | Flow run 'astute-roadrunner' - Running tasks (5): [UUID('e32434f1-e056-4698-b51b-acc497dc4a08'), UUID('f48813f6-8226-4557-8b1a-1a072380961a'), UUID('6aa2e4f7-8332-41e1-bca3-673cda208ac7'), UUID('aa9b031f-4c4b-4bfd-be18-2389053eb10b'), UUID('ab33ad84-9d2f-4e7f-aea6-6604bcb5df49')]
14:34:12.412 | INFO    | Flow run 'astute-roadrunner' - Active tasks (5): [UUID('63bf234c-a4de-49c8-8133-ea19d847a193'), UUID('cb910ad0-126e-4571-af30-fb6a72006454'), UUID('ab33ad84-9d2f-4e7f-aea6-6604bcb5df49'), UUID('e32434f1-e056-4698-b51b-acc497dc4a08'), UUID('1ee75ef1-2b5f-4a0a-90bf-6b75e2902715')]
14:34:16.046 | INFO    | Flow run 'astute-roadrunner' - Running tasks (5): [UUID('e32434f1-e056-4698-b51b-acc497dc4a08'), UUID('f48813f6-8226-4557-8b1a-1a072380961a'), UUID('6aa2e4f7-8332-41e1-bca3-673cda208ac7'), UUID('aa9b031f-4c4b-4bfd-be18-2389053eb10b'), UUID('ab33ad84-9d2f-4e7f-aea6-6604bcb5df49')]
14:34:16.048 | INFO    | Flow run 'astute-roadrunner' - Active tasks (5): [UUID('63bf234c-a4de-49c8-8133-ea19d847a193'), UUID('cb910ad0-126e-4571-af30-fb6a72006454'), UUID('ab33ad84-9d2f-4e7f-aea6-6604bcb5df49'), UUID('e32434f1-e056-4698-b51b-acc497dc4a08'), UUID('1ee75ef1-2b5f-4a0a-90bf-6b75e2902715')]
14:34:19.805 | INFO    | Flow run 'astute-roadrunner' - Running tasks (5): [UUID('e32434f1-e056-4698-b51b-acc497dc4a08'), UUID('f48813f6-8226-4557-8b1a-1a072380961a'), UUID('6aa2e4f7-8332-41e1-bca3-673cda208ac7'), UUID('aa9b031f-4c4b-4bfd-be18-2389053eb10b'), UUID('ab33ad84-9d2f-4e7f-aea6-6604bcb5df49')]
14:34:19.806 | INFO    | Flow run 'astute-roadrunner' - Active tasks (5): [UUID('63bf234c-a4de-49c8-8133-ea19d847a193'), UUID('cb910ad0-126e-4571-af30-fb6a72006454'), UUID('ab33ad84-9d2f-4e7f-aea6-6604bcb5df49'), UUID('e32434f1-e056-4698-b51b-acc497dc4a08'), UUID('1ee75ef1-2b5f-4a0a-90bf-6b75e2902715')]
14:34:23.408 | INFO    | Flow run 'astute-roadrunner' - Running tasks (5): [UUID('e32434f1-e056-4698-b51b-acc497dc4a08'), UUID('f48813f6-8226-4557-8b1a-1a072380961a'), UUID('6aa2e4f7-8332-41e1-bca3-673cda208ac7'), UUID('aa9b031f-4c4b-4bfd-be18-2389053eb10b'), UUID('ab33ad84-9d2f-4e7f-aea6-6604bcb5df49')]
14:34:23.409 | INFO    | Flow run 'astute-roadrunner' - Active tasks (5): [UUID('63bf234c-a4de-49c8-8133-ea19d847a193'), UUID('cb910ad0-126e-4571-af30-fb6a72006454'), UUID('ab33ad84-9d2f-4e7f-aea6-6604bcb5df49'), UUID('e32434f1-e056-4698-b51b-acc497dc4a08'), UUID('1ee75ef1-2b5f-4a0a-90bf-6b75e2902715')]
14:34:26.605 | INFO    | Flow run 'astute-roadrunner' - Running tasks (5): [UUID('e32434f1-e056-4698-b51b-acc497dc4a08'), UUID('f48813f6-8226-4557-8b1a-1a072380961a'), UUID('6aa2e4f7-8332-41e1-bca3-673cda208ac7'), UUID('aa9b031f-4c4b-4bfd-be18-2389053eb10b'), UUID('ab33ad84-9d2f-4e7f-aea6-6604bcb5df49')]
14:34:26.606 | INFO    | Flow run 'astute-roadrunner' - Active tasks (5): [UUID('63bf234c-a4de-49c8-8133-ea19d847a193'), UUID('cb910ad0-126e-4571-af30-fb6a72006454'), UUID('ab33ad84-9d2f-4e7f-aea6-6604bcb5df49'), UUID('e32434f1-e056-4698-b51b-acc497dc4a08'), UUID('1ee75ef1-2b5f-4a0a-90bf-6b75e2902715')]
14:34:27.311 | INFO    | Task run 'sleep_task-f305a62b-0' - Finished in state Completed()
14:34:27.356 | INFO    | Task run 'sleep_task-f305a62b-1' - Finished in state Completed()
14:34:27.363 | INFO    | Task run 'sleep_task-f305a62b-2' - Finished in state Completed()
14:34:27.371 | INFO    | Task run 'sleep_task-f305a62b-11' - Finished in state Completed()
14:34:27.436 | INFO    | Task run 'sleep_task-f305a62b-7' - Finished in state Completed()
14:34:29.698 | INFO    | Flow run 'astute-roadrunner' - Running tasks (0): []
14:34:29.699 | INFO    | Flow run 'astute-roadrunner' - Active tasks (3): [UUID('cb910ad0-126e-4571-af30-fb6a72006454'), UUID('1ee75ef1-2b5f-4a0a-90bf-6b75e2902715'), UUID('63bf234c-a4de-49c8-8133-ea19d847a193')]
14:34:32.999 | INFO    | Flow run 'astute-roadrunner' - Running tasks (0): []
14:34:33.000 | INFO    | Flow run 'astute-roadrunner' - Active tasks (3): [UUID('cb910ad0-126e-4571-af30-fb6a72006454'), UUID('1ee75ef1-2b5f-4a0a-90bf-6b75e2902715'), UUID('63bf234c-a4de-49c8-8133-ea19d847a193')]
14:34:36.556 | INFO    | Flow run 'astute-roadrunner' - Running tasks (0): []
14:34:36.556 | INFO    | Flow run 'astute-roadrunner' - Active tasks (3): [UUID('cb910ad0-126e-4571-af30-fb6a72006454'), UUID('1ee75ef1-2b5f-4a0a-90bf-6b75e2902715'), UUID('63bf234c-a4de-49c8-8133-ea19d847a193')]
14:34:38.561 | INFO    | Flow run 'astute-roadrunner' - Updating limits
14:34:39.765 | INFO    | Flow run 'astute-roadrunner' - Running tasks (4): [UUID('c3c08c5e-17ac-4569-9e5f-d919749e7830'), UUID('383d6ce6-7192-4172-bf0e-2ceb1bb92669'), UUID('35d252ea-947f-442c-9ae0-649cc855b779'), UUID('ed9c9bd6-31a3-4a39-94e4-ce85ba481e73')]
14:34:39.767 | INFO    | Flow run 'astute-roadrunner' - Active tasks (5): [UUID('1ee75ef1-2b5f-4a0a-90bf-6b75e2902715'), UUID('35d252ea-947f-442c-9ae0-649cc855b779'), UUID('cb910ad0-126e-4571-af30-fb6a72006454'), UUID('63bf234c-a4de-49c8-8133-ea19d847a193'), UUID('c3c08c5e-17ac-4569-9e5f-d919749e7830')]
14:34:43.259 | INFO    | Flow run 'astute-roadrunner' - Running tasks (4): [UUID('c3c08c5e-17ac-4569-9e5f-d919749e7830'), UUID('383d6ce6-7192-4172-bf0e-2ceb1bb92669'), UUID('35d252ea-947f-442c-9ae0-649cc855b779'), UUID('ed9c9bd6-31a3-4a39-94e4-ce85ba481e73')]
14:34:43.260 | INFO    | Flow run 'astute-roadrunner' - Active tasks (5): [UUID('1ee75ef1-2b5f-4a0a-90bf-6b75e2902715'), UUID('35d252ea-947f-442c-9ae0-649cc855b779'), UUID('cb910ad0-126e-4571-af30-fb6a72006454'), UUID('63bf234c-a4de-49c8-8133-ea19d847a193'), UUID('c3c08c5e-17ac-4569-9e5f-d919749e7830')]
14:34:46.397 | INFO    | Flow run 'astute-roadrunner' - Running tasks (4): [UUID('c3c08c5e-17ac-4569-9e5f-d919749e7830'), UUID('383d6ce6-7192-4172-bf0e-2ceb1bb92669'), UUID('35d252ea-947f-442c-9ae0-649cc855b779'), UUID('ed9c9bd6-31a3-4a39-94e4-ce85ba481e73')]
14:34:46.398 | INFO    | Flow run 'astute-roadrunner' - Active tasks (5): [UUID('1ee75ef1-2b5f-4a0a-90bf-6b75e2902715'), UUID('35d252ea-947f-442c-9ae0-649cc855b779'), UUID('cb910ad0-126e-4571-af30-fb6a72006454'), UUID('63bf234c-a4de-49c8-8133-ea19d847a193'), UUID('c3c08c5e-17ac-4569-9e5f-d919749e7830')]
14:34:49.683 | INFO    | Flow run 'astute-roadrunner' - Running tasks (4): [UUID('c3c08c5e-17ac-4569-9e5f-d919749e7830'), UUID('383d6ce6-7192-4172-bf0e-2ceb1bb92669'), UUID('35d252ea-947f-442c-9ae0-649cc855b779'), UUID('ed9c9bd6-31a3-4a39-94e4-ce85ba481e73')]
14:34:49.685 | INFO    | Flow run 'astute-roadrunner' - Active tasks (5): [UUID('1ee75ef1-2b5f-4a0a-90bf-6b75e2902715'), UUID('35d252ea-947f-442c-9ae0-649cc855b779'), UUID('cb910ad0-126e-4571-af30-fb6a72006454'), UUID('63bf234c-a4de-49c8-8133-ea19d847a193'), UUID('c3c08c5e-17ac-4569-9e5f-d919749e7830')]
14:34:53.226 | INFO    | Flow run 'astute-roadrunner' - Running tasks (4): [UUID('c3c08c5e-17ac-4569-9e5f-d919749e7830'), UUID('383d6ce6-7192-4172-bf0e-2ceb1bb92669'), UUID('35d252ea-947f-442c-9ae0-649cc855b779'), UUID('ed9c9bd6-31a3-4a39-94e4-ce85ba481e73')]
14:34:53.227 | INFO    | Flow run 'astute-roadrunner' - Active tasks (5): [UUID('1ee75ef1-2b5f-4a0a-90bf-6b75e2902715'), UUID('35d252ea-947f-442c-9ae0-649cc855b779'), UUID('cb910ad0-126e-4571-af30-fb6a72006454'), UUID('63bf234c-a4de-49c8-8133-ea19d847a193'), UUID('c3c08c5e-17ac-4569-9e5f-d919749e7830')]
14:34:56.883 | INFO    | Flow run 'astute-roadrunner' - Running tasks (4): [UUID('c3c08c5e-17ac-4569-9e5f-d919749e7830'), UUID('383d6ce6-7192-4172-bf0e-2ceb1bb92669'), UUID('35d252ea-947f-442c-9ae0-649cc855b779'), UUID('ed9c9bd6-31a3-4a39-94e4-ce85ba481e73')]
14:34:56.884 | INFO    | Flow run 'astute-roadrunner' - Active tasks (5): [UUID('1ee75ef1-2b5f-4a0a-90bf-6b75e2902715'), UUID('35d252ea-947f-442c-9ae0-649cc855b779'), UUID('cb910ad0-126e-4571-af30-fb6a72006454'), UUID('63bf234c-a4de-49c8-8133-ea19d847a193'), UUID('c3c08c5e-17ac-4569-9e5f-d919749e7830')]
14:34:57.416 | INFO    | Task run 'sleep_task-f305a62b-3' - Finished in state Completed()
14:34:57.490 | INFO    | Task run 'sleep_task-f305a62b-12' - Finished in state Completed()
14:34:57.492 | INFO    | Task run 'sleep_task-f305a62b-17' - Finished in state Completed()
14:34:57.495 | INFO    | Task run 'sleep_task-f305a62b-18' - Finished in state Completed()
14:35:00.431 | INFO    | Flow run 'astute-roadrunner' - Running tasks (0): []
14:35:00.433 | INFO    | Flow run 'astute-roadrunner' - Active tasks (4): [UUID('35d252ea-947f-442c-9ae0-649cc855b779'), UUID('63bf234c-a4de-49c8-8133-ea19d847a193'), UUID('cb910ad0-126e-4571-af30-fb6a72006454'), UUID('1ee75ef1-2b5f-4a0a-90bf-6b75e2902715')]
14:35:03.724 | INFO    | Flow run 'astute-roadrunner' - Running tasks (0): []
14:35:03.726 | INFO    | Flow run 'astute-roadrunner' - Active tasks (4): [UUID('35d252ea-947f-442c-9ae0-649cc855b779'), UUID('63bf234c-a4de-49c8-8133-ea19d847a193'), UUID('cb910ad0-126e-4571-af30-fb6a72006454'), UUID('1ee75ef1-2b5f-4a0a-90bf-6b75e2902715')]
14:35:06.919 | INFO    | Flow run 'astute-roadrunner' - Running tasks (0): []
14:35:06.920 | INFO    | Flow run 'astute-roadrunner' - Active tasks (4): [UUID('35d252ea-947f-442c-9ae0-649cc855b779'), UUID('63bf234c-a4de-49c8-8133-ea19d847a193'), UUID('cb910ad0-126e-4571-af30-fb6a72006454'), UUID('1ee75ef1-2b5f-4a0a-90bf-6b75e2902715')]
14:35:10.189 | INFO    | Flow run 'astute-roadrunner' - Running tasks (7): [UUID('cf790a3d-8920-4ed3-866d-17fe291e830e'), UUID('0d53c1f9-90aa-4d31-a218-55e7683da72c'), UUID('b651e991-16ef-4852-ae4a-04c176553bbf'), UUID('87257f4a-d884-4c65-8d81-7f1b06d7c9c2'), UUID('f17df682-af05-4ac6-a37b-ebaa481f1ff9'), UUID('4c35cec1-e502-41ba-b6e3-f7d1f970e6a4'), UUID('74945339-b7a6-427c-bfd7-046e0a506754')]
14:35:10.190 | INFO    | Flow run 'astute-roadrunner' - Active tasks (10): [UUID('74945339-b7a6-427c-bfd7-046e0a506754'), UUID('cb910ad0-126e-4571-af30-fb6a72006454'), UUID('1ee75ef1-2b5f-4a0a-90bf-6b75e2902715'), UUID('87257f4a-d884-4c65-8d81-7f1b06d7c9c2'), UUID('b651e991-16ef-4852-ae4a-04c176553bbf'), UUID('0d53c1f9-90aa-4d31-a218-55e7683da72c'), UUID('35d252ea-947f-442c-9ae0-649cc855b779'), UUID('cf790a3d-8920-4ed3-866d-17fe291e830e'), UUID('f17df682-af05-4ac6-a37b-ebaa481f1ff9'), UUID('63bf234c-a4de-49c8-8133-ea19d847a193')]
14:35:13.690 | INFO    | Flow run 'astute-roadrunner' - Running tasks (7): [UUID('cf790a3d-8920-4ed3-866d-17fe291e830e'), UUID('0d53c1f9-90aa-4d31-a218-55e7683da72c'), UUID('b651e991-16ef-4852-ae4a-04c176553bbf'), UUID('87257f4a-d884-4c65-8d81-7f1b06d7c9c2'), UUID('f17df682-af05-4ac6-a37b-ebaa481f1ff9'), UUID('4c35cec1-e502-41ba-b6e3-f7d1f970e6a4'), UUID('74945339-b7a6-427c-bfd7-046e0a506754')]
14:35:13.691 | INFO    | Flow run 'astute-roadrunner' - Active tasks (10): [UUID('74945339-b7a6-427c-bfd7-046e0a506754'), UUID('cb910ad0-126e-4571-af30-fb6a72006454'), UUID('1ee75ef1-2b5f-4a0a-90bf-6b75e2902715'), UUID('87257f4a-d884-4c65-8d81-7f1b06d7c9c2'), UUID('b651e991-16ef-4852-ae4a-04c176553bbf'), UUID('0d53c1f9-90aa-4d31-a218-55e7683da72c'), UUID('35d252ea-947f-442c-9ae0-649cc855b779'), UUID('cf790a3d-8920-4ed3-866d-17fe291e830e'), UUID('f17df682-af05-4ac6-a37b-ebaa481f1ff9'), UUID('63bf234c-a4de-49c8-8133-ea19d847a193')]
14:35:17.042 | INFO    | Flow run 'astute-roadrunner' - Running tasks (7): [UUID('cf790a3d-8920-4ed3-866d-17fe291e830e'), UUID('0d53c1f9-90aa-4d31-a218-55e7683da72c'), UUID('b651e991-16ef-4852-ae4a-04c176553bbf'), UUID('87257f4a-d884-4c65-8d81-7f1b06d7c9c2'), UUID('f17df682-af05-4ac6-a37b-ebaa481f1ff9'), UUID('4c35cec1-e502-41ba-b6e3-f7d1f970e6a4'), UUID('74945339-b7a6-427c-bfd7-046e0a506754')]
14:35:17.043 | INFO    | Flow run 'astute-roadrunner' - Active tasks (10): [UUID('74945339-b7a6-427c-bfd7-046e0a506754'), UUID('cb910ad0-126e-4571-af30-fb6a72006454'), UUID('1ee75ef1-2b5f-4a0a-90bf-6b75e2902715'), UUID('87257f4a-d884-4c65-8d81-7f1b06d7c9c2'), UUID('b651e991-16ef-4852-ae4a-04c176553bbf'), UUID('0d53c1f9-90aa-4d31-a218-55e7683da72c'), UUID('35d252ea-947f-442c-9ae0-649cc855b779'), UUID('cf790a3d-8920-4ed3-866d-17fe291e830e'), UUID('f17df682-af05-4ac6-a37b-ebaa481f1ff9'), UUID('63bf234c-a4de-49c8-8133-ea19d847a193')]
14:35:20.494 | INFO    | Flow run 'astute-roadrunner' - Running tasks (7): [UUID('cf790a3d-8920-4ed3-866d-17fe291e830e'), UUID('0d53c1f9-90aa-4d31-a218-55e7683da72c'), UUID('b651e991-16ef-4852-ae4a-04c176553bbf'), UUID('87257f4a-d884-4c65-8d81-7f1b06d7c9c2'), UUID('f17df682-af05-4ac6-a37b-ebaa481f1ff9'), UUID('4c35cec1-e502-41ba-b6e3-f7d1f970e6a4'), UUID('74945339-b7a6-427c-bfd7-046e0a506754')]
14:35:20.496 | INFO    | Flow run 'astute-roadrunner' - Active tasks (10): [UUID('74945339-b7a6-427c-bfd7-046e0a506754'), UUID('cb910ad0-126e-4571-af30-fb6a72006454'), UUID('1ee75ef1-2b5f-4a0a-90bf-6b75e2902715'), UUID('87257f4a-d884-4c65-8d81-7f1b06d7c9c2'), UUID('b651e991-16ef-4852-ae4a-04c176553bbf'), UUID('0d53c1f9-90aa-4d31-a218-55e7683da72c'), UUID('35d252ea-947f-442c-9ae0-649cc855b779'), UUID('cf790a3d-8920-4ed3-866d-17fe291e830e'), UUID('f17df682-af05-4ac6-a37b-ebaa481f1ff9'), UUID('63bf234c-a4de-49c8-8133-ea19d847a193')]
14:35:23.782 | INFO    | Flow run 'astute-roadrunner' - Running tasks (7): [UUID('cf790a3d-8920-4ed3-866d-17fe291e830e'), UUID('0d53c1f9-90aa-4d31-a218-55e7683da72c'), UUID('b651e991-16ef-4852-ae4a-04c176553bbf'), UUID('87257f4a-d884-4c65-8d81-7f1b06d7c9c2'), UUID('f17df682-af05-4ac6-a37b-ebaa481f1ff9'), UUID('4c35cec1-e502-41ba-b6e3-f7d1f970e6a4'), UUID('74945339-b7a6-427c-bfd7-046e0a506754')]
14:35:23.783 | INFO    | Flow run 'astute-roadrunner' - Active tasks (10): [UUID('74945339-b7a6-427c-bfd7-046e0a506754'), UUID('cb910ad0-126e-4571-af30-fb6a72006454'), UUID('1ee75ef1-2b5f-4a0a-90bf-6b75e2902715'), UUID('87257f4a-d884-4c65-8d81-7f1b06d7c9c2'), UUID('b651e991-16ef-4852-ae4a-04c176553bbf'), UUID('0d53c1f9-90aa-4d31-a218-55e7683da72c'), UUID('35d252ea-947f-442c-9ae0-649cc855b779'), UUID('cf790a3d-8920-4ed3-866d-17fe291e830e'), UUID('f17df682-af05-4ac6-a37b-ebaa481f1ff9'), UUID('63bf234c-a4de-49c8-8133-ea19d847a193')]
14:35:27.070 | INFO    | Flow run 'astute-roadrunner' - Running tasks (7): [UUID('cf790a3d-8920-4ed3-866d-17fe291e830e'), UUID('0d53c1f9-90aa-4d31-a218-55e7683da72c'), UUID('b651e991-16ef-4852-ae4a-04c176553bbf'), UUID('87257f4a-d884-4c65-8d81-7f1b06d7c9c2'), UUID('f17df682-af05-4ac6-a37b-ebaa481f1ff9'), UUID('4c35cec1-e502-41ba-b6e3-f7d1f970e6a4'), UUID('74945339-b7a6-427c-bfd7-046e0a506754')]
14:35:27.072 | INFO    | Flow run 'astute-roadrunner' - Active tasks (10): [UUID('74945339-b7a6-427c-bfd7-046e0a506754'), UUID('cb910ad0-126e-4571-af30-fb6a72006454'), UUID('1ee75ef1-2b5f-4a0a-90bf-6b75e2902715'), UUID('87257f4a-d884-4c65-8d81-7f1b06d7c9c2'), UUID('b651e991-16ef-4852-ae4a-04c176553bbf'), UUID('0d53c1f9-90aa-4d31-a218-55e7683da72c'), UUID('35d252ea-947f-442c-9ae0-649cc855b779'), UUID('cf790a3d-8920-4ed3-866d-17fe291e830e'), UUID('f17df682-af05-4ac6-a37b-ebaa481f1ff9'), UUID('63bf234c-a4de-49c8-8133-ea19d847a193')]
14:35:27.567 | INFO    | Task run 'sleep_task-f305a62b-6' - Finished in state Completed()
14:35:27.680 | INFO    | Task run 'sleep_task-f305a62b-10' - Finished in state Completed()
14:35:27.807 | INFO    | Task run 'sleep_task-f305a62b-15' - Finished in state Completed()
14:35:27.814 | INFO    | Task run 'sleep_task-f305a62b-8' - Finished in state Completed()
14:35:27.846 | INFO    | Task run 'sleep_task-f305a62b-9' - Finished in state Completed()
14:35:27.962 | INFO    | Task run 'sleep_task-f305a62b-13' - Finished in state Completed()
14:35:28.016 | INFO    | Task run 'sleep_task-f305a62b-19' - Finished in state Completed()
14:35:30.286 | INFO    | Flow run 'astute-roadrunner' - Running tasks (0): []
14:35:30.287 | INFO    | Flow run 'astute-roadrunner' - Active tasks (4): [UUID('63bf234c-a4de-49c8-8133-ea19d847a193'), UUID('35d252ea-947f-442c-9ae0-649cc855b779'), UUID('1ee75ef1-2b5f-4a0a-90bf-6b75e2902715'), UUID('cb910ad0-126e-4571-af30-fb6a72006454')]
14:35:33.579 | INFO    | Flow run 'astute-roadrunner' - Running tasks (0): []
14:35:33.580 | INFO    | Flow run 'astute-roadrunner' - Active tasks (4): [UUID('63bf234c-a4de-49c8-8133-ea19d847a193'), UUID('35d252ea-947f-442c-9ae0-649cc855b779'), UUID('1ee75ef1-2b5f-4a0a-90bf-6b75e2902715'), UUID('cb910ad0-126e-4571-af30-fb6a72006454')]
14:35:37.463 | INFO    | Flow run 'astute-roadrunner' - Running tasks (0): []
14:35:37.465 | INFO    | Flow run 'astute-roadrunner' - Active tasks (4): [UUID('63bf234c-a4de-49c8-8133-ea19d847a193'), UUID('35d252ea-947f-442c-9ae0-649cc855b779'), UUID('1ee75ef1-2b5f-4a0a-90bf-6b75e2902715'), UUID('cb910ad0-126e-4571-af30-fb6a72006454')]
14:35:40.743 | INFO    | Flow run 'astute-roadrunner' - Running tasks (4): [UUID('eddbfb08-9a99-4937-828d-bf2509f7da99'), UUID('b1de9cb5-e825-40ec-aaca-bce20c573726'), UUID('6e2237df-964e-4ea3-b58f-0dadf950b7d8'), UUID('69ce5a29-1c7d-4b87-abc0-ac169a036846')]
14:35:40.744 | INFO    | Flow run 'astute-roadrunner' - Active tasks (7): [UUID('35d252ea-947f-442c-9ae0-649cc855b779'), UUID('69ce5a29-1c7d-4b87-abc0-ac169a036846'), UUID('63bf234c-a4de-49c8-8133-ea19d847a193'), UUID('eddbfb08-9a99-4937-828d-bf2509f7da99'), UUID('6e2237df-964e-4ea3-b58f-0dadf950b7d8'), UUID('1ee75ef1-2b5f-4a0a-90bf-6b75e2902715'), UUID('cb910ad0-126e-4571-af30-fb6a72006454')]
14:35:44.317 | INFO    | Flow run 'astute-roadrunner' - Running tasks (4): [UUID('eddbfb08-9a99-4937-828d-bf2509f7da99'), UUID('b1de9cb5-e825-40ec-aaca-bce20c573726'), UUID('6e2237df-964e-4ea3-b58f-0dadf950b7d8'), UUID('69ce5a29-1c7d-4b87-abc0-ac169a036846')]
14:35:44.318 | INFO    | Flow run 'astute-roadrunner' - Active tasks (7): [UUID('35d252ea-947f-442c-9ae0-649cc855b779'), UUID('69ce5a29-1c7d-4b87-abc0-ac169a036846'), UUID('63bf234c-a4de-49c8-8133-ea19d847a193'), UUID('eddbfb08-9a99-4937-828d-bf2509f7da99'), UUID('6e2237df-964e-4ea3-b58f-0dadf950b7d8'), UUID('1ee75ef1-2b5f-4a0a-90bf-6b75e2902715'), UUID('cb910ad0-126e-4571-af30-fb6a72006454')]
14:35:47.331 | INFO    | Flow run 'astute-roadrunner' - Running tasks (4): [UUID('eddbfb08-9a99-4937-828d-bf2509f7da99'), UUID('b1de9cb5-e825-40ec-aaca-bce20c573726'), UUID('6e2237df-964e-4ea3-b58f-0dadf950b7d8'), UUID('69ce5a29-1c7d-4b87-abc0-ac169a036846')]
14:35:47.332 | INFO    | Flow run 'astute-roadrunner' - Active tasks (7): [UUID('35d252ea-947f-442c-9ae0-649cc855b779'), UUID('69ce5a29-1c7d-4b87-abc0-ac169a036846'), UUID('63bf234c-a4de-49c8-8133-ea19d847a193'), UUID('eddbfb08-9a99-4937-828d-bf2509f7da99'), UUID('6e2237df-964e-4ea3-b58f-0dadf950b7d8'), UUID('1ee75ef1-2b5f-4a0a-90bf-6b75e2902715'), UUID('cb910ad0-126e-4571-af30-fb6a72006454')]
14:35:50.434 | INFO    | Flow run 'astute-roadrunner' - Running tasks (4): [UUID('eddbfb08-9a99-4937-828d-bf2509f7da99'), UUID('b1de9cb5-e825-40ec-aaca-bce20c573726'), UUID('6e2237df-964e-4ea3-b58f-0dadf950b7d8'), UUID('69ce5a29-1c7d-4b87-abc0-ac169a036846')]
14:35:50.435 | INFO    | Flow run 'astute-roadrunner' - Active tasks (7): [UUID('35d252ea-947f-442c-9ae0-649cc855b779'), UUID('69ce5a29-1c7d-4b87-abc0-ac169a036846'), UUID('63bf234c-a4de-49c8-8133-ea19d847a193'), UUID('eddbfb08-9a99-4937-828d-bf2509f7da99'), UUID('6e2237df-964e-4ea3-b58f-0dadf950b7d8'), UUID('1ee75ef1-2b5f-4a0a-90bf-6b75e2902715'), UUID('cb910ad0-126e-4571-af30-fb6a72006454')]
14:35:53.501 | INFO    | Flow run 'astute-roadrunner' - Running tasks (4): [UUID('eddbfb08-9a99-4937-828d-bf2509f7da99'), UUID('b1de9cb5-e825-40ec-aaca-bce20c573726'), UUID('6e2237df-964e-4ea3-b58f-0dadf950b7d8'), UUID('69ce5a29-1c7d-4b87-abc0-ac169a036846')]
14:35:53.502 | INFO    | Flow run 'astute-roadrunner' - Active tasks (7): [UUID('35d252ea-947f-442c-9ae0-649cc855b779'), UUID('69ce5a29-1c7d-4b87-abc0-ac169a036846'), UUID('63bf234c-a4de-49c8-8133-ea19d847a193'), UUID('eddbfb08-9a99-4937-828d-bf2509f7da99'), UUID('6e2237df-964e-4ea3-b58f-0dadf950b7d8'), UUID('1ee75ef1-2b5f-4a0a-90bf-6b75e2902715'), UUID('cb910ad0-126e-4571-af30-fb6a72006454')]
14:35:56.552 | INFO    | Flow run 'astute-roadrunner' - Running tasks (4): [UUID('eddbfb08-9a99-4937-828d-bf2509f7da99'), UUID('b1de9cb5-e825-40ec-aaca-bce20c573726'), UUID('6e2237df-964e-4ea3-b58f-0dadf950b7d8'), UUID('69ce5a29-1c7d-4b87-abc0-ac169a036846')]
14:35:56.553 | INFO    | Flow run 'astute-roadrunner' - Active tasks (7): [UUID('35d252ea-947f-442c-9ae0-649cc855b779'), UUID('69ce5a29-1c7d-4b87-abc0-ac169a036846'), UUID('63bf234c-a4de-49c8-8133-ea19d847a193'), UUID('eddbfb08-9a99-4937-828d-bf2509f7da99'), UUID('6e2237df-964e-4ea3-b58f-0dadf950b7d8'), UUID('1ee75ef1-2b5f-4a0a-90bf-6b75e2902715'), UUID('cb910ad0-126e-4571-af30-fb6a72006454')]
14:35:58.095 | INFO    | Task run 'sleep_task-f305a62b-14' - Finished in state Completed()
14:35:58.172 | INFO    | Task run 'sleep_task-f305a62b-16' - Finished in state Completed()
14:35:58.266 | INFO    | Task run 'sleep_task-f305a62b-5' - Finished in state Completed()
14:35:58.279 | INFO    | Task run 'sleep_task-f305a62b-4' - Finished in state Completed()
14:35:59.680 | INFO    | Flow run 'astute-roadrunner' - Running tasks (0): []
14:35:59.682 | INFO    | Flow run 'astute-roadrunner' - Active tasks (4): [UUID('35d252ea-947f-442c-9ae0-649cc855b779'), UUID('63bf234c-a4de-49c8-8133-ea19d847a193'), UUID('cb910ad0-126e-4571-af30-fb6a72006454'), UUID('1ee75ef1-2b5f-4a0a-90bf-6b75e2902715')]
14:36:03.590 | INFO    | Flow run 'astute-roadrunner' - Running tasks (0): []
14:36:03.591 | INFO    | Flow run 'astute-roadrunner' - Active tasks (4): [UUID('35d252ea-947f-442c-9ae0-649cc855b779'), UUID('63bf234c-a4de-49c8-8133-ea19d847a193'), UUID('cb910ad0-126e-4571-af30-fb6a72006454'), UUID('1ee75ef1-2b5f-4a0a-90bf-6b75e2902715')]
14:36:06.703 | INFO    | Flow run 'astute-roadrunner' - Running tasks (0): []
14:36:06.704 | INFO    | Flow run 'astute-roadrunner' - Active tasks (4): [UUID('35d252ea-947f-442c-9ae0-649cc855b779'), UUID('63bf234c-a4de-49c8-8133-ea19d847a193'), UUID('cb910ad0-126e-4571-af30-fb6a72006454'), UUID('1ee75ef1-2b5f-4a0a-90bf-6b75e2902715')]
14:36:09.958 | INFO    | Flow run 'astute-roadrunner' - Running tasks (0): []
14:36:09.959 | INFO    | Flow run 'astute-roadrunner' - Active tasks (4): [UUID('35d252ea-947f-442c-9ae0-649cc855b779'), UUID('63bf234c-a4de-49c8-8133-ea19d847a193'), UUID('cb910ad0-126e-4571-af30-fb6a72006454'), UUID('1ee75ef1-2b5f-4a0a-90bf-6b75e2902715')]
14:36:13.437 | INFO    | Flow run 'astute-roadrunner' - Running tasks (0): []
14:36:13.438 | INFO    | Flow run 'astute-roadrunner' - Active tasks (4): [UUID('35d252ea-947f-442c-9ae0-649cc855b779'), UUID('63bf234c-a4de-49c8-8133-ea19d847a193'), UUID('cb910ad0-126e-4571-af30-fb6a72006454'), UUID('1ee75ef1-2b5f-4a0a-90bf-6b75e2902715')]
14:36:16.795 | INFO    | Flow run 'astute-roadrunner' - Running tasks (0): []
14:36:16.796 | INFO    | Flow run 'astute-roadrunner' - Active tasks (4): [UUID('35d252ea-947f-442c-9ae0-649cc855b779'), UUID('63bf234c-a4de-49c8-8133-ea19d847a193'), UUID('cb910ad0-126e-4571-af30-fb6a72006454'), UUID('1ee75ef1-2b5f-4a0a-90bf-6b75e2902715')]
14:36:20.011 | INFO    | Flow run 'astute-roadrunner' - Running tasks (0): []
14:36:20.012 | INFO    | Flow run 'astute-roadrunner' - Active tasks (4): [UUID('35d252ea-947f-442c-9ae0-649cc855b779'), UUID('63bf234c-a4de-49c8-8133-ea19d847a193'), UUID('cb910ad0-126e-4571-af30-fb6a72006454'), UUID('1ee75ef1-2b5f-4a0a-90bf-6b75e2902715')]
14:36:23.490 | INFO    | Flow run 'astute-roadrunner' - Running tasks (0): []
14:36:23.490 | INFO    | Flow run 'astute-roadrunner' - Active tasks (4): [UUID('35d252ea-947f-442c-9ae0-649cc855b779'), UUID('63bf234c-a4de-49c8-8133-ea19d847a193'), UUID('cb910ad0-126e-4571-af30-fb6a72006454'), UUID('1ee75ef1-2b5f-4a0a-90bf-6b75e2902715')]
14:36:26.578 | INFO    | Flow run 'astute-roadrunner' - Running tasks (0): []
14:36:26.578 | INFO    | Flow run 'astute-roadrunner' - Active tasks (4): [UUID('35d252ea-947f-442c-9ae0-649cc855b779'), UUID('63bf234c-a4de-49c8-8133-ea19d847a193'), UUID('cb910ad0-126e-4571-af30-fb6a72006454'), UUID('1ee75ef1-2b5f-4a0a-90bf-6b75e2902715')]
14:36:30.126 | INFO    | Flow run 'astute-roadrunner' - Running tasks (0): []
14:36:30.128 | INFO    | Flow run 'astute-roadrunner' - Active tasks (4): [UUID('35d252ea-947f-442c-9ae0-649cc855b779'), UUID('63bf234c-a4de-49c8-8133-ea19d847a193'), UUID('cb910ad0-126e-4571-af30-fb6a72006454'), UUID('1ee75ef1-2b5f-4a0a-90bf-6b75e2902715')]
14:36:34.116 | INFO    | Flow run 'astute-roadrunner' - Running tasks (0): []
14:36:34.117 | INFO    | Flow run 'astute-roadrunner' - Active tasks (4): [UUID('35d252ea-947f-442c-9ae0-649cc855b779'), UUID('63bf234c-a4de-49c8-8133-ea19d847a193'), UUID('cb910ad0-126e-4571-af30-fb6a72006454'), UUID('1ee75ef1-2b5f-4a0a-90bf-6b75e2902715')]
14:36:37.641 | INFO    | Flow run 'astute-roadrunner' - Running tasks (0): []
14:36:37.642 | INFO    | Flow run 'astute-roadrunner' - Active tasks (4): [UUID('35d252ea-947f-442c-9ae0-649cc855b779'), UUID('63bf234c-a4de-49c8-8133-ea19d847a193'), UUID('cb910ad0-126e-4571-af30-fb6a72006454'), UUID('1ee75ef1-2b5f-4a0a-90bf-6b75e2902715')]
14:36:39.722 | INFO    | Flow run 'astute-roadrunner' - Finished in state Completed('All states completed.')

@github-actions
Copy link
Contributor

This issue is stale because it has been open 30 days with no activity. To keep this issue open remove stale label or comment.

@github-actions
Copy link
Contributor

This issue was closed because it has been stale for 14 days with no activity. If this issue is important or you have more to add feel free to re-open it.

@github-actions github-actions bot closed this as not planned Won't fix, can't repro, duplicate, stale Apr 13, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
api Related the Prefect REST API bug Something isn't working cloud Related to Prefect Cloud
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@anticorrelator @zanieb @discdiver @j-tr