Cluster dump utilities

[sjperkins]

• Closes Utility functions to debug cluster_dump #5873
• Tests added / passed
• Passes pre-commit run --all-files

Based on #5863 by @gjoseph92 which this PR will track further.

[gjoseph92]

Are you using much code from #5863, or just the cluster_dump.py file? If just the file, it might be worth suffering some easy merge conflicts to have a much smaller diff here.

[github-actions]

Unit Test Results

       12 files  ±  0         12 suites  ±0   7h 45m 57s ⏱️ + 25m 21s
  2 669 tests +  3    2 587 ✔️ +  2    80 💤 ±0  2 ❌ +1 
15 926 runs  +18  15 061 ✔️ +11  862 💤 +5  3 ❌ +2 

For more details on these failures, see this check.

Results for commit 9f81cfc. ± Comparison against base commit 4866615.

♻️ This comment has been updated with latest results.

[sjperkins]

Are you using much code from #5863, or just the cluster_dump.py file? If just the file, it might be worth suffering some easy merge conflicts to have a much smaller diff here.

I'm just using cluster_dump.py. I'll go with your suggestion.

[fjetter]

I just merged #5863. Could you rebase/merge? This would make a review much easier.

[sjperkins]

I've done some manual testing with the dataset from #5675 as the default added tests are not very representative of real world cluster deadlocks.

[fjetter]

A method that would likely be helpful is to split an existing dump artifact up into multiple files in case we need to look into the raw data. I've been using the below snippet to do this in the past.

@sjperkins would you mind adding something like this?

I see you are having a bunch of merge conflicts. This is due to us changing imports to absolute. They should be easy enough to fix, see #5924

logs = load_cluster_dump()


top_level = "workers"
workers = logs[top_level]
for ix, (worker, info) in enumerate(workers.items()):
    print(f"Processing worker {worker} {ix+1}/{len(workers)}")
    try:
        for name, _logs in info.items():

            directory = os.path.join(main_dir, top_level, worker)
            if not os.path.exists(directory):
                os.makedirs(directory)
            with open(os.path.join(directory, f"{name}.yaml"), "w") as fd:
                dump(_logs, fd, Dumper=Dumper)
    except:
        pass


import json
top_level = "scheduler"
info = logs[top_level]
for name, logs in info.items():
    directory = os.path.join(main_dir, top_level)
    if not os.path.exists(directory):
        os.makedirs(directory)
    with open(os.path.join(directory, f"{name}.yaml"), "w") as fd:
        dump(logs, fd, Dumper=Dumper)

[sjperkins]

A method that would likely be helpful is to split an existing dump artifact up into multiple files in case we need to look into the raw data. I've been using the below snippet to do this in the past.

@sjperkins would you mind adding something like this?

Yes, I'll add it to the DumpInspector class.

I see you are having a bunch of merge conflicts. This is due to us changing imports to absolute. They should be easy enough to fix, see #5924

I've resolved these locally.

[gjoseph92]

@fjetter or @crusaderky this just needs a merge. Also @sjperkins maybe you should merge main, see if it helps CI at all?

[fjetter]

Test failures appear to be unrelated, thank you @sjperkins !

[sjperkins]

Test failures appear to be unrelated, thank you @sjperkins !

First Coiled commit. Woop woop!