Skip to content

BUG: Progressbar does not update on streamed stdout #7757

Open
@tvwenger

Description

@tvwenger

Describe the issue:

I'm running pymc in a condor HTC environment, where my models are sampled on compute nodes and the output is streamed or copied back to the host.

Without any options to checkpoint sampling with pymc, I want a way to check the status of the sampling even though I don't have access to stdout on the compute nodes.

Currently, we only pass refresh=True on the last iteration, so the only time condor streams stdout back to the host machine is after the last iteration. This makes me sad because no updates are streamed back to the host during sampling, so I have no way to satisfy my curiosity on the expected runtime.

In summary, the issue is that progressbar=True doesn't actually print the progress bar to stdout in such a way that condor can stream it until sampling is completed.

I want refresh=True here for all iterations, not just the last:

pymc/pymc/util.py

Lines 893 to 908 in 2842401

self._progress.update(
self.tasks[chain_idx],
completed=draw,
draws=draw,
sampling_speed=speed,
speed_unit=unit,
**more_updates,
)
if is_last:
self._progress.update(
self.tasks[chain_idx],
draws=draw + 1 if not self.combined_progress else draw,
**more_updates,
refresh=True,
)

Reproduceable code example:

pm.sample() # on HTC

Error message:

PyMC version information:

Latest

Context for the issue:

No response

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions