Skip to content

Coerce tensorboard metrics into numpy arrays #21504

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
pganssle-google wants to merge 3 commits into
base: master
Choose a base branch
from
+12 −0
Open

Coerce tensorboard metrics into numpy arrays #21504

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
1fd2114
Coerce tensorboard metrics into numpy arrays
pganssle-google Jan 23, 2026
57cee0f
update
deependujha Jan 24, 2026
a93d2e4
[pre-commit.ci] auto fixes from pre-commit.com hooks
pre-commit-ci[bot] Jan 24, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions src/lightning/fabric/CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,8 @@ The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/).

- Fixed DDP notebook CUDA fork check to allow passive initialization when CUDA is not actively used ([#21402](https://github.com/Lightning-AI/pytorch-lightning/pull/21402))

- Added logic to prevent numpy from trying to convert arrays with ndim > 0 to scalar in tensorboard, which is an error starting in numpy 2.4.0. ([#21503](https://github.com/Lightning-AI/pytorch-lightning/issues/21503))

### Removed

-
Expand Down
10 changes: 10 additions & 0 deletions src/lightning/fabric/loggers/tensorboard.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@

from lightning.fabric.loggers.logger import Logger, rank_zero_experiment
from lightning.fabric.utilities.cloud_io import _is_dir, get_filesystem
from lightning.fabric.utilities.imports import _NUMPY_AVAILABLE
from lightning.fabric.utilities.logger import _add_prefix, _convert_params, _flatten_dict
from lightning.fabric.utilities.logger import _sanitize_params as _utils_sanitize_params
from lightning.fabric.utilities.rank_zero import rank_zero_only, rank_zero_warn
Expand Down Expand Up @@ -202,10 +203,19 @@ def log_metrics(self, metrics: Mapping[str, float], step: Optional[int] = None)

metrics = _add_prefix(metrics, self._prefix, self.LOGGER_JOIN_CHAR)

if _NUMPY_AVAILABLE:
import numpy as np

for k, v in metrics.items():
if isinstance(v, Tensor):
v = v.item()

if _NUMPY_AVAILABLE:
if isinstance(v, (int, float, np.number)):
v = np.array(v)
elif isinstance(v, np.ndarray) and v.ndim > 0 and v.size == 1:
v = np.array(v.item())
Comment on lines +214 to +217
Copy link
Collaborator

@deependujha deependujha Jan 27, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
if isinstance(v, (int, float, np.number)):
v = np.array(v)
elif isinstance(v, np.ndarray) and v.ndim > 0 and v.size == 1:
v = np.array(v.item())
if isinstance(v, (np.number)) or isinstance(v, np.ndarray) and v.ndim > 0 and v.size == 1:
v = float(v.item())

Why are we converting them into np.array? Is there any additional benefit?

Copy link
Author

@pganssle-google pganssle-google Jan 27, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When I skipped that part the tests failed, at least for me.

TBH, this whole thing is a bug in tensorboardX. tensorboardX does some inappropriate conversion with scalars which hits the deprecated-now-removed code path. IIRC, if you don't coerce to a numpy array with the right shape before passing it to tensorboardX, it does it anyway, it just does it incorrectly.

So I guess it's up to you how much you want to have code here to work around a bug upstream. In practical terms it means this part of the library is not compatible with numpy 2.4 until tensorboardX is fixed, which may never happen, and even if it does, it will require a more recent version of tensorboardX to work properly. But from a purity perspective, this isn't actually your bug 😅


if isinstance(v, dict):
self.experiment.add_scalars(k, v, step)
else:
Expand Down
Loading