Refactor TrainLogger

[kacpnowak]

Is your feature request related to a problem? Please describe.

The current logging system for training metrics has two critical shortcomings:

• Lack of granular logging: It is currently impossible to log loss function values for individual channels within each data stream. This limits visibility into model performance at a per-channel level, hindering detailed analysis.

• Fragile and unreadable format: Metrics are stored in unstructured plain-text (txt) files, which are not human-readable and lack a consistent schema. Minor changes to the order of logged metrics or their contents break backward compatibility, making post-processing, visualization, or comparison across runs error-prone and unmaintainable.

Describe the solution you'd like

Replace the current plain-text logging format with CSV files using multi-level column headers to organize metrics hierarchically.

This structure would:

• Separate global training statistics (e.g., epoch, total loss) from per-stream and per-channel metrics.

• Ensure human readability while maintaining machine-readability.

• Prevent compatibility breaks due to explicit column naming and hierarchical organization.

Example:

global ,global                     ,global  ,global   ,global   ,global                 ,global              ,FESOM               ,FESOM                ,FESOM               ,FESOM               ,FESOM               ,FESOM              ,FESOM              ,FESOM                 ,FESOM                 ,FESOM                 ,FESOM               ,FESOM               ,FESOM              ,FESOM
step   ,time                       ,samples ,perf_gpu ,perf_mem ,learning_rate          ,loss_mean           ,mse                 ,a_ice                ,evap                ,fh                  ,fw                  ,prec               ,snow               ,ssh                   ,sss                   ,sst                   ,swr                 ,tx_sur              ,ty_sur             ,std
1      ,2025-02-27 14:38:31.542443 ,320     ,98.5     ,48.75    ,2.9802549459833396e-06 ,0.9765826463699341  ,0.9765826463699341  ,0.9043928980827332   ,0.9301251173019409  ,0.972670316696167   ,0.936567485332489   ,0.9416562914848328 ,1.0318517684936523 ,1.0079926252365112    ,1.0319256782531738    ,0.941299319267273     ,0.9138585329055786  ,1.0519397258758545  ,1.054713249206543  ,

This solution was implemented in my fork: kacpnowak#3

Describe alternatives you've considered

An alternative solution is to use an embedded database (e.g., DuckDB or SQLite) to store metrics in structured tables. Benefits include:

• Support for complex queries across multiple training runs.
• Native schema enforcement, eliminating fragility caused by format changes.
• Efficient storage and retrieval of large-scale experiments.

However, CSV files provide a simpler, more accessible intermediate solution that meets immediate needs without introducing database dependencies. Either approach would be preferable to the current system, which is unmaintainable and error-prone due to its reliance on unstructured text.

Additional context

The current logging implementation’s lack of structure and flexibility actively impedes debugging, analysis, and iterative improvements. A structured format (CSV or database) is critical for scaling experimentation and ensuring reproducibility.

Organisation

AWI

[clessig]

Thanks @kacpnowak .

• Any changes in the logger should keep in mind that we ultimately want to support multiple backends (mlflow, wandb).
• On the questions of the channels: why does this need to be part of the logging. In my experience, having this as part of the analysis was sufficient (when training with multiple datasets also the number of channels explodes).
• A "real" DB is in my opinion not an option through the long list of dependencies and complications that is introduced through this
• Backward compatibility: the implemented requirements were that the file generated by TrainLogger was readable with the TrainLogger read() method. That's quite standard, in my opinion.

What is the hierarchical structure you suggest. The coarse level is streams, then channels, then different metrics per channel? Are hierarchical CSVs standardized, e.g. could I just import it into excel or any other standard software with CSV support?

[kacpnowak]

Thank you for the feedback! To clarify:

• The proposed CSV-based logging is designed to be a backend-agnostic foundation. Structured CSV files can serve as a universal format that integrates cleanly with existing backends (e.g., MLflow/WandB can log CSV artifacts directly).
• Excel Compatibility: Multi-level CSV headers (e.g., stream1/channel1/loss) are readable in Excel, as demonstrated in this StackOverflow example. While Excel won’t render hierarchical headers natively, the data remains accessible and filterable. For better visualization, we could flatten headers (e.g., stream1_channel1_loss) without sacrificing structure.
  Most data frameworks (Pandas, Polars, etc.) natively support multi-level columns via pd.read_csv + header=[0,1].
• Per-Channel Logging: This is critical for debugging multi-stream models during training (e.g., isolating unstable channels). We can make this optional via a flag to reduce clutter.
• I agree that embedded databases introduce undue complexity for most users. The CSV proposal intentionally avoids this, prioritizing simplicity while addressing the core issues.
• To ensure backward compatibility we retain the current TrainLogger.read() method to parse the new CSV format (no breaking changes)

[clessig]

The training instability is curious. Which channel was this? It was properly normalized to zero mean/unit variance? But that's really for a different issue.

I remain a bit unconvinced about hierarchical CSV. We could have one CSV per stream? @tjhunter : any opinion on this matter?

[clessig]

Having one CSV file per stream seems like the better technical solution since it makes the logging more modular and one can easily access the data for one stream and with any standard tool that can read CSV.

Once this is in place, please open a PR with the changes and we can merge it to develop.

[kacpnowak]

Thank you for considering the proposal. While splitting logs into one CSV per stream is technically feasible, this approach introduces significant drawbacks compared to a single structured CSV:

1. Inefficiency and Overhead

• Inode exhaustion: Generating 2x (num_of_streams + 1) of CSV files bloats filesystems, especially in large-scale experiments. We are already producing a lot of files anyway.
• I/O bottlenecks: Opening/reading dozens of files per training run is slower than reading a single file, particularly on network-mounted storage.

2. Reduced Usability

• Fragmented analysis: Cross-stream comparisons (e.g., correlating channel losses across streams) require merging multiple files, adding complexity to post-processing scripts.
• Tool compatibility: Most visualization tools (TensorBoard, pandas, Excel) are optimized for single-file analysis. Forcing users to manually concatenate CSVs undermines the "standard tool" argument.

3. No Compelling Advantages

• "Modularity" is achievable in a single file: Filtering by stream in a structured CSV (e.g., df.filter(regex="stream1.*") in pandas) is trivial and equally performant.
• Logs are small: Training metrics (even for 100 streams) can barely exceed MBs in size — loading them entirely into memory is not a practical concern.

Proposed Path Forward

1. Adopt a single CSV with hierarchical columns as the default.
2. Add a split_log_to_streams() method to optionally export per-stream CSVs post-hoc for niche use cases.

This balances efficiency, usability, and flexibility without proliferating files. If the team remains committed to per-stream CSVs, please clarify the technical rationale so I can address specific concerns.

---

P.S.: The training instability was just an argument, it might happen to someone.

[clessig]

• I seriously doubt that reading O(10) files will be an I/O bottleneck (even on large file systems with their exotic behavior, though one maybe should keep the file handle open). Similar for inodes, O(10) compared to O(1) files cannot be a bottleneck (if it is then the machine will be completely unusable).

• No need to merge multiple files. All what is needed is reading multiple files and np.concatenate/stack if you want to align the different streams.
• Opposite: reading standard CSV files is easier and more reliable. Also, we always need to read multiple files since the config is in a separate file and it will always be needed, in my experience.

• The advantage is modularity and having standard CSV files. The models we will be training and targeting will have O(10) streams and then this will become clearer (I have trained models with 15-20 streams). I don't see a serious downside for the 1-stream case (then both solutions should become identical ...).

P.S. I have seen streams and individual channels go NaN during training, so it's something I dealt with before. We need to focus on the features we need and avoid feature bloat--we don't have the resources to support hypothetical use cases in the code that are unused.

[kacpnowak]

Thank you for your insights and expertise — your experience building this framework is invaluable, and I appreciate your focus on practicality. However, as we develop this tool for a broader audience, I believe prioritizing user-friendliness is critical.

---

**1.

• Indeed it's unlikely that either soution will make a difference in performace and resource consumption, but why use more resources while making things more complicated then they need to be?

---

**2.

• Less experienced users: Requiring them to write custom scripts to merge (even with np.concatenate) adds friction and room for error. Additionally, one needs to find those channel that are correlatating across all files. A unified log allows immediate analysis with tools like Excel, pandas, or TensorBoard — no coding required.
• Debugging urgency: When a channel’s loss diverges, correlating failures across streams in a single file is faster than hunting through dozens of CSVs.
• Personal dependency: I use per-channel metrics in every training run to detect issues like:
  • Channels with loss values 10–100x higher than others
  • Channels that collapse to NaN/zero early in training
    A single log makes this routine analysis seamless without a need for any analysis software.

---

Compromise: Flexibility Without Bloat
To respect your modularity preference:

• Default to a single CSV (e.g., training_log.csv) with columns like stream1.channel1, so it so has single-line header. The hierahical logic is retained, so one can easily filter the columns.
• Add a utils.split_log_to_streams(log_path) helper for users who opt-in to per-stream files post-training.

This ensures:

• Beginner-friendly defaults: "Just works" for most users.
• No breaking changes: Retain backward compatibility with existing workflows.

[tjhunter]

I suggest that we move away from CSV for logging the metrics. It is not flexible enough as mentioned above (the schema has to be fully determined before writing the first record). Incidentally, all the ML logging systems (tensorboard, mlflow, WandB) use a semi-structured event scheme. If we want to be flexible, the best way in my opinion is to store a JSON dictionary for each row in a text file:

'{"foo":1,"bar":6}\n
{"foo":2,"bar":7}\n
{"foo":3,"bar":8}\n'

1. it is most likely as fast as CSV (never underestimate the amount of work that has gone into JSON!)
2. each row can contain whichever info to log: the schema can evolve along the way -> no need to centralize
3. it is fast to parse as a pandas DF. If you really want performance, then you can use polars which is using vectorized rust code under the hood:
   https://docs.pola.rs/api/python/dev/reference/api/polars.read_ndjson.html
4. We will have more metrics coming from different places (for example, we have a sophisticated loading system -> it will be good to understand what happens there)

If you want a CSV view on some columns, it is a 1-liner in python:

polars.read_ndjson("logs.json").select(["epoch", "loss"]).drop_na().to_csv("loss.csv")

pandas:
https://www.reddit.com/r/learnpython/comments/12bjqgd/python_pandas_convert_a_list_of_json_strings_into/?rdt=46368

[clessig]

Sounds good to me. Just one point that is not clear to me: you would have one json file per row in the text file then or would there be one json file with a dict per row?

…

________________________________ From: Timothy Hunter ***@***.***> Sent: Friday, March 7, 2025 4:58:06 PM To: ecmwf/WeatherGenerator ***@***.***> Cc: Christian Lessig ***@***.***>; Comment ***@***.***> Subject: Re: [ecmwf/WeatherGenerator] Refactor TrainLogger (Issue #29) I suggest that we move away from CSV for logging the metrics. It is not flexible enough as mentioned above (the schema has to be fully determined before writing the first record). Incidentally, all the ML logging systems (tensorboard, mlflow, WandB) use a semi-structured event scheme. If we want to be flexible, the best way in my opinion is to store a JSON dictionary for each row in a text file: '{"foo":1,"bar":6}\n {"foo":2,"bar":7}\n {"foo":3,"bar":8}\n' 1. it is most likely as fast as CSV (never underestimate the amount of work that has gone into JSON!) 2. each row can contain whichever info to log: the schema can evolve along the way -> no need to centralize 3. it is fast to parse as a pandas DF. If you really want performance, then you can use polars which is using vectorized rust code under the hood: https://docs.pola.rs/api/python/dev/reference/api/polars.read_ndjson.html 4. We will have more metrics coming from different places (for example, we have a sophisticated loading system -> it will be good to understand what happens there) If you want a CSV view on some columns, it is a 1-liner in python: polars.read_ndjson("logs.json").select(["epoch", "loss"]).drop_na().to_csv("loss.csv") pandas: https://www.reddit.com/r/learnpython/comments/12bjqgd/python_pandas_convert_a_list_of_json_strings_into/?rdt=46368 — Reply to this email directly, view it on GitHub<#29 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AHCHOHT7NLCZYPUZIHL2JE32TG6Y5AVCNFSM6AAAAABYAMTCL6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDOMBWHAZDEOJYG4>. You are receiving this because you commented.Message ID: ***@***.***> [tjhunter]tjhunter left a comment (ecmwf/WeatherGenerator#29)<#29 (comment)> I suggest that we move away from CSV for logging the metrics. It is not flexible enough as mentioned above (the schema has to be fully determined before writing the first record). Incidentally, all the ML logging systems (tensorboard, mlflow, WandB) use a semi-structured event scheme. If we want to be flexible, the best way in my opinion is to store a JSON dictionary for each row in a text file: '{"foo":1,"bar":6}\n {"foo":2,"bar":7}\n {"foo":3,"bar":8}\n' 1. it is most likely as fast as CSV (never underestimate the amount of work that has gone into JSON!) 2. each row can contain whichever info to log: the schema can evolve along the way -> no need to centralize 3. it is fast to parse as a pandas DF. If you really want performance, then you can use polars which is using vectorized rust code under the hood: https://docs.pola.rs/api/python/dev/reference/api/polars.read_ndjson.html 4. We will have more metrics coming from different places (for example, we have a sophisticated loading system -> it will be good to understand what happens there) If you want a CSV view on some columns, it is a 1-liner in python: polars.read_ndjson("logs.json").select(["epoch", "loss"]).drop_na().to_csv("loss.csv") pandas: https://www.reddit.com/r/learnpython/comments/12bjqgd/python_pandas_convert_a_list_of_json_strings_into/?rdt=46368 — Reply to this email directly, view it on GitHub<#29 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AHCHOHT7NLCZYPUZIHL2JE32TG6Y5AVCNFSM6AAAAABYAMTCL6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDOMBWHAZDEOJYG4>. You are receiving this because you commented.Message ID: ***@***.***>