ref: clarify about types of metrics/plots

[jorgeorpinel]

Follow up to #2839 (review)

Both the metrics and plots cmd refs. explain 2 types of "metrics" (metrics and plots) -- done to to keep both references connected.

Calling plots a kind of metric is already confusing. Furthermore, in #2839 the concept of "2 types of plots files" was introduced (data series rendered with Vega vs. arbitrary image files) to the latter, which may be confusing.

• Also, the plots usage block needs wrapping to avoid scroll

  

UPDATE: Jump to #2956 (comment)

[jorgeorpinel]

Maybe we need to put the types of metrics in a proper metrics/plots guide and not explain the concepts in the ref...

@dberenbaum do you think we the core team should address this for now, or plan for a guide? Thanks

[dberenbaum]

Maybe we need to put the types of metrics in a proper metrics/plots guide and not explain the concepts in the ref...

👍

I do think the current plots command ref doesn't incorporate image support enough at the moment. It still refers mostly to "plots metrics," and I don't think the "Types of Metrics" section at the top makes much sense now.

Also, it doesn't look like it's addressed in any of show, diff, or modify, nor is it specified anywhere which options are supported for image files.

[pared]

When the time comes, maybe we need some visual aid to explain how we handle different metrics?

          +--------+                              
          | metric |                              
          +--------+                              
          *         *                             
        **           **                           
       *               *                          
+---------+         +-------+                     
| metrics |         | plots |                     
+---------+        *+-------+**                   
                 **            **                 
               **                **               
             **                    **             
    +-------------+         +-------------------+ 
    | image_plots |         | data_series_plots | 
    +-------------+         +-------------------+ 

[dberenbaum]

I like the tree, but do we consider image plots as metrics at all? They could be actual images, like computer vision examples. I'm not sure it's intuitive to think of an image file as a type of metric.

[daavoo]

I like the tree, but do we consider image plots as metrics at all? They could be actual images, like computer vision examples. I'm not sure it's intuitive to think of an image file as a type of metric.

Also... parallel coordinates plots 🤔 (would be part of data_series_plots?)

[jorgeorpinel]

do we consider image plots as metrics at all?

I think this is the key question. By "plots" we refer both to metrics that come in "data series" form (term "data" here is also not ideal) which can be visualized by plot classic (using Vega), and to custom images that can be anything really (not necessarily a plot per se right?).

Conceptually I'd organize it like:

metrics

• scalar (short text file)
• series (long text file)
• image (binary file)

Separately,

plots feature (may need renaming)

• generate from series metrics
• associate/render custom user images

But at that point this becomes a product question because command names and semantics or even file schemes could be changed a bit.

So should we transfer to the core repo? Try to patch the explanations in the ref. for now and/or plan for a guide? Thanks

[dberenbaum]

Since we are still clarifying the types of metrics and plots and we already have the discussion here, why don't we decide how to describe them here and then decide what we need to edit in help text, command reference, user guide, etc.?

I would only describe scalars as metrics because:

1. It's what I think of as metrics and what I generally see described as ML metrics: https://neptune.ai/blog/performance-metrics-in-machine-learning-complete-guide.
2. It's consistent with language in experiment tracking tools like mlflow and wandb. Mlflow specifically defines metrics as numeric key-value pairs. Wandb doesn't define metrics but only uses the term when describing scalars.
3. It fits with the dvc metrics terminology.

I don't think we should describe any kind of plots as metrics.

[jorgeorpinel]

why don't we decide how to describe them here
I don't think we should describe any kind of plots as metrics.

OK in this case we mainly have to edit the command refs. for both metrics and plots, and unlink them (don't call plots metrics).

Then we need to incorporate this approach to #2925 (Get Started) Cc @iesahin

Finally, to consider it as well for #2572 (User Guide)

[iesahin]

I have separated plots from metrics/params to describe them more independently in #3050

A visualization section can describe plots as CSV files or plots as images, as you see fit.

[jorgeorpinel]

we mainly have to edit the command refs. for both metrics and plots, and unlink them (don't call plots metrics)

Should we expect a PR from the core team on this? Or do we consider it docs work as the feature is already documented (in a way we don't love) @dberenbaum