[Feature]: separate tally of warnings vs notes in warnings.sso?

[iantaylor-NOAA]

Describe the solution you would like.

The warning file contains messages that are just informative and don't require any action, such as
"NOTE: Max data length bin: 200 < max pop len bins: 250; so will accumulate larger pop len bins" in
https://github.com/nmfs-stock-synthesis/test-models/blob/main/models/BigSkate_2019/warning.sso#L1-L5

It would be helpful for users to be able to separate the
Would it be possible to change the write_warning function: https://github.com/nmfs-stock-synthesis/stock-synthesis/blob/6b092fdc34dc347f6766163c3c9e5546de6c63f5/SS_global.tpl#L194-L201
so that it provides a separate tally of notes vs warnings, like:

N warnings: 3
N notes: 1

Right now I only see 4 instances of warnstream << "note in the source code, but more could be added in the future especially if they don't get added to the warnings tally so users can be more relaxed when they see them.

Tagging @MOshima-PIFSC who's query prompted this.

Describe alternatives you have considered

Leave things as they are.

Statistical validity, if applicable

No response

Describe if this is needed for a management application

No response

Additional context

No response

[Rick-Methot-NOAA]

Ha! We could create a warning severity scale.
Seriously, this will require some scrutiny of many warning messages to determine the threshold.

[k-doering-NOAA]

Good point, @iantaylor-NOAA , I have gotten emails from users before trying to get rid of all the warnings, when some of these are notes. For example, one that seems to come a lot is when users include -1 timing setting for a fishery, they get a warning about this setting, and it seems to worry folks who intentionally chose this setting.

Another one I can think of is the recr_dist_method 4 note:
https://github.com/nmfs-stock-synthesis/stock-synthesis/blob/fe1acdb48b43994b8f87e4621ae2a938e46e7fc9/SS_readcontrol_330.tpl#L212

[iantaylor-NOAA]

Good point, Rick about needing to decide on a threshold.
An easy alternative would be to make clear to users that zero warnings should not be the goal.

[Rick-Methot-NOAA]

Here is my first attempt to pull out a list of all lines with either the new "warnstream" format or old "warnings". Many warning message get missed because they are on a following wrapped line of the code. Another category to use are those "warnings" that are telling user that SS3 made a change on their behalf.
Possible categories:

• FATAL: fatal input error
• ADJUSTED: SS3 detected problem with input and modified. will continue with run
• WARNING: out of bounds or incorrect or ambiguous input, but still might play through
• SUGGESTION: a better approach is available

warn_list2.txt

[iantaylor-NOAA]

I like all these categories. I think that adding the category to the message would be enough and we wouldn't need a separate tally for each category (though we could change the line at the end to "N messages:"

I converted your text file into this spreadsheet where we could start slowly categorizing things: https://docs.google.com/spreadsheets/d/1B4bsMdrnB8s9qWrWNA3dQDSc7UzZY6llFvcN2iJHHfo/edit#gid=0

Did you want to also retain the "NOTE" category, which could be described like "NOTE: information for the user that may not require any further changes"?

[Rick-Methot-NOAA]

I like the addition of NOTE category.

Let's wait on this idea as it will be a bit of work to do. Also, what I extracted is a distorted list of the relevant warning messages. Better after more get converted to the warnstream approach so they can be extracted more easily. Neal - what is your advice in that regard? @nschindler-noaa

[nschindler-noaa]

I like this idea. It is one I have worked with in another application.
I suggest that write_warning would add a descriptive at the beginning such as "!Warning: . . . ", "*Suggestion: . . . ". etc.
We can also create #defines for FATAl, WARN, SUGGEST, ADJUST, and NOTE, so that, to call write_warning, it would take the form
warnstream << "some value is out of line, so was adjusted, because we know what you meant";
write_warning (ADJUST, 1, 1) // writing about an adjustment to echoinput and to warning.sso
cout would be written to automatically for FATAL and perhaps ADJUST to get the user's attention.
I think it makes sense to track only Warnings, Suggestions, and Adjustments

[nschindler-noaa]

Question: do you want to track each of these separately (except Fatal, since there can only be one!) ?? Or, track Warnings, Suggestions, and Adjustments together with Notes separate?

[Rick-Methot-NOAA]

I do not see great merit in separate counts, but probably better to err on side of creating these counts now while the patient is on the table. The main point of having any count is to get the user to look at warning.sso.
I was thinking of just adding the category into the text of the warning, but your idea to make it an entity itself seems good.
Another category is model performance (bad gradients, poor MSY convergence, etc.)

[nschindler-noaa]

Those would fall into the category of Suggestions, wouldn't they?

[Rick-Methot-NOAA]

No. I think that model performance is separate from suggested changes to model setup.

[nschindler-noaa]

Okay, here is my current thinking:
Categories (1-6) increasing in importance: Note information:, Note suggestion:, Note performance:, Warning error:, Warning adjustment:, Warning FATAL!
I consider Notes as information that does not require user action and Warnings as actions the user should take.
The notes and warnings tracked separately so at the end we can notify the user of each number (per Ian above).

[Rick-Methot-NOAA]

Looks good.