[Fleet] Testing over Ability to Set Agent Log Level in Kibana Fleet UI

[EricDavisX]

This is the test ticket for:
#81872
elastic/beats#20756

PRs:
#83707
#84112

See issues for details and this design issue for context:
#81872 (comment)

A/C for feature:

• User can access the log-level drop down in UI and pick the desired level, which changes it at that moment and so subsequent logs will be shown including logs up to that level
• User can see the drop down in the bottom left part of the page underneath the Agent logs*
• The default log level is 'Info' for all Agents deployed to Fleet
• User sees levels presented from the top down in increasing verbosity 'error, warning, info, debug'
• For Stand-alone agent, this can be set in the yaml file and when Agent is re-started the new log level is displayed (this is not a change to spec, just re-iterating expectations).
• Endpoint log level will be modified as well to any new value set
• Beats log levels will be modified as well to any new value set

• screenshot, feature should look like this:

Testing Considerations:

• are tests needed for all variants of Agent? macOS / Windows / Linux: yes
  Recommend to tie the testing together with the Logs-in-Kibana-UI feature:
  [Fleet] Agent logs UI #83356
  in this ticket: [Fleet] Testing over Agent collection of Agent & Endpoint log files beats#22549

Where are the tests?

• Unit tests
• API tests / Agent side tests
• e2e-testing
• manual tests?

[elasticmachine]

Pinging @elastic/ingest-management (Team:Ingest Management)

[dikshachauhan-qasource]

Hi @EricDavisX

We have created 02 testcases for points shown below, as per AC mentioned in ticket. Links are as follows:
The default log level is 'Info' for all Agents deployed to Fleet : https://elastic.testrail.io/index.php?/cases/view/35115
User sees levels presented from the top down in increasing verbosity 'error, warning, info, debug' : https://elastic.testrail.io/index.php?/cases/view/35118

For below point
User can access the log-level drop down in Kibana and pick the desired level, which changes it at that moment and so subsequent logs will be shown at that level
Testcase case has been created already at : https://elastic.testrail.io/index.php?/tests/view/156490

However, we have queries for below points:

Query 1:
For Stand-alone agent, this can be set in the yml file and when Agent is re-started the new log level is displayed (this is not a change to spec, just re-iterating expectations).

• Could you please confirm if we need to create testcase here for all operating systems will all packages option we have like for linux: tar, deb and rpm
  or single testcase : 'Validate a new log as set in the yml file for stand alone agent is displayed under log level filter on Agents page. ' is enough.

Query 2:
Endpoint log level will be modified as well to any new value set

• Could you please provide more information to us on this regarding location where this new value set will be displayed.

Query 3:
Beats log levels will be modified as well to any new value set

• Could you please provide more information to us on this regarding location where this new value set will be displayed.

[EricDavisX]

Hi - for Q1, no we do not need to create more cases, just executing what we have is enough for stand-alone, basically just need to run the regression test cases that exist.

for Q2 and Q3, I have updated the description to include that it will be on the bottom left of the page under the logs display. it was a screenshot in one of the linked issues, glad to pull it out here and elaborate. The log level set here is applied to all processes Agent runs if they are set up to send logs (which all of Agent, Endpoint, and Metricbeat, Filebeat are configurable as such.

[dikshachauhan-qasource]

Hi @EricDavisX

Thank you for resolving our queries.

We have created 06 testcases accordingly for Q2 and Q3 based on screenshot shared to us in updated description. Testcases links are as follows:
https://elastic.testrail.io/index.php?/cases/view/35153
https://elastic.testrail.io/index.php?/cases/view/35152
https://elastic.testrail.io/index.php?/cases/view/35154
https://elastic.testrail.io/index.php?/cases/view/35158
https://elastic.testrail.io/index.php?/cases/view/35159
https://elastic.testrail.io/index.php?/cases/view/35160

However, we have 01 query after viewing reference video at comment #81872 (comment)

We observed selected agent logging level filter option is getting displayed on Agent details page.

• So, could you please confirm if, only selected level logs will get displayed in logs section on Agents logs page, after setting has been changed to verify root cause[ in the event that the agent has an error]. OR how this filter will affect logs display.

[EricDavisX]

@jen-huang @nchaulet can confirm @dikshachauhan-qasource it is a good question. I had a different idea, but after playing with it I believe the the log-level selector works like this:

• with nothing selected, all log levels are shown (the picker shows the total # of levels)
• with one or more selected, only those specific log levels selected are shown

[nchaulet]

. I had a different idea, but after playing with it I believe the the log-level selector works like this:

Yes when nothing is selected the filter show the number of all the possible filter (in grey) and it show all the logs.
With something selected the filter show the number of applied filter (in rose) and only the specific log are shown

[dikshachauhan-qasource]

Hi @EricDavisX and @nchaulet

Thank you for resolving our queries.

We have created 03 testcases based on feedback. Testcases links are as follows:
https://elastic.testrail.io/index.php?/cases/view/35168
https://elastic.testrail.io/index.php?/cases/view/35169
https://elastic.testrail.io/index.php?/cases/view/35170

And also updated 02 testcases. Link:
https://elastic.testrail.io/index.php?/cases/view/35154
https://elastic.testrail.io/index.php?/cases/view/35118

Please let us now if we missed anything.

[dikshachauhan-qasource]

Hi @EricDavisX,

We tested this ticket on the latest 8.0 build and here are our observations:

Build Details:

Version: 8.0.0 snapshot

BUILD 38640
COMMIT 008c6a0e19a1c9e130152a6082c574057c841f89
Artifact:  https://artifacts-api.elastic.co/v1/search/8.0.0-SNAPSHOT/

Observations:
We created a test plan and executed 09 testcases here: Agent log level Test run

• While performing testing, we observed one of scenarios is failing for the debug filter, when we apply debug filter, we can still view the rows for 'info' logs in the activity logs list.
  We have opened defect for the same: #22986

Refer screenshot:

Please let us know if we missed anything.

[nchaulet]

Maybe it's a little confusing but the select below the log table is not used to filter logs by log level, but to set the log level on the agent, if you want to filter the log level you should use the log level field on top of the log level table.

[EricDavisX]

Thank you Nicolas. @dikshachauhan-qasource further, if I understand your question, the short discussion is this:

• the support on the Agent is just now merged for 8.0, so you can test again tomorrow tho not all fixes are in
• the Agent log level sector will set it's level of verbosity... anything less verbose will be displayed. So, setting it to 'debug' is the most verbose and all less-verbose log messages will still show. If you set it to 'Error' then only Error is displayed. But set it to 'warning' and both Warning and Error are displayed.
  ... as a deeper test / review: For Beats or Endpoint where the log levels do not fit this enumeration of the core 4 levels (error, warning, info, debug), I expect the beat or endpoint log level set should match fairly closely, and what we get is probably ok unless it is very egregious.

As an example of today's 8.0 deploy (tho it has code which we know is not fully complete yet on Agent side) I can see interesting nuances that show it is working and complex the feature is. Note that the Agent level combo box only shows the core-4 levels to set. But the 'filter' drop down will show what is in the data, and level 'notice' will generally fall between info and warning. : / And other log levels from other products / frameworks will exist:
emerg
fatal
alert
crit
error - option 1
warn - option 2
notice
info - option 3
debug - option 4
trace

The first 4 items in the list are all 'really bad' and would show always, but when you select option 3 level, you won't get debug (or lower) levels logged. if you select option 2 you won't get anything below option 2 in the logs. The filtering is always dependent on what is in the data already tho, so once a log exists (within a given time range selected) it will show regardless of current setting. And so Agent doesn't control what levels may come from the various integrations / beats, hence the discussion on the larger set of log levels even tho you can't select / set them all explicitly.

[dikshachauhan-qasource]

Hi @EricDavisX

Thank you for the clarification and deeper look to functionality.

We will re-validated same once code is available for feature as mentioned in comment .

Moreover, we have updated 03 testcases as per above information. Updated testcases links are as follows:
https://elastic.testrail.io/index.php?/cases/view/35132
https://elastic.testrail.io/index.php?/cases/view/35118
https://elastic.testrail.io/index.php?/cases/view/35093

Please let us know if anything else is required.

[dikshachauhan-qasource]

Hi @EricDavisX

We have executed 09 testcases under test plan Agent Logging Level test run

Build details are as follows:

BUILD 37399
COMMIT f3abc08ac648f8b302733c5c22a39048314a027c
Artifact link: https://staging.elastic.co/7.11.0-710164a0/downloads/beats/elastic-agent/elastic-agent-7.11.0-windows-x86_64.zip

Please let us know if anything is required from our side.

Thanks
QAS

[EricDavisX]

Thanks I'll review the feature testing and report back!