Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[TC-SC-5.2] Test case is failing due to TH sends AddGroup command #33

Closed
raju-apple opened this issue Oct 19, 2023 · 9 comments
Closed

[TC-SC-5.2] Test case is failing due to TH sends AddGroup command #33

raju-apple opened this issue Oct 19, 2023 · 9 comments
Labels
Awaiting Originator Feedback

Comments

@raju-apple
Copy link

From chip-certification-tool created by mideayanghui: CHIP-Specifications/chip-certification-tool#881

Describe the bug

[TC-SC-5.2] Test case is failing , just as the Test Harness's screenshot as below
1
2

Note
TC-SC-5.2 is passing while verifying manually.

Additional Info
TestHarness:
Version: v2.9-beta1
Sha: b0593a99

Test Harness UI execution log
TC-SC-5.2-1E1dQtKS_TH_log__2023_07_07_15_58_37.log
TC-SC-5.2-1E1dQtKS_TH_log_DUT_2023-07-07_15-52-49.log

Test Harness manual execution log
Uploading TC-SC-5.2-1E1dQtKS_TH_2023-07-04_15-13-06.log…
TC-SC-5.2-1E1dQtKS_DUT_2023-07-04_15-13-08.log

Steps to reproduce the behavior

No response

Expected behavior

No response

Log files

No response

PICS file

No response

Screenshots

No response

Environment

No response

Additional Information

No response
@raju-apple
Copy link
Author

DUT firmware is built from all-cluster-app demo (https://github.com/project-chip/connectedhomeip master banch commit: 7806dd19111513a61bb90d70a0e392dc89eeb454)
PICS comes from https://groups.csa-iot.org/wg/matter-csg/document/folder/2756

@raju-apple
Copy link
Author

Hi @mideayanghui, can you please upload the first 'Test Harness manual execution log' again? It seems like the file didn't upload correctly and I can't download it here.

@raju-apple
Copy link
Author

Hello @ccruzagralopes , here is the 'Test Harness manual execution log'
TC-SC-5.2-1E1dQtKS_TH_2023-07-04_15-13-06.log

@raju-apple
Copy link
Author

It seems like the test case is executing the command for the failing test step on endpoint 1, but the manual steps performed previously were all using endpoint 0.

INFO       | 2023-07-07 07:59:32.291157 | Executing Test Step: TH sends AddGroup command
CHIPTOOL   | 2023-07-07 07:59:32.292721 | �[37m		10. Running �[0m�[1mTH sends AddGroup command�[0m
CHIPTOOL   | 2023-07-07 07:59:32.353814 | �[31m✗�[0m �[37m90.04ms�[0m
CHIPTOOL   | 2023-07-07 07:59:32.355396 | �[37m		    [PostProcessCheckType.IM_STATUS check] The test expects no error and no error occurred.�[0m
CHIPTOOL   | 2023-07-07 07:59:32.356830 | �[37m		  �[31m✗�[0m [PostProcessCheckType.RESPONSE_VALIDATION check] The test expectation "Status == 0" is false�[0m
CHIPTOOL   | 2023-07-07 07:59:32.358045 | �[37m		    [PostProcessCheckType.RESPONSE_VALIDATION check] The test expectation "GroupID == 259" is true�[0m
CHIPTOOL   | 2023-07-07 07:59:32.359192 | 		    [TOO] Command: groups add-group 259 Test Group 0103 0x381b4ac6a9bcadaf 1 
CHIPTOOL   | 2023-07-07 07:59:32.360467 | 		    [TOO] Sending command to node 0x381b4ac6a9bcadaf
CHIPTOOL   | 2023-07-07 07:59:32.369248 | 		    [CSM] FindOrEstablishSession: PeerId = [1:381B4AC6A9BCADAF]
CHIPTOOL   | 2023-07-07 07:59:32.370357 | 		    [CSM] FindOrEstablishSession: No existing OperationalSessionSetup instance found
CHIPTOOL   | 2023-07-07 07:59:32.371347 | 		    [DIS] Found an existing secure session to [1:381B4AC6A9BCADAF]!
CHIPTOOL   | 2023-07-07 07:59:32.372396 | 		    [DIS] OperationalSessionSetup[1:381B4AC6A9BCADAF]: State change 1 --> 5
CHIPTOOL   | 2023-07-07 07:59:32.383305 | [TOO] Sending cluster (0x00000004) command (0x00000000) on endpoint 1

I don't know whether the manual steps should be performed on endpoint 1 or the test case should use endpoint 0.

@manjunath-grl or @kvikrambhat, can you please take a look at this?

@raju-apple
Copy link
Author

@mideayanghui As from the logs for Add group step response is 126 (0x7E ---> UNSUPPORTED_ACCESS) 

CHIPTOOL   | 2023-07-07 07:59:33.443594 | [1mExpected: [
CHIPTOOL   | 2023-07-07 07:59:33.444879 | 		                {
CHIPTOOL   | 2023-07-07 07:59:33.445871 | 		                  "values": [
CHIPTOOL   | 2023-07-07 07:59:33.446815 | 		                    {
CHIPTOOL   | 2023-07-07 07:59:33.447791 | 		                      "name": "Status",
CHIPTOOL   | 2023-07-07 07:59:33.450731 | 		                      "value": 0
CHIPTOOL   | 2023-07-07 07:59:33.451873 | 		                    },
CHIPTOOL   | 2023-07-07 07:59:33.452872 | 		                    {
CHIPTOOL   | 2023-07-07 07:59:33.453834 | 		                      "name": "GroupID",
CHIPTOOL   | 2023-07-07 07:59:33.454916 | 		                      "value": 259
CHIPTOOL   | 2023-07-07 07:59:33.455957 | 		                    }
CHIPTOOL   | 2023-07-07 07:59:33.456914 | 		                  ]
CHIPTOOL   | 2023-07-07 07:59:33.457844 | 		                }
CHIPTOOL   | 2023-07-07 07:59:33.458776 | 		              ]
CHIPTOOL   | 2023-07-07 07:59:33.459735 | 		    [1mReceived: [
CHIPTOOL   | 2023-07-07 07:59:33.460879 | 		                 {
CHIPTOOL   | 2023-07-07 07:59:33.461864 | 		                   "cluster": "Groups",
CHIPTOOL   | 2023-07-07 07:59:33.462816 | 		                   "command": "AddGroupResponse",
CHIPTOOL   | 2023-07-07 07:59:33.463810 | 		                   "endpoint": 1,
CHIPTOOL   | 2023-07-07 07:59:33.464797 | 		                   "value": {
CHIPTOOL   | 2023-07-07 07:59:33.465735 | 		                     "GroupID": 259,
CHIPTOOL   | 2023-07-07 07:59:33.466664 | 		                     "Status": 126
CHIPTOOL   | 2023-07-07 07:59:33.467624 | 		                   }
CHIPTOOL   | 2023-07-07 07:59:33.468609 | 		                 }
CHIPTOOL   | 2023-07-07 07:59:33.469543 | 		               ]
CHIPTOOL   | 2023-07-07 07:59:33.470472 | 		***** Test Failure : The test expectation "Status == 0" is false

For step 1 can you please try once with the below command and send me the logs.
./chip-tool accesscontrol write acl '[{"fabricIndex": 1, "privilege": 5, "authMode": 2, "subjects": [112233], "targets": null },{"fabricIndex": 1, "privilege": 4, "authMode": 3, "subjects": [0x0103], "targets": null }]' 1 0

@raju-apple
Copy link
Author

@manjunath-grl Here's the log, thanks.
chip-tool_2023-07-14_13-34-17.log
dut_2023-07-14_13-33-02.log

@raju-apple
Copy link
Author

@mideayanghui TC-SC-5.1 test going to be fully automated in PR: project-chip/connectedhomeip#27960.

@raju-apple
Copy link
Author

@manjunath-grl @ccruzagralopes Thanks for your help. So how can I update this automated test script into my test harness?based on Version: v2.9-beta1

@raju-apple raju-apple removed the v1.2 label Oct 31, 2023
@fabiowmm
Copy link
Contributor

@mideayanghui would you verify and close this issue?

rquidute added a commit that referenced this issue Sep 23, 2024
@ccruzagralopes @mikaelhm
@antonio-amjr @rquidute @raul-marquez-csa @hiltonlima
Merge 2.10-beta1.1+spring2024 release into main (#34)
30f7d99 
* Fix prblem with sdk_tests collection using hardcoded paths

* Enable github workflows for main branch (#17)

* Enable github workflows for main branch

* Ignore Dockerfile

* mypy: ignore python_testing in sdk_collection

* minor linting issues fixed

* ignore .devcontainer in spell checker

* move .env file back in root

* Update reference to fetch_sdk_tests_and_runner.sh

* Use `pull_request_target` event to support forks

* Update yaml_test_folder.py (#21)

* Update yaml_test_folder.py

* Update test_sdk_yaml_collection.py

* Update unittests.yml

Don't run old code for unit tests

* [Fix] remove TestRunConfig usage from TH Backend (#581) (#6)

* Add the repeat endpoint, deprecate the test_run_config endpoints and

force the test run execution creation to use selected tests, and not test_run_config_id.

* Deprecating unit tests related with
test_run_config, creating new endpoint with required selected_tests
parameter and ignoring the test_run_config reference, renaming the
repeat endpoint method and adding comments.

* Updating select_tests logic, moving to a utils
file, and adding unit test for the repeat endpoint

* Updating comments and adding missing attribution to
None for test_run_config_id

* Adding missing TODO comment.

* Fixing selected_tests logic

* Updating the 'repeat' unit test to use more
suites to cover more cases.

* Removing "/new" endpoint, updating logic of the
default test run execution endpoint and updating unit tests, "repeat"
endpoint and utils

* Adding the new DB column "collection_id" to the
TestSuiteExecution table. A DB revision file was created and the code
was refactored to use this new info.

* Removing unnecessary import

* Set default log level to info, to eliminate chatter from python runner (#23)

* Update SDK SHA (#25)

* Added support to reset db for test (#28)

* Missing submit button (#30)

* Updates MessageTypeEnum to include options and message types

* Update app/user_prompt_support/user_prompt_manager.py

Co-authored-by: antonio-amjr <116589331+antonio-amjr@users.noreply.github.com>

* Update app/constants/websockets_constants.py

Co-authored-by: hiltonlima <116589806+hiltonlima@users.noreply.github.com>

* Update app/user_prompt_support/user_prompt_manager.py

---------

Co-authored-by: Romulo Quidute Filho <116586593+rquidute@users.noreply.github.com>
Co-authored-by: antonio-amjr <116589331+antonio-amjr@users.noreply.github.com>
Co-authored-by: hiltonlima <116589806+hiltonlima@users.noreply.github.com>
Co-authored-by: Carolina Lopes <116589288+ccruzagralopes@users.noreply.github.com>

* Update version

* Prompt request refactor (#31)

* Refactor to use only 1 enum

* Removed unsed import

* Fix isort

* Adds a new Message prompt request

---------

Co-authored-by: raul-marquez-csa <rmarquez@csa-iot.org>

* update backend version in FE (#32)

* update backend version in FE

* Update .version_information

* Update .version_information

* change ssh to https (#33)

* Undo version information update

---------

Co-authored-by: Mikael H. Moeller <mikaelhm@apple.com>
Co-authored-by: Mikael Møller <mikaelhm@hey.com>
Co-authored-by: antonio-amjr <116589331+antonio-amjr@users.noreply.github.com>
Co-authored-by: Romulo Quidute Filho <116586593+rquidute@users.noreply.github.com>
Co-authored-by: Raul Marquez <130402456+raul-marquez-csa@users.noreply.github.com>
Co-authored-by: hiltonlima <116589806+hiltonlima@users.noreply.github.com>
Co-authored-by: raul-marquez-csa <rmarquez@csa-iot.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Awaiting Originator Feedback
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@raju-apple @fabiowmm