YANG Push Test Automation Tool

Overview

This tool automates the testing of YANG Push and NETCONF server capabilities. It is designed to:

• Run a suite of YANG/NETCONF-related tests against network devices.
• Collect, log, and save results for further analysis.
• Support vendor-specific and generic test cases.
• Provide extensibility for new tests and vendor modules.
• Support for dynamic test generation from XML files and test sequences for grouped/ordered execution.

---

Install

Verified and working on Red Hat Enterprise Linux release 8.8 (Ootpa)

 git clone https://github.com/network-analytics/yp_test.git
 python3.9 -m venv yp_test/
 cd yp_test/
 source bin/activate
 pip install -r

---

Example Usage

python3 main.py --config config/yp_test_config.yml --vendor 6wind --all --help
usage: yp_test [-h] --vendor {6wind,cisco_iosxr,huawei_ma5800t,huawei_vrp_ne} -c CONFIG [-u RECEIVER_USERNAME] [-i RECEIVER_INSTANCE_NAME]
               [--get_schema_to_file GET_SCHEMA_TO_FILE] [--backup_subscriptions BACKUP_SUBSCRIPTIONS]
               [--restore_subscriptions RESTORE_SUBSCRIPTIONS] [--all] [--override_result_dir OVERRIDE_RESULT_DIR]
               [--override_timer OVERRIDE_TIMER] [-v] [-vv] [--tests TESTS [TESTS ...]] [--list-tests]

YANG Push Test Suite (modular, multivendor version)

optional arguments:
  -h, --help            show this help message and exit
  
  --vendor {6wind,cisco_iosxr,huawei_ma5800t,huawei_vrp_ne}
                        Vendor name (choices: 6wind, cisco_iosxr, huawei_ma5800t, huawei_vrp_ne)
  
  -c CONFIG, --config CONFIG
                        YAML config file
  
  -u RECEIVER_USERNAME, --receiver-username RECEIVER_USERNAME
                        user used for SSH connection (overrides config file)
  
  -i RECEIVER_INSTANCE_NAME, --receiver-instance-name RECEIVER_INSTANCE_NAME
                        YANG-Push receiver-instance-name (overrides config file)
  
  --get_schema_to_file GET_SCHEMA_TO_FILE
                        get YANG schema to file
  
  --backup_subscriptions BACKUP_SUBSCRIPTIONS
                        backup YANG Push configuration to file
  
  --restore_subscriptions RESTORE_SUBSCRIPTIONS
                        restore YANG Push configuration from file
  
  --all
  
  --override_result_dir OVERRIDE_RESULT_DIR
  
  --override_timer OVERRIDE_TIMER
  
  -v, --verbose
  
  -vv, --very_verbose

  --tests TESTS [TESTS ...]
                        Select one or more tests or test sequences to run in order. Use --list-tests to see the choices.
 
  --list-tests          List all available tests and sequences, then exit.

Current valid vendor_name are:

• 6wind
• cisco_iosxr
• huawei_vrp_ne
• huawei_ma5800t

• Additional arguments can control verbosity, result directories, and test/sequence selection.

---

Directory Structure

yp_test/
├── main.py
├── config/
│   ├── yp_test_config.yml
├── tests/
│   ├── yp_tests.py
│   ├── test_utils.py
├── utils/
│   ├── netconf_utils.py
│   ├── pyang_tree.py
│   ├── info_packet.py
│   └── ...
├── vendors/
│   ├── vendor_base.py
│   └── vendor_<vendor>.py
└── ...

---

Key Components

1. main.py

• Entry point for running tests.
• Handles argument parsing, device connection, vendor loading, and test execution.
• Collects and displays test results in a tabular format.
• Supports both individual tests and test sequences.

2. tests/yp_tests.py

• Contains all test case definitions as functions.
• Supports dynamic test generation for XML-driven NETCONF/YANG tests.
• Supports test sequences (ordered groups of tests) via the YPTestSequence class.
• Tests are dynamically registered as instances of YPTest based on their function names (e.g., yangpush_, netconf_).
• Each test receives a unique step number and shared context via **kwargs.

3. tests/test_utils.py

• Defines the YPTest class for test registration, execution, and step tracking.
• Defines the YPTestSequence class for grouping and registering test sequences.
• Provides utility functions for waiting, saving results, and logging.

4. utils/netconf_utils.py

• Implements NETCONF operations such as nc_config, nc_rpc, nc_get_schema, nc_get_subscriptions, and deletion helpers.
• Handles XML parsing and NETCONF message construction.

5. utils/pyang_tree.py

• Functions for parsing and extracting YANG modules from NETCONF replies.
• Includes robust handling for YANG module extraction from XML payloads.
• Can save YANG source and tree output to the results directory.

6. vendors/vendor_base.py

• Base class for vendor-specific logic.
• Handles XML file location, value overrides, and device-specific configuration.

7. utils/tcpdump_over_ssh and utils/kcat_over_ssh

• Allow remote capture of received raw ip packets and received udpnotif messages after data collection.
• supports vrf and simple authentication.

---

Test Types

1. Static Test Functions

• Manually written Python functions, e.g.:

  def yangpush_delete_subsciptions(self, **kwargs):
      ...

2. Dynamically Generated Tests

• Created from lists of XML files using a factory function.
• Example for capabilities tests:

  capabilities_tests = [
      'get-system-capabilities.xml',
      'get-discover-yang.xml',
      ...
  ]
  def make_capabilities_tests(xml_file):
      ...
  for xml_file in capabilities_tests:
      make_capabilities_tests(xml_file)

• Example for MVP1 YANG schema tests:

  mvp1_yang_schema_tests = [
      'get-schema-notification-transport-capabilities.xml',
      ...
  ]
  def make_mvp1_yang_schema_tests(xml_file):
      ...
  for xml_file in mvp1_yang_schema_tests:
      make_mvp1_yang_schema_tests(xml_file)

3. Test Sequences

• Ordered groups of tests, registered via YPTestSequence.
• Example:

  MVP1 = YPTestSequence(
      "MVP1",
      [
          "yangpush_delete_subsciptions",
          "yangpush_delete_receivers",
          ...
      ]
  )

• Sequences are auto-registered and can be selected by name or letter in the CLI.
• Test sequences allow you to run a logical group of tests in a specific order, useful for standards coverage or scenario validation.

---

How It Works

1. Test Registration

   • All functions in yp_tests.py matching the naming pattern are automatically registered as YPTest instances.
   • Dynamically generated tests are added to the module namespace and registered the same way.
   • Test sequences are registered via YPTestSequence.

2. Test Execution

   • main.py builds a dictionary of test names to YPTest instances and a dictionary of sequence names to YPTestSequence instances.
   • Tests and sequences can be selected via CLI arguments or interactive prompt.
   • Sequences expand to their constituent test names for execution.

3. Result Collection

   • Each test’s status, duration, and error (if any) are collected.
   • Results are displayed in a table and can be logged or saved.

4. Vendor Support

   • Vendor-specific logic is encapsulated in modules under vendors/.
   • The correct vendor class is loaded dynamically based on configuration.
   • Generic vendor agnostic YANG-Push subscriptions are placed in xml files in the vendors folder. If a vendor has deviation e.g. not supporting transport leaf a adapted xml file is placed in the vendor specific folder e.g. Cisco

5. Extensibility

   • To add a new test, simply define a new function in yp_tests.py with the appropriate prefix, or add an XML file to a dynamic test list.
   • To add a new test sequence, instantiate a new YPTestSequence with a list of test names.
   • To support a new vendor, add a new class in vendors/ inheriting from VendorBase.

---

Configuration File (yp_test_config.yml)

The tool uses a YAML configuration file (yp_test_config.yml) to define device, vendor, and test environment settings. This file is placed in the config folder of the project root.

Example yp_test_config.yml

DUTS:
  6wind:
    hostname: ipf-zbl1843-r-daisy-58
    ip: 203.0.113.58
    username: daisy
    password: daisy123
  cisco:
    hostname: ipf-zbl1327-r-daisy-91
    ip: 203.0.113.91
    username: ......
    password: ......
  huawei:
    hostname: ipf-zbl1243-r-daisy-21
    ip: 203.0.113.21
    username: ......
    password: ......
RECEIVER:
  hostname: ietf-col-left01
  ip: 138.187.58.24
  udp_notif_port: 10003
  instance_name: DAISYTEST # Use any name no coliding with existant config. Huawei doesn't support _ in name
  mgmt_ip: 10.212.242.2
  interface: ens224
  username: your_user_for_remote_sesion_to_collector
INPUT:
  xmls: ./xmls/
OUTPUT_DIR: ./results/


# Additional custom settings as needed

---

Flexible connectivity settings

Your lab environment might be segemented and your test host constrained to use different VRFs

In that case you can override the default VRF being used for netconf connections in config/ncclient_vrf_config.cfg as follows:

Host 10.190.64.79 
    ProxyCommand nc %h %p
Host *
    ProxyCommand ip vrf exec vrf-ietf nc %h %p

Adding a New Test

1. Static:
   Open tests/yp_tests.py and define a new function, e.g.:

   def yangpush_new_feature(self, **kwargs):
       """Test description here."""
       # Test logic here

2. Dynamic:
   Add an XML file to the relevant list and ensure the factory function is called for it.

---

Adding a New Test Sequence

1. Open tests/yp_tests.py.
2. Define a new sequence:

   my_sequence = YPTestSequence(
       "my_sequence",
       [
           "yangpush_delete_subsciptions",
           "netconf_get_yang_library",
           ...
       ]
   )

---

Adding a New Vendor

1. Create a new file in vendors/, e.g., vendor_acme.py.
2. Define a class inheriting from VendorBase:

   from .vendor_base import VendorBase
   class VendorAcme(VendorBase):
       # Override methods as needed

3. The tool will load this class when --vendor acme is specified.

---

Error Handling

• Exceptions in tests are logged and reported in the results table.
• Helper functions should re-raise exceptions after logging to ensure errors propagate to the test runner.

---

Logging and Results

• Logs are written to the console and/or files as configured.
• Test results are displayed in a tabular format with status, duration, and error messages.

Example:

 --------------------------------------------------------------------------------
=== Test Results:
+-------------------------------------------------------------+----------+------------+---------------------------------------------+
| Test                                                        | Status   | Duration   | Error                                       |
+=============================================================+==========+============+=============================================+
| netconf_get_sw_version                                      | FAIL     | 0.00s      | get-version.xml not found for vendor 6wi... |
+-------------------------------------------------------------+----------+------------+---------------------------------------------+
| yangpush_delete_subsciptions                                | PASS     | 0.64s      |                                             |
+-------------------------------------------------------------+----------+------------+---------------------------------------------+
| yangpush_delete_receivers                                   | PASS     | 0.44s      |                                             |
+-------------------------------------------------------------+----------+------------+---------------------------------------------+
| netconf_get_ietf_notification_capabilities                  | FAIL     | 0.21s      | Received empty response for get-ietf-not... |
+-------------------------------------------------------------+----------+------------+---------------------------------------------+
| netconf_get_ietf_system_capabilities                        | PASS     | 0.22s      |                                             |
+-------------------------------------------------------------+----------+------------+---------------------------------------------+
| netconf_get_ietf_yang_library                               | PASS     | 0.28s      |                                             |
+-------------------------------------------------------------+----------+------------+---------------------------------------------+
| netconf_get_ietf_yang_library_augmentedby                   | PASS     | 0.19s      |                                             |
+-------------------------------------------------------------+----------+------------+---------------------------------------------+
| netconf_get_schema_list                                     | PASS     | 0.30s      |                                             |
+-------------------------------------------------------------+----------+------------+---------------------------------------------+
| netconf_get_and_validate_yang_library                       | PASS     | 0.67s      |                                             |
+-------------------------------------------------------------+----------+------------+---------------------------------------------+
| netconf_get_schema_ietf_notification_transport_capabilities | FAIL     | 0.06s      | The requested module was not found....      |
+-------------------------------------------------------------+----------+------------+---------------------------------------------+
| netconf_get_schema_ietf_udp_notif_transport                 | PASS     | 1.45s      |                                             |
+-------------------------------------------------------------+----------+------------+---------------------------------------------+
| netconf_get_schema_ietf_yang_library_augmentedby            | PASS     | 1.16s      |                                             |
+-------------------------------------------------------------+----------+------------+---------------------------------------------+
| netconf_get_schema_ietf_yang_push                           | PASS     | 1.06s      |                                             |
+-------------------------------------------------------------+----------+------------+---------------------------------------------+
| netconf_get_schema_ietf_yang_push_revision                  | PASS     | 0.97s      |                                             |
+-------------------------------------------------------------+----------+------------+---------------------------------------------+
| netconf_get_schema_ietf_yp_notification                     | PASS     | 1.03s      |                                             |
+-------------------------------------------------------------+----------+------------+---------------------------------------------+
| netconf_get_schema_ietf_yp_observation                      | PASS     | 1.16s      |                                             |
+-------------------------------------------------------------+----------+------------+---------------------------------------------+
| netconf_get_schema_ietf_yp_transport_capabilities           | PASS     | 0.98s      |                                             |
+-------------------------------------------------------------+----------+------------+---------------------------------------------+
| netconf_get_schema_ietf_system_capabilities_yang            | PASS     | 0.84s      |                                             |
+-------------------------------------------------------------+----------+------------+---------------------------------------------+
| yangpush_add_receiver                                       | PASS     | 10.62s     |                                             |
+-------------------------------------------------------------+----------+------------+---------------------------------------------+
| yangpush_ietf_interface                                     | PASS     | 70.35s     |                                             |
+-------------------------------------------------------------+----------+------------+---------------------------------------------+
| yangpush_vendor_interface                                   | PASS     | 70.56s     |                                             |
+-------------------------------------------------------------+----------+------------+---------------------------------------------+
| netconf_get_schema_ietf_interfaces_yang                     | PASS     | 0.69s      |                                             |
+-------------------------------------------------------------+----------+------------+---------------------------------------------+
| netconf_yang_schema_tree                                    | PASS     | 3.21s      |                                             |
+-------------------------------------------------------------+----------+------------+---------------------------------------------+
| yangpush_get_subscriptions                                  | PASS     | 0.20s      |                                             |
+-------------------------------------------------------------+----------+------------+---------------------------------------------+
| netconf_delete_all_subscriptions_and_receivers              | PASS     | 1.08s      |                                             |
+-------------------------------------------------------------+----------+------------+---------------------------------------------+
| yangpush_post_test_cleanup                                  | PASS     | 0.43s      |                                             |
+-------------------------------------------------------------+----------+------------+---------------------------------------------+
Size of file is : 928251 bytes
Size of file (./results/ipf-zbl1843-r-daisy-58_6wind_20250716_142001/kafka_output.json) is : 9385 bytes
Selected test sequences: []
Results saved in ./results/ipf-zbl1843-r-daisy-58_6wind_20250716_142001

Results directory tree:
ipf-zbl1843-r-daisy-58_6wind_20250716_142001/
    stdout.log (62960 bytes)
    udp_notif_capture.pcap (928251 bytes)
    kafka_output.json (9385 bytes)
    yp_config_backup.xml (2711 bytes)
    yp_test.log (1531 bytes)
    get-ietf-system-capabilities.txt (5004 bytes)
    get-ietf-yang-library.txt (53698 bytes)
    get-ietf-yang-library-augmentedby.txt (26190 bytes)
    get-schema-list.txt (82983 bytes)
    ietf-yang-library.yang (16156 bytes)
    ietf-datastores.yang (2594 bytes)
    ietf-yang-library.tree (2145 bytes)
    ietf-udp-notif-transport.yang (5464 bytes)
    ietf-udp-notif-transport.tree (0 bytes)
    ietf-yang-library-augmentedby.yang (3367 bytes)
    ietf-yang-library-augmentedby.tree (265 bytes)
    ietf-yang-push.yang (26957 bytes)
    ietf-yang-push.tree (6248 bytes)
    ietf-yang-push-revision.yang (8758 bytes)
    ietf-yang-push-revision.tree (1563 bytes)
    ietf-yp-notification.yang (5473 bytes)
    ietf-yp-notification.tree (347 bytes)
    ietf-yp-observation.yang (4180 bytes)
    ietf-yp-observation.tree (260 bytes)
    ietf-yp-transport-capabilities.yang (4345 bytes)
    ietf-yp-transport-capabilities.tree (0 bytes)
    ietf-system-capabilities.yang (6197 bytes)
    ietf-system-capabilities.tree (366 bytes)
    ietf-interfaces.yang (39075 bytes)
    ietf-interfaces.tree (2915 bytes)
    vrouter-interface.yang (24248 bytes)
    vrouter.yang (6097 bytes)
    extra-conditions.yang (2126 bytes)
    vrouter-api.yang (4948 bytes)
    vrouter-extensions.yang (11870 bytes)
    vrouter-commands.yang (7249 bytes)
    vrouter-inet-types.yang (14170 bytes)
    vrouter-types.yang (12882 bytes)
    vrouter-if-types.yang (15411 bytes)
    vrouter-ip.yang (39457 bytes)
    vrouter-system.yang (37508 bytes)
    iana-timezones.yang (37969 bytes)
    vrouter-fast-path.yang (84711 bytes)
    vrouter-qos.yang (62680 bytes)
    vrouter-interface.tree (1364 bytes)

---

Dependencies

• netgauze-pcap-decoder (experimental see ./utils/ )
• yanglint
• pyang
• openssh
• sshpass
• Python 3.7+
• colorama==0.4.6
• lxml==5.4.0
• ncclient==0.6.19
• PyYAML==6.0.2
• tabulate==0.9.0
• Other standard libraries

On collector host:

• tcpdump
• kcat

---

Support & Contribution

• For issues, open a ticket in the repository.
• Contributions are welcome: fork, branch, and submit a pull request.

---

This tool is designed for extensible YANG/NETCONF test automation.