Update documentation for nautobot and new backend system

README.md

@@ -12,6 +12,7 @@ The main use cases for the network importer:
 - [Configuration file](docs/configuration.md)
 - [Supported Features and Architecture](docs/architecture.md)
 - [Extensibility](docs/extensibility.md)
+- [Upgrade procedure](docs/upgrade.md)
 
 ## Questions
 

docs/architecture.md

@@ -1,12 +1,12 @@
 
 # Architecture
 
-Internally the network-importer is leveraging the [diffsync](https://github.com/networktocode/diffsync) library to compare the state of the network and the state of the Source of Truth. The diffsync library is designed to compare the state of 2 **adapters** based on a shared data-models.
+Internally the network-importer is leveraging the [diffsync](https://github.com/networktocode/diffsync) library to compare the state of the network and the state of the Source of Truth. [The diffsync library is designed to compare the state of 2 **adapters** based on a shared data-models](https://blog.networktocode.com/post/intro-to-diffing-and-syncing-data-with-diffsync/).
 
 To communicate to the network devices, the network importer is leveraging [Nornir](https://github.com/nornir-automation/nornir) and incorporate a concept of **drivers** to easily add support for more platforms if needed.
 By default the Network Importer supports 3 main actions to execute against the network devices:
 - `get_config`: Retrieve the running configuration and store it in the `config_directory` folder.
-- `get_neighbors`: Retrieve the list of all neighbors, eiher from LLDP or CDP (based on the configuration)
+- `get_neighbors`: Retrieve the list of all neighbors, either from LLDP or CDP (based on the configuration)
 - `get_vlans`: Retrieve the list of vlans present on the device.
 
 ## Internal Datamodel
@@ -22,13 +22,35 @@ The internal/shared datamodel is defined in the [network_importer/models.py](../
 
 > It's possible to extend the default models and add your own, please check the [extensibility section](extensibility.md) of the documentation
 
-## Adapters
+## Backend, Adapters & Inventory
+
+To operate the Network Importer needs 1 inventory and 2 adapters:
+- An inventory to get the list of devices to analyze and get the minimum information to connect to them (platform, address, cred
+entials ..)
+- One adapter to read the information from the network and one to read/write information to the Source of Truth backend.
+
+Since the inventory is usually leveraging the SOT, the SOT adapter and the inventory are packaged into a **backend**. Both Nautobot and Netbox are supported as backend systems. When a specific backend is selected it will update both the SOT adapter and the inventory.
+With or without leveraging the default backends, it's possible to provide your own adapter or extend one of the default adapters.
+
+> The base adapter for Network Importer is defined in [network_importer/adapters/base.py](../network_importer/adapters/base.py). The main difference with a standard diffsync adapter is that a Network Importer adapter needs to accept a NetworkImporter inventory (based on a Nornir inventory) as parameters at init time (nornir).
 
-To operate the Network Importer needs 2 adapters. 
+### Nautobot API Adapter
 
-An adapter to read the information from the network and one to read/write information to Netbox via its Rest API are provided by default but it's possible to provide your own adapter or extend either or both default adapters.
+The Nautobot API adapter is designed to read the status of a Nautobot server over its Rest API and update Nautobot based on the status of the network.
 
-> The base adapter for Network Importer is defined in [network_importer/adapters/base.py](../network_importer/adapters/base.py). The main difference with a standard diffsync adapter is that a Network Importer adapter needs to accept a nornir inventory as parameters at init time (nornir).
+The table below present the capabilities in term of : Read, Create, Update and Delete supported for each model by the netbox_api adapter.
+
+| Model              | Inherit from | Create | Read   | Update | Delete |
+|--------------------|--------------|--------|--------|--------|--------|
+| NautobotSite       | Site         | No     | Yes    | No     | No     | 
+| NautobotDevice     | Device       | No     | Yes    | No     | No     | 
+| NautobotInterface  | Interface    | Yes    | Yes    | Yes    | Yes    | 
+| NautobotIPAddress  | IPAddress    | Yes    | Yes    | Yes    | Yes    | 
+| NautobotPrefix     | Prefix       | Yes    | Yes    | Yes    | No     | 
+| NautobotVlan       | Vlan         | Yes    | Yes    | Yes    | No     | 
+| NautobotCable      | Cable        | Yes    | Yes    | No     | No     | 
+
+> It's possible to extend the default models and add your own, please check the [extensibility section](extensibility.md) of the documentation
 
 ### NetBox API Adapter
 
@@ -54,7 +76,7 @@ The Network Importer Adapter is designed to read the status of the network prima
 
 ## Drivers
 
-The communicate with the network devices, the network importer is leveraging Nornir and support some drivers per platform to easily support more device type.
+The communicate with the network devices, the network-importer is leveraging Nornir and support some drivers per platform to easily support more device type.
 
 Each driver, should support each of the following actions: 
 - `get_config`: Retrieve the running configuration and store it in the `config_directory` folder.
@@ -72,7 +94,7 @@ By default, 4 drivers are available `default`, `cisco_default`, `juniper_junos`
 | juniper_junos   | juniper_junos  |
 | arista_eos      | arista_eos     |
 
-> The name of the platform must match the name of the slug platform defined in NetBox for a given device
+> The name of the platform must match the name of the slug platform defined in the inventory for a given device
 
 ### Drivers available by default
 

docs/backend/nautobot.md

@@ -0,0 +1,39 @@
+
+# Nautobot Backend
+
+The Nautobot backend is composed of an inventory and a SOT adapter designed to work with Nautobot 1.0+.
+Both the inventory and the SOT adapter can accept some specific settings as described below.
+
+## Inventory
+
+### Available Settings
+
+```toml
+[inventory.settings]
+
+address = "http://localhost:8080"                   # Alternative Env Variable : NAUTOBOT_ADDRESS
+token = "113954578a441fbe487e359805cd2cb6e9c7d317"  # Alternative Env Variable : NAUTOBOT_TOKEN
+verify_ssl = true                                   # Alternative Env Variable : NAUTOBOT_VERIFY_SSL
+
+
+# The default method is to use the primary IP defined in Nautobot.
+# As an alternative it's possible to use the name of the device and provide your own FQDN.
+use_primary_ip = false (default: true)
+fqdn = "mydomain.com"
+
+# Optional filter to limit the scope of the inventory, takes a comma separated string of key value pair"
+filter = "site=XXX,site=YYY,status=active"    # Alternative Env Variable : INVENTORY_FILTER
+```
+
+## SOT Adapter
+
+### Available Settings
+```toml
+[adapters.sot_settings]
+# Settings for applying diffsync flags to diffsync model objects, in order to alter 
+# the underlying sync behaviour. The model_flag is applied to any objects that have a 
+# tag assigned within model_flag_tags. Further details on model_flags can be found 
+# at: https://github.com/networktocode/diffsync/blob/269df51ce248beaef17d72374e96d19e6df95a13/diffsync/enum.py
+model_flag_tags = ["your_tag"]
+model_flag = 1 # flag enum int() representation
+```

docs/backend/netbox.md

@@ -0,0 +1,39 @@
+
+# Netbox Backend
+
+The NetBox backend is composed of an inventory and a SOT adapter designed to work with NetBox 2.8+.
+Both the inventory and the SOT adapter can accept some specific settings as described below.
+
+## Inventory
+
+### Available Settings
+
+```toml
+[inventory.settings]
+
+address = "http://localhost:8080"                   # Alternative Env Variable : NETBOX_ADDRESS
+token = "113954578a441fbe487e359805cd2cb6e9c7d317"  # Alternative Env Variable : NETBOX_TOKEN
+verify_ssl = true                                   # Alternative Env Variable : NETBOX_VERIFY_SSL
+
+
+# The default method is to use the primary IP defined in NetBox.
+# As an alternative it's possible to use the name of the device and provide your own FQDN.
+use_primary_ip = false (default: true)
+fqdn = "mydomain.com"
+
+# Optional filter to limit the scope of the inventory, takes a comma separated string of key value pair"
+filter = "site=XXX,site=YYY,status=active"    # Alternative Env Variable : INVENTORY_FILTER
+```
+
+## SOT Adapter
+
+### Available Settings
+```toml
+[adapters.sot_settings]
+# Settings for applying diffsync flags to diffsync model objects, in order to alter 
+# the underlying sync behaviour. The model_flag is applied to any objects that have a 
+# tag assigned within model_flag_tags. Further details on model_flags can be found 
+# at: https://github.com/networktocode/diffsync/blob/269df51ce248beaef17d72374e96d19e6df95a13/diffsync/enum.py
+model_flag_tags = ["your_tag"]
+model_flag = 1 # flag enum int() representation
+```

docs/configuration.md

@@ -1,10 +1,10 @@
 # Configuration File
 
-By default the network importer will try to load a configuration file name `network_importer.toml`, this configuration file is not mandatory as long as the required information to connect to NetBox, Batfish and/or the network devices are provided via environment variables.
+By default the network importer will try to load a configuration file name `network_importer.toml`, this configuration file is not mandatory as long as the required information to connect to the SOT, Batfish and/or the network devices are provided via environment variables.
 
 It's possible to specify which configuration file should be used in cli with the option `--config`.
 
-The configuration file is organized in 8 sections detailed below. 
+The configuration file is organized in 7 sections detailed below. 
 
 ## Main Section
 
@@ -32,39 +32,35 @@ import_vlans = "config"         # Valid options are ["cli", "config", "no", true
 import_cabling = "lldp"         # Valid options are ["lldp", "cdp", "config", "no", true, false]
 excluded_platforms_cabling = ["cisco_asa"]
 
-# Number of Nornir tasks to execute at the same tim
-nbr_workers= 25
+# Number of Nornir tasks to execute at the same time
+nbr_workers = 25
 
 # Directory where the configuration can be find, organized in Batfish format
 configs_directory = "configs"
+
+# Valid Backend
+# Only Netbox and Nautobot backend are included by default, if you want to use another backend
+# you must leave backend empty and define inventory.inventory_class and adapters.sot_class manually.
+backend = "nautobot"
 ```
 
-# NetBox Section
+# Inventory Section
 
-The `[netbox]` section regroup all parameters to connect to NetBox and it also include an optional list of supported platforms.
+The `[inventory]` section regroup all parameters related to the inventory and also includes an optional list of supported platforms. network-importer supports multiple inventories, you can define your own inventory by defining `inventory_class` and each inventory can define its own list of settings which will need to be configurated under `[inventory.settings]`. The settings for the [nautobot inventory](backend/nautobot.md) and for the [netbox inventory](backend/netbox.md) are available in their respective documentation.
 
 ```toml
-[netbox]
-address = "http://localhost:8080"                   # Alternative Env Variable : NETBOX_ADDRESS
-token = "113954578a441fbe487e359805cd2cb6e9c7d317"  # Alternative Env Variable : NETBOX_TOKEN
-verify_ssl = true                                   # Alternative Env Variable : NETBOX_VERIFY_SSL
+[inventory]
 
 # Define a list of supported platform, 
 # if defined all devices without platform or with a different platforms will be removed from the inventory
 supported_platforms = [ "cisco_ios", "cisco_nxos" ]
 
-# Settings for applying diffsync flags to diffsync model objects, in order to alter 
-# the underlying sync behaviour. The model_flag is applied to any objects that have a 
-# tag assigned within model_flag_tags. Further details on model_flags can be found 
-# at: https://github.com/networktocode/diffsync/blob/269df51ce248beaef17d72374e96d19e6df95a13/diffsync/enum.py
-model_flag_tags = ["your_tag"]
-model_flag = 1 # flag enum int() representation
-```
-
-
-    model_flag_tags: List[str] = list() # List of tags that defines what objects to assign the model_flag to.
-    model_flag: Optional[DiffSyncModelFlags]  # The model flag that will be applied to objects based on tag.
+# Configure which Inventory will be loaded by the network importer.
+inventory_class = "network_importer.inventory.NetboxInventory"
 
+[inventory.settings]
+# Inventory specific settings, please refer to the documentation of each backend/inventory.
+```
 
 ## Batfish Section
 
@@ -93,43 +89,40 @@ login = "username"          # Alternative Env Variable : NETWORK_DEVICE_LOGIN
 password = "password"       # Alternative Env Variable : NETWORK_DEVICE_PWD
 enable = true               # Alternative Env Variable : NETWORK_DEVICE_ENABLE
 
-# Connection parameters for Netmiko 
-global_delay_factor = 5
-banner_timeout = 15
-conn_timeout = 5
-
 # List of valid FQDN that can be found in the network,
 # The FQDNs in this list will be automatically removed from all neighbors discovered from LLDP/CDP
 fqdns = [ ]
-```
-
-## Inventory Section
 
-Define what method should be used to connect to the network devices. 
+[network.netmiko_extras]
+# Any additional parameters for Netmiko defined in this section will be automatically configured 
+# as part of the Nornir inventory.
 
-```toml
-[inventory]
-# The default method is to use the primary IP defined in NetBox.
-# As an alternative it's possible to use the name of the device and provide your own FQDN.
-use_primary_ip = false (default: true)
-fqdn = "mydomain.com"
-
-# Optional filter to limit the scope of the inventory, takes a comma separated string of key value pair"
-filter = "site=XXX,site=YYY,status=active"    # Alternative Env Variable : INVENTORY_FILTER
-
-# Configure what Inventory will be loaded bu the network importer.
-inventory_class = "network_importer.inventory.NetboxInventory"
+[network.napalm_extras]
+# Any additional parameters for Napalm defined in this section will be automatically configured 
+# as part of the Nornir inventory.
 ```
 
 ## Adapters Section
 
 Configure which adapters will be loaded by the network importer.
 Please see the [extensibility section](extensibility.md) of the documentation for more details on how to create your own adapter.
+The settings for the [nautobot adapter](backend/nautobot.md) and for the [netbox adapter](backend/netbox.md) are available in their respective documentation.
 
 ```toml
 [adapters]
 network_class = "network_importer.adapters.network_importer.adapter.NetworkImporterAdapter"
+
+
 sot_class = "network_importer.adapters.netbox_api.adapter.NetBoxAPIAdapter"
+
+[adapters.network_settings]
+# Network Adapter specific settings, any settings defined in this section will be passed to the Network Adapter. 
+# The default network adapter doesn't accept additional settings, nonetheless this section remains available if you want to
+# customize the default Network Adapter
+
+[adapters.sot_settings]
+# SOT Adapter specific settings, any settings defined in this section will be passed to the SOT Adapter. 
+# Please refer to the documentation of each adapter to see what settings are required/supported.
 ```
 
 ## Drivers Section

docs/extensibility.md

@@ -7,9 +7,10 @@ The extensibility principles leverages Python object extensibility. It's recomme
 You can use [setuptools](https://python-packaging-tutorial.readthedocs.io/en/latest/setup_py.html) or [Poetry](http://blog.networktocode.com/post/upgrade-your-python-project-with-poetry/) to quickly create a new python package.
 
 There are different ways the Network Importer can be extended depending on what you are trying to achieve:
-- [Add support for new device platform by extending the default drivers](#extend-the-default-drivers)
-- [Add support for new features buy extending the adapters](#extend-an-existing-adapter-or-provide-your-own)
-- [Add support for new data by extending the models](#extend-the-default-models)
+  - [Extend the default drivers](#extend-the-default-drivers)
+  - [Extend an existing adapter or provide your own](#extend-an-existing-adapter-or-provide-your-own)
+  - [Extend the default inventory](#extend-the-default-inventory)
+  - [Extend the default models](#extend-the-default-models)
 
 ## Extend the default drivers
 
@@ -40,7 +41,8 @@ class NetworkImporterDriver(DefaultNetworkImporterDriver):
             Result: Nornir Result object with a dict as a result containing the running configuration
                 { "config: <running configuration> }
         """
-        pass 
+        pass
+
     @staticmethod
     def get_neighbors(task: Task) -> Result:
         """Get a list of neighbors from the device.
@@ -99,7 +101,7 @@ class MyAdapter(NetworkImporterAdapter):
 
 ### Create a new adapter
 
-All adapters must implement the `load` method. The load method is called during the initilization process and is expected to load all the data from the remote system into the local cache, following the Models defined.
+All adapters must implement the `load` method. The load method is called during the initialization process and is expected to load all the data from the remote system into the local cache, following the Models defined.
 
 ```python
 from network_importer.adapters.base import BaseAdapter
@@ -120,6 +122,14 @@ network_class = "my_python_package.adapters.MyAdapter"
 sot_class = "my_python_package.adapters.MyOtherAdapter"
 ```
 
+## Extend the default inventory
+
+It's possible to extend the default inventory or provide your own inventory.
+
+A Network Importer inventory must be a valid Nornir 3.x inventory and it must be based of the NetworkImporterInventory class.
+
+Once you have created your own inventory, you need to register it with Nornir in order for Nornir to successfully load it. You than need to define your inventory name in [inventory.inventory_class]
+
 ## Extend the default models
 
 It's possible to extend the default models by creating new classes and attaching them to an adapter.