Advanced Topics

This section introduces some advanced features of the NETCONF protocol and YANG data modeling language.

Data Retrieval

Basic NETCONF Retrieval Operations

The NETCONF protocol has 3 different retrieval operations:

<get>: get state data and the running configuration database.
<get-config>: get just the specified configuration database.
<get-data>: get NMDA datastore data

Each of these operations accepts a <filter> parameter, which has 2 forms:

subtree filter: retrieve just the subtrees in the database that match the XML subtrees in the filter.
XPath filter: retrieve just the subtrees that match the result node set produced by evaluating the specified XPath expression against the database. This mode cannot be used unless the :xpath capability must be advertised by the server.

The yangcli-pro program supports 3 different forms of each command:

plain: plain NETCONF operation with user-supplied filter
subtree: XPath path expression or user variable is converted to XML for the <filter> parameter subtree XML.
xpath: XPath path expression or user variable is converted to XML for the <filter> parameter 'select' XML attribute

yangcli-pro Retrieval Commands

Command	NETCONF operation	Example
get	plain <get> operation	get with-defaults=trim
get-config	plain <get-config> operation	get-config source=candidate
get-data	plain <get-data> operation	get-data datastore=operational
sget	<get> with a subtree filter	sget /system
sget-config	<get-config> with a subtree filter	sget-config source=running /nacm/rules
sget-data	<get-data> with a subtree filter	sget-data datastore=running /nacm/rules
xget	<get> with an XPath filter	xget /system
xget-config	<get-config> with an XPath filter	xget-config source=running /nacm/rules
xget-data	<get-data> with an XPath filter	xget-data datastore=running /nacm/rules

The retrieval commands return an element named <data> containing the requested XML subtrees.

If any identifier nodes (YANG key leafs) are needed to distinguish the data in the reply, they will be added as needed by the server. In the xget example above, the <name> element for each interface would be returned, even though it was not directly requested by the XPath expression.

Default Value Filtering

The data will also be filtered according to the defaults handling behavior of the server, unless the <with-defaults> parameter is added to the command. This parameter is only supported if the server advertised the :with-defaults capability, If not, the client does not get any indication from the server what type of defaults filtering is being done (if any).

There are 4 types of defaults filtering provided:

report-all-tagged: no filtering -- return all nodes even those the server might normally suppress because they are considerer to be default values by the server. Add the 'default' XML attribute to nodes that were set by default.
report-all: no filtering -- return all nodes even those the server might normally suppress because they are considerer to be default values by the server.
trim: return all nodes except skip any leaf nodes that match the schema defined default value
explicit: return all nodes that were set by the client or the server to some value, even if the value happens to be the schema defined default. This is normally the default behavior for the netconfd-pro server.

The defaults handling behavior can be changed just for a specific NETCONF session, using the <set-my-session> operation. This is only available on the netconfd-pro server.

guest@localhost> set-my-session with-defaults=report-all

RPC OK Reply 12 for session 1:

guest@localhost>

In this example, the 'basic' behavior is changed from 'explicit' to 'report-all', but just for session 1. This setting is temporary, and will not be remembered when the session is terminated. If the <with-defaults> parameter is present, it will be used instead of this value.

Special Retrieval Operations

Any YANG module can add new operations with the 'rpc' statement.

New retrieval operations may also be added which are associated with a protocol capability.

Just like any other data model content, the operator (or application) needs to understand the YANG file definitions, including the description statements, to understand how each custom retrieval operation works.

There are some custom retrieval operations supported by netconfd-pro:

Special Retrieval Operations

Command	Description
get-bulk	Iterate through YANG list data from the server This is a proprietary operation defined in the yumaworks-getbulk.yang module.
get-walk	yangcli-pro command to utilize the <get-bulk> operation to step through retrieval of YANG list entries
get-schema	Retrieve the YANG or YIN source file for one of the modules advertised by the server The <get-schema> operation is defined in the ietf-netconf-monitoring.yang module.
get-my-session	Retrieve the customizable settings for my session. This is a proprietary operation defined in the yuma-mysession.yang module.

Notification Messages

Notifications are used in NETCONF to send server event information to the client application.

A session must request notifications with the create-subscription command. The netconfd-pro server also supports new notifications that are started with the establish-subscription command.

Most NETCONF notifications are published to 'event streams'. (The YANG Push feature allows subscriptions to datastores, instead of event streams).

The standard 'NETCONF' event stream must be supported by all servers, but proprietary event streams can be configured as well. A notification subscription request specifies the stream name (and perhaps more parameters).

A NETCONF session on the netconfd-pro server will never expire due to inactivity, while a notification subscription is active. This allows notification processing applications to maintain long-lived connections without worrying about a NETCONF timeout. Note that the SSH server may also be configured to drop idle SSH sessions, whether a notification subscription is active or not.

Notification Contents

The 'notification' element is sent from the server to the client, if an event occurs, and the client has created a notification subscription.

The child nodes of this element comprise the notification content, and it is divided into 3 sections:

event generation time-stamp: This standard NETCONF leaf is always the first child element within the notification element.
event payload: The module-specific event payload is represented as a container with the name of the notification. Any data nodes defined within the YANG notification statement appear (in order) as child nodes of the event type container.
proprietary extensions: Zero or more vendor-specific elements may appear after the event payload element. For example, the monotonically increasing 'sequence-id' element is added to each notification saved in the netconfd-pro event log.

Notification Replay

The NETCONF server will maintain an ordered buffer of saved notification events, if the :notification-replay capability is supported by the server. For the netconfd-pro server, this is a configurable feature, set by the --eventlog-size parameter.

The netconfd-pro default is to save the most recent 1000 notification events.

Only system events are saved and are available for retrieval. The <replayComplete> Event and <subscription-terminated> Event are session-specific events, and are therefore not saved in the replay buffer.

The create-subscription command has 2 parameters to request that stored notifications be delivered to the client session:

startTime: the date (or date-and-time) to compare against the event generation time-stamp. Only notification events that occurred after this time are delivered.
stopTime: the date (or date-and-time) to compare against the event generation time-stamp. Only notification events that occurred before this time are delivered. This parameter can specify a time in the future. When that time has passed, the subscription will be terminated. The stopTime does not cause the server to wait that period of time to generate an event. If the stopTime is in the past, then the subscription will terminate after all the matching event timestamps in the replay buffer have been delivered.

Notifications are delivered in the order they are stored. Each new netconfd-pro notification contains a monotonically increasing sequence-id (unsigned integer). This can be used to help determine if any configured notification filters are working as expected.

The interleave capability

The netconfd-pro server supports the :interleave capability, which means that all commands (except create-subscription) will be accepted by the server. The client should expect <rpc-reply> and <notification> messages. The server will always maintain proper message serialization. These messages will always be sent in their entirety, which may impact applications (e.g., a really long <get> response on the same session will delay notification delivery).

If the NETCONF server does not support the :interleave capability, then it may only allow the <close-session> operation while the notification subscription is active. In this case, a new NETCONF session is required to perform any management operations.

This special mode is only applicable while a notification subscription is active. It is possible for a replay subscription to terminate, without terminating the session as well. In this case, the 'notificationComplete' event will be generated, and the session will return to accepting all possible operations.

Database Editing

NETCONF supports multiple conceptual configuration databases. Only the 'running' database is actually active. All other databases are scratch-pad databases, or some other special-purpose off-line database.

Every NETCONF server must allow arbitrary partial (and concurrent) editing to its configuration with the <edit-config> operation. The yangcli-pro program has simplified editing commands, which are explained below.

The <config> element within an <edit-config> PDU represents the 'root node' (/) in the path expression for each node in the conceptual database. Each top-level YANG object that is supported and configured will be represented as child nodes this root node. The conceptual database can be processed as an XML instance document with multiple top nodes (similar to XSLT rules).

Database editing in NETCONF has several variants, but basically, it follows this simple procedure:

Lock the database(s) that will be affected.
Use <edit-config> or <copy-config> on the target database to make changes.
Activate and save the database edits.
Unlock the database(s) that were previously locked.

The Target Database

Usually a NETCONF server supports the <edit-config> operation on only one database, which is either the candidate or the running database. This is called the 'target' database, which corresponds to the <target> parameter in the <edit-config> operation.

If the target database is the candidate configuration, then the <edit-config> operation does not always cause all possible database validation checking to be done by the server. Since the candidate database is just a scratch-pad for (possibly) incremental edits, the server is not required to completely validate its contents. Instead, these 'final validation' tests are only required to be done when the <commit> operation is invoked.

The yangcli-pro program will automatically handle the target database management, based on the server capabilities reported each session, if the save command is used. The manual procedure (<commit> and/or maybe <copy-config> operations) is also supported, but do not mix them within the same editing session.

NETCONF Database Locking

NETCONF supports database locking so a session can have exclusive write access to the configuration.

These locks are intended to be short-lived, but there is no actual time limit on a lock. If the session terminates for any reason with any locks, they will be released automatically by the server.

All the databases that are involved in the edit should be locked. This always includes the running database, and the candidate and startup databases, if they are supported by the server.

Refer to the Database Locking section for more details on netconfd-pro database locking.

The yangcli-pro program has 2 special commands to handle all locking:

get-locks: Wait until all database locks have been acquired or the timeout occurs
release-locks: Release any locks that were obtained with get-locks

Non-Volatile Storage

The startup configuration is the conceptual database used on the next reboot of the NETCONF server. It is important to know whether the NETCONF server supports the :startup capability or not. If yes, then the operator must explicitly save the running database to non-volatile storage (the startup database), using the <copy-config> operation. If no, then the server will keep the running and startup databases synchronized.

The yangcli-pro program has a high-level save command, used after the editing operations, that will automatically issue the correct protocol operations to complete the edit, and save the changes in non-volatile storage.

The startup database is configurable in the netconfd-pro server. The --with-startup configuration parameter controls whether the startup database will be used or not. The --startup parameter can be used to control the initial load of the running configuration in 3 different ways:

no startup: skip this step and just use factory defaults
default startup: look for the default startup-cfg.xml file in the configured data path.
specific startup: use a specified file, either absolute file-spec, or a relative path in the configured data path

Refer to the YumaPro User Manual for more details on controlling non-volatile storage.

Editing Commands

The <edit-config> operation should be used make configuration changes. The <copy-config> operation can also be used, but this is a blunt hammer approach. Although the netconfd-pro server will always analyze the edit request and only affect the nodes that actually changed, this is not a requirement in the standard.

The <edit-config> operation allows the operator to have precise control of the server. These database edits are performed by the server using a combination of 3 factors:

The nodes that currently exist in the target database.
The nodes that exist in the 'source' of the edits (either the inline <config> element or indirectly through the <url> element.
The <default-operation> parameter and any XML attributes in the source XML elements (nc:operation attribute and YANG insert operation attributes).

The yangcli-pro program provides some high-level commands to automatically handle the complexity of the <edit-config> operation. These commands use XPath expressions and a series of interactive prompts (e.g., for the mandatory nodes and key leafs) to fill in the specified data structures, and construct an optimized NETCONF message.

yangcli-pro Editing Commands

Command	Description
create	Create a new sub-tree, only if it does not already exist
delete	Delete an existing sub-tree. It is an error if it does not exist
delete-all	Delete all instances of a list or leaf-list. It is an error if no entries exist.
merge	Merge the source sub-tree into the target sub-tree, keeping any existing nodes that are not explicitly contained in the source.
remove	Delete an existing sub-tree. It is not an error if it does not exist.
remove-all	Delete all instances of a list or leaf-list. It is not an error if no entries exist.
replace	Merge the source sub-tree into the target sub-tree, deleting any existing nodes that are not explicitly contained in the source. This is the mode used for the <copy-config> operation.
insert	Insert or move a YANG list or leaf-list entry

NETCONF Access Control

The netconfd-pro server can be configured to give precise access rights to each user (the SSH user name associated with the NETCONF session). By default, the access control model used is 'ietf' (RFC 8341).

Refer to the Access Control section for additional NACM details.

Some important points to remember about access control:

There are 3 types of access -- read, write, and execute.
If a user does not have read access to some data, then it is silently omitted from the reply.
The 'access-denied' error is not generated for read requests. It is only generated for write requests to the database, or <rpc> operation execution requests.
An access request results in 1 of 2 outcomes: permit or deny
The server resolves the access request by searching the access control rules. Either an explicit rule will apply, or the default access rights will be checked if no rule is found.
The default access rights are configurable, but usually set as follows:
- read access is permitted
- write access is denied
- exec access is permitted
The nacm:default-deny-write and nacm:default-deny-all extensions can be used by the YANG module author to override the default access rights, and deny access instead. For example, the <shutdown> operation is not permitted by default.
There is a configurable 'superuser' user name. If desired, one or more specific user names will be considered to be 'superuser' accounts, and all access control will be bypassed for these user (e.g.,--superuser=joe). By default, there is no superuser account configured.

yangcli-pro Variables

The yangcli-pro program supports variables for easier reuse and script-based operations.

There are 2 types of variables:

file variables: the variable name is a file name, and the contents of the variable are stored in this file.
internal variables: the variable name is just an internal identifier, and the contents of the variable are stored in memory

Variables are set with assignment statements. Here are some examples:

guest@localhost> $$backup = get-config source=running

guest@localhost> $$bad-data = "warn"

guest@localhost> $itf = "//interface[name='eth0']"

Refer to the Variables section for details on using yangcli-pro variables.

yangcli-pro Scripts

Scripts are simply a collection of yangcli-pro commands and/or assignment statements that are stored in a text file, instead of typed directly. Scripts can call other scripts and pass parameter values.

The run command is used to invoke a yangcli-pro script.

Bash (and other) shell scripts are also supported by yangcli-pro. The run-shell command is used to invoke such a script.

Refer to the Scripts section for details on using yangcli-pro scripts and shell scripts.