Troubleshooting

This section addresses common problems that can occur using the server.

Session Connection Issues

Note

Symptom: Client cannot start a session with the server

Client connection issues depend on the protocol.

SSH Connection Issues

If a client session is using the SSH server, then this may need to be checked first. This affects protocols:

  • CLI over SSH

  • NETCONF over SSH

../_images/SSH_path.png

If there is no 'debug' log activity in the netconfd-pro program when the client session is attempted, then the session request is not reaching the NETCONF server. If there is log activity related to the client session, then skip ahead to the Check Server Restricting New Sessions section.

Check the Client Parameters

If a 'connection refused' message is given in the client, then make sure the host and port parameters used by the client are correct in the connection attempt.

  • Usually ports 22 and 830 are enabled

  • Additional ports or different ports can actually be used

Check the Server Log Activity

If a 'connection refused' error is received then the server is not listening on that port. No log entries for the failed connection will be created in this case.

If no netconfd-pro log activity occurs when --log-level is set to debug2 or higher, then the SSH server is most likely not invoking /usr/sbin/netconfd-subsystem-pro for various reasons. The NETCONF server or yp-shell session is never getting the session started.

Check the activity in the SSH server log files to determine if the SSH session is getting handled correctly:

  • Connection failures can usually be found in the /var/log/auth.log file.

  • SSH server activity can usually be found in the /var/log/syslog file.

An example of a SSH connection failure in the log may appear in 'auth.log'

Aug 12 14:32:33 andy-i9-homedev sshd[1752789]: Invalid user fred from 127.0.0.1 port 52218
Aug 12 14:32:38 andy-i9-homedev sshd[1752789]: pam_unix(sshd:auth): check pass; user unknown
Aug 12 14:32:38 andy-i9-homedev sshd[1752789]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=127.0.0.1
Aug 12 14:32:41 andy-i9-homedev sshd[1752789]: Failed password for invalid user fred from 127.0.0.1 port 52218 ssh2
Aug 12 14:32:43 andy-i9-homedev sshd[1752789]: Connection closed by invalid user fred 127.0.0.1 port 52218 [preauth]

An example of a successful SSH connection may appear in 'auth.log':

Aug 12 14:37:40 andy-i9-homedev sshd[1753016]: Accepted password for admin1 from 127.0.0.1 port 40726 ssh2
Aug 12 14:37:40 andy-i9-homedev sshd[1753016]: pam_unix(sshd:session): session opened for user admin1(uid=1002) by (uid=0)
Aug 12 14:37:40 andy-i9-homedev systemd-logind[853]: New session 3019 of user admin1.
Aug 12 14:37:40 andy-i9-homedev systemd: pam_unix(systemd-user:session): session opened for user admin1(uid=1002) by (uid=0)

The 'syslog' file will also have an entry for the successful SSH connection. Example:

Aug 12 14:37:40 andy-i9-homedev systemd[1]: Started Session 3019 of User admin1.

Check the SSH Server Config

Check if the correct ports are configured for the SSH server.

Example 'Port' lines in sshd_config:

Port 22
Port 830
If the correct ports are set then make sure the netconf

subsystem is invoked correctly. The exact line should appear in the ssd_config file:

Check the SSH server configuration, usually /etc/ssh/sshd_config

Subsystem       netconf /usr/sbin/netconf-subsystem-pro

Check Local SSH Connection

If the sshd_config file is correct, make sure the SSH server is running, by using the Linux 'ssh' command to connect to the server.

From the same host as the server, the following command should work if 'Port 22' is enabled in the sshd_config file. Use a different port number if needed with the -p option.

> ssh localhost

If the SSH server is running and OK then a terminal session should be established. Use 'exit' to terminate the SSH session.

Check Remote SSH Connection

If the SSH server is working for local terminal sessions then test if it is working for remote sessions

If the SSH server is running and OK then a terminal session should be established. Use 'exit' to terminate the SSH session.

Check the 'netconf' subsystem

If the SSH server is accepting plain terminal sessions correctly, then check if the 'ssh' program connects to the 'netconf' subsystem from a local session:

> ssh -s -p 830 localhost netconf

If the NETCONF server is working properly, then a 'hello' message will be sent by the server, and it will be waiting for the client to send its 'hello' message.

Example Server Hello:

<?xml version="1.0" encoding="UTF-8"?>
<hello xmlns="urn:ietf:params:xml:ns:netconf:base:1.0">
 <capabilities>
  <capability>urn:ietf:params:netconf:base:1.0</capability>
  <capability>urn:ietf:params:netconf:base:1.1</capability>
  <!-- Most capability elements removed for brevity -->
 </capabilities>
 <session-id>7</session-id>
</hello>]]>]]>

Note

If the SSH server is not starting a NETCONF session but it is accepting the SSH session, then the NETCONF subsystem program needs to be checked.

Refer to the Checking the NETCONF Subsystem section for more details.

TLS Connection Issues

The NETCONF Over TLS feature uses the OpenSSL library to directly handle incoming client sessions. The thin client used by SSH is not used for this protocol.

../_images/TLS_path.png

If there is no 'debug' log activity in the netconfd-pro program when the client session is attempted, then the session request is not reaching the server. If there is log activity related to the client session, then check the Check Server Restricting New Sessions section.

Check the TLS Configuration

Example Server Config File to Enable NETCONF over TLS

This is just an example. A real config file will have more settings and may not use all the settings shown here:

netconfd-pro {
 with-netconf-tls true
 netconf-tls-certificate /home/andy/certs/server.crt
 netconf-tls-key /home/andy/certs/server.key
 cert-usermap admin1@68:E5:71:6C:C9:8D:33:F2:DC:01:43:F8:E8:8B:CB:3D:BD:9C:2E:2F
}

Example yangcli-pro Config File to Enable NETCONF over TLS

This is just an example. A real config file will have more settings and may not use all the settings shown here:

yangcli-pro {
  ssl-certificate ~/certs/client.crt
  ssl-key ~/certs/client.key
  message-indent 1
  use-rawxml true
}

Check the Client Certificate

If the Client Certificate is rejected, then the following log error message may appear in the server log when the client session attempt is made:

agt_openssl: Got other error during SSL handshake, status:-1, err:1
agt_openssl: SSL_accept failed

Example connect command if the yangcli-pro is used:

> connect user=admin1 no-password server=localhost transport=tls

Example TLS Connect Failure

If the --log-level is set to 'debug4' then the following example shows how a client verify fail error may appear:

 Connect attempt with following parameters:
 connect {
   user admin1
   server localhost
   no-password
   transport tls
   timeout 30
   public-key $HOME/.ssh/id_rsa.pub
   private-key $HOME/.ssh/id_rsa
   ssl-fallback-ok true
   ssl-certificate ~/certs/client.crt
   ssl-key ~/certs/client.key
   ssl-trust-store $HOME/.ssl/trust-store.pem
   ncport 830
 }

 ses_msg: new out buff 0x55c927b855d0 for s 0
 Starting NETCONF session for admin1 on localhost over TLS on port 6513
 OpenSSL verify callback
 Subject: /C=ca/ST=ca/L=ca/O=ca/CN=ca
 Issuer: /C=ca/ST=ca/L=ca/O=ca/CN=ca
 depth: 1
 err: 0
 preverify: 1
 return: 1

 OpenSSL verify callback
 Subject: /C=rc/ST=rc/L=rc/O=rc/CN=restconf
 Issuer: /C=ca/ST=ca/L=ca/O=ca/CN=ca
 depth: 0
 err: 10
 preverify: 0
 return: 0

 Error: BIO_do_connect failed
     [library name]: SSL routines
     [function name]: OPENSSL_internal
     [reason string]: certificate verify failed
 Error: failed to establish secure connection with server (localhost:6513)

Check the Cert to User Map Settings

If the Client and Server are both accepting the certificates from the other peer, then the user-name assignment for the session needs to be checked.

There are generally 2 ways the user-name can be assigned to a NETCONF over TLS or RESTCONF over TLS session:

  1. Derive a user-name from the SAN in the certificate

  2. Find a cert-usermap entry that matches the fingerprint of the client certificate

The OpenSSL internal APIs will authenticate the client certificate and the client CA certificate.

  • If the certificate is not accepted then the session will be dropped

  • If the certificate is accepted then the client user identity is derived from the certificate. This is only possible if a SAN is properly configured. See the Generating Certificates with a SAN section for details.

  • If no SAN is present then the server will look for a cert to usermap entry, to assign a user-name to the session.

  • the --cert-usermap CLI parameter can be used to create static mappings at boot-time

  • the yumaworks-cert-usermap.yang module can be used to manage dynamic mappings at run-time

  • In a DEBUG=1 build, the --cert-default-user parameter can be used to assign a user-name if none is derived.

  • In a non-DEBUG build, the session will be dropped if no user-name is assigned to the session.

Example Server Log if Cert to Usermap Entry Found

 agt_openssl: enter verify_callback
 Certificate details:
   Subject: /C=ca/ST=ca/L=ca/O=ca/CN=ca
   Issuer: /C=ca/ST=ca/L=ca/O=ca/CN=ca
   depth: 1
   err: 0
   errstr: ok
   preverify: 1
   return: 1
 agt_openssl: Checking digest: 21:80:9E:4F:FA:76:D9:03:3F:03:E1:8A:34:DD:AF:21:00:CE:05:AE
 Checking --cert-usermap param 1
 agt_openssl: No username found
 agt_openssl: exit verify_callback

 agt_openssl: enter verify_callback
 Certificate details:
   Subject: /C=cl/ST=cl/L=cl/O=cl/CN=client.com
   Issuer: /C=ca/ST=ca/L=ca/O=ca/CN=ca
   depth: 0
   err: 0
   errstr: ok
   preverify: 1
   return: 1
 agt_openssl: Checking digest: D8:F4:90:DE:45:75:F5:04:C8:A5:7E:D1:13:4E:21:9A:F2:0C:EC:F4
 Checking --cert-usermap param 1
 Got certmap type (1) specified
 Found --user-certmap in entry 1 user=admin1
 agt_openssl: exit verify_callback

RESTCONF Connection Issues

The RESTCONF protocol is not handled directly by netconfd-pro. Instead a WEB server must be installed and configured to invoke the 'thin client' using the FastCGI interface.

../_images/RESTCONF_path.png

Usually the thin client program is installed as /usr/sbin/restconf, and it is invoked when a client request is processed by the WEB server.

RESTCONF does not actually have sessions like NETCONF:

  • Each WEB request causes a new session request to the NCX socket.

  • A separate RESTCONF session within netconfd-pro is used for each WEB request.

To debug RESTCONF sessions:

  • Confirm the WEB server is working correctly

  • Confirm the 'restconf' thin client program is working correctly

  • Confirm the session is accepted by netconfd-pro

Check the RESTCONF CLI Parameters

Make sure the RESTCONF CLI parameters are correct:

Check the RESTCONF WEB Server

If there is no 'debug' log activity in the netconfd-pro process when the client session is attempted then the WEB request from the client is not getting to the netconfd-pro process.

Make sure the WEB server that is supposed to invoke the 'restconf' subsystem is running and is configured properly.

[TBD]

Check the RESTCONF Subsystem

If the WEB server is working properly then the FastCGI program called 'restconf' is invoked from the WEB server.

  • This program gathers the RESTCONF request parameters from the environment variables passed from the WEB server.

  • A NCX session is then started with the netconfd-pro process.

  • The session lasts for the duration of the message.

Refer to the Checking the NETCONF Subsystem section for details on checking the subsystem log files.

Check the RESTCONF Server Log Activity

If there is log activity related to the client session, then check the Check Server Restricting New Sessions section.

Check the RESTCONF User Name

If the REMOTE_USER environment variable is set correctly, then this value should be passed to the netconfd-pro process as the user name for the session.

If this variable is missing then the following message should be present in the subsys log file:

Error: Missing REMOTE_USER

If no user name is found then the string restconf will be used as the user-name.

Checking the NETCONF Subsystem

There are some programs which are used as a 'thin client' to connect to the netconfd-pro process. This is usually done with an 'AFLOCAL' socket.

The following programs use this thin client to connect to the netconfd-pro process.

  • netconf-subsystem-pro

  • restconf

The special 'NCX' socket is used by this thin client.

/tmp/subsys-err Log Files

Warning

  • Enabling subsystem logging will impact server performance and potentially use significant disk space.

  • Only use as a temporary measure during debugging.

  • These log files are not deleted after the subsystem exits.

  • They must be deleted manually from the /tmp directory

The thin client program can produce a log file. This can be examined to determine if any settings or errors are occurring in this program.

The log files must be enabled, 1 of 2 ways:

  1. Compile the subsystem sources with DEBUG=1 and DEBUG2=1

  2. Invoke the subsystem with the '-t' option. This can be done for SSH by modifying the /etc/ssh/sshd_config file.

    • Existing line to invoke subsystem

    Subsystem        netconf    /usr/sbin/netconf-subsystem-pro
    
    • Change line to enable trace level 3 (highest)

    Subsystem        netconf    /usr/sbin/netconf-subsystem-pro -t 3
    

The log file names have a specific format, using the process PID number to make each file name unique:

/tmp/subsys-err.PPPPPP.log

Where PPPPPP is the PID number.

Example Log File Name:

/tmp/subsys-err.304840.log

Example Log File If netconfd-pro Not Running

If the SSH server is running but the netconfd-pro process cannot be reached, then the thin client program will fail, and the log file will usually indicate why the session was 'shut by remote peer'.

 *** New NETCONF Session Started ***

 traceLevel 3
 content_len -1
 Got USER variable 'admin1'
 Got SSH_CONNECTION variable '127.0.0.1 35182 127.0.0.1 830'
 ERROR: init_subsys(): NCX Socket Connect failed (errno=No such file or directory) FD:4
 ERROR: return 315

Example Log File If netconfd-pro Starts OK

If the SSH server is running and the netconfd-pro process can be reached, then the thin client program will not fail, and the log file will contain the entire session activity.

The start of the log file may look as follows:

 *** New NETCONF Session Started ***

 traceLevel 3
 content_len -1
 Got USER variable 'admin1'
 Got SSH_CONNECTION variable '127.0.0.1 43638 127.0.0.1 830'
 DEBUG:  init_subsys(): NCX Socket Connected on FD: 4
 DEBUG: starting io_loop()
 DEBUG: io_loop: about to call select
 DEBUG: read STDIN
 DEBUG: do_read: OK (250)
 DEBUG: io_loop: send to NCXSOCK (250)
 DEBUG: io_loop(): Sending buff

Check Server Restricting New Sessions

There are CLI parameters which may affect how sessions are processed, that can cause a 'NCX Connect' request to be denied.

Allowed Users List

The --allowed-user leaf-list parameter allows the specific user names to be configured.

  • If this list is empty, then any user name will be accepted

  • If this list is not empty then only user names in this list will be allowed to start a client session.

  • This does not affect subsystem sessions, only client sessions.

The server will simply terminate the session. The log file will indicate that the allowed-user test failed:

 agt_connect: got node for session 4
 agt_connect: got valid version attr
 agt_connect: got valid magic attr
 agt_connect: transport='ssh'
 agt_connect: protocol='netconf'
 agt_connect: got valid port attr
 agt: allowed-user check failed for 'admin1'
 agt_connect error: user 'admin1' not in allowed-user list
 agt_connect error (access denied)
   dropping session 4

The subsys log file may appear as follows when the NCX Connect succeeds but the server immediately drops the session:

 *** New NETCONF Session Started ***

 traceLevel 3
 content_len -1
 Got USER variable 'admin1'
 Got SSH_CONNECTION variable '127.0.0.1 40400 127.0.0.1 830'
 DEBUG:  init_subsys(): NCX Socket Connected on FD: 4
 DEBUG: starting io_loop()
 DEBUG: io_loop: about to call select
 DEBUG: read STDIN
 DEBUG: do_read: OK (250)
 DEBUG: io_loop: send to NCXSOCK (250)
 DEBUG: io_loop(): Sending buff

 <?xml version="1.0" encoding="UTF-8"?><hello xmlns="urn:ietf:params:xml:ns:netconf:base:1.0"><capabilities><capability>urn:ietf:params:netconf:base:1.    0</capability><capability>urn:ietf:params:netconf:base:1.1</capability></capabilities></hello>]]>]]
 DEBUG: io_loop: about to call select
 DEBUG: read NCXSOCK
 INFO: do_read(): closed connection
 INFO: io_loop(): exited OK
 OK return

Max Sessions Limit

The maximum number of concurrent client sessions is limited by the --max-sessions parameter. The default value is 8.

  • After this number of client sessions is already active, then new client session requests will be rejected

  • Sessions from subsystems are not affected, only client sessions

  • yp-shell sessions are constrained by the --max-cli-sessions parameter as well.

The server log file will contain a message about the dropped session. Example:

 Enter agt_ses_new_session for transport 'netconf-ssh', fd 7
 agt_ses: Drop session because max-sessions reached

 agt_ncxserver: new session failed (7)

YP-HA Waiting Role

If the server is running with --ha-enabled='true' then it is possible the server is waiting for its HA role and is rejecting new client sessions.

In this case the server will generate a log entry that says 'init2 not done'.

An example of the server log may appear as follows in this case:

 agt_top: got node
 agt_top: start dispatch yuma-ncx:ncx-connect
 agt_connect: got node for session 7
 agt_connect: got valid version attr
 agt_connect: got valid magic attr
 agt_connect: transport='ssh'
 agt_connect: protocol='netconf'
 agt_connect: got valid port attr
 agt: skip allowed-user check for 'andy'; not configured
 agt_connect error: init2 not done
 agt_connect error (access denied)
   dropping session 7

Session Behavior Issues

Note

Symptom: Client session starts but is not working correctly

Unexpected Error Responses

If the session is working correctly, then requests will cause a reply from the server. If the server rejects an RPC request then it will respond with an error:

  • NETCONF: <rpc-error> response

  • RESTCONF: <error> response

  • CLI: <rpc-error> log message

Check the Session Timeout Issues section if the response is not getting received in a normal amount of time.

Wrong Config State Error

The error message wrong config state is used by the following error-tag:

  • ERR_NCX_NO_ACCESS_STATE = 302

There are two different causes for receiving this error:

  1. Waiting for SIL-SA Bundle Load

  • Operation is a retrieval or edit operation

  • Error number 302 returned

Any attempt to read or write any datastore data will fail if the server has not loaded the datastores yet.

  • This usually means there is a --bundle parameter used and no SIL library was found for the bundle.

  • The server does not know the modules used in the bundle until the bundle is actually loaded.

  • The datastores cannot be loaded yet in case there are any data nodes from missing bundles.

  • The server is waiting for a sil-sa-app process to register with the server and register for the missing bundle(s).

  1. Unlock a Datastore that is Not Locked

  • Operation is <unlock>

  • Error 302 returned

Any attempt to unlock a datastore that is already unlocked will fail with a '302' error-number.

Example RPC Error:

<rpc-reply message-id="1" xmlns:ncx="http://netconfcentral.org/ns/yuma-ncx"
 xmlns="urn:ietf:params:xml:ns:netconf:base:1.0">
 <rpc-error>
  <error-type>protocol</error-type>
  <error-tag>operation-failed</error-tag>
  <error-severity>error</error-severity>
  <error-app-tag>no-access</error-app-tag>
  <error-message xml:lang="en">wrong config state</error-message>
  <error-info>
   <error-number>302</error-number>
  </error-info>
 </rpc-error>
</rpc-reply>

No Access Error

The error message access denied is used by the following error-tag:

  • ERR_NCX_ACCESS_DENIED = 267

Example RPC Error:

<rpc-reply message-id="3" xmlns:ncx="http://netconfcentral.org/ns/yuma-ncx"
 xmlns="urn:ietf:params:xml:ns:netconf:base:1.0">
 <rpc-error>
  <error-type>protocol</error-type>
  <error-tag>access-denied</error-tag>
  <error-severity>error</error-severity>
  <error-app-tag>no-access</error-app-tag>
  <error-message xml:lang="en">access denied</error-message>
  <error-info>
   <error-number>267</error-number>
  </error-info>
 </rpc-error>
</rpc-reply>

Session Timeout Issues

The --timeout parameter used in yangcli-pro sessions has a default value of 30 seconds. For some operations, such as a complex commit on a large datastore, this may not be enough time.

  • Try a higher timeout value for the request if the server is sending late responses

If the server is single-threaded (i.e. no PTHREADS=1 make flag used) then the server may be busy with another session.

Even if the server is multi-threaded, it is possible for sessions to wait on other sessions. E.g.:

  • module or bundle is being loaded or unloaded

  • datastore is being edited

  • subsystem is initializing

If no other sessions are causing the server to be busy, then the server log activity needs to be checked.

Configuration Editing Issues

Note

Symptom: Edit operation is not working correctly

YANG Validation Issues

If the --target parameter is set to candidate, then the YANG validation 'root-check' is usually done during the <commit> operation. It may be done during <edit-config> if the 'test-option' has the value 'test-then-set'.

  • The YANG Constraints are defined in section 8 of RFC 7950. There are many different errors that may be returned.

  • The NETCONF Error Responses for YANG-Related Errors are defined in section 16 of RFC 7950. The server follows these requirements for YANG validation errors.

  • To determine if the <candidate> datastore passes YANG validation, use the <validate> operation.

Required Instance Test Failed

The error message require-instance test failed is used by the following error-tag:

  • ERR_NCX_MISSING_INSTANCE = 350

Leafref nodes usually require that the 'pointed-at' instances contain any values used by a 'pointing-at' leaf. Errors can occur different ways:

  • Attempt to create a 'pointing-at' leaf or leaf-list but none of the 'pointed-at' leaf instances match this value

  • Attempt to delete or change a 'pointed-at' leaf but at least one of the 'pointing-at' leaf instances match this value, and deleting or changing the leaf would cause that value to be unavailable.

Example YANG:

leaf one {
  type string;
}

container top {
  leaf two {
    type leafref {
      path "/one";
    }
  }
}

The same error is returned for different edit operations

  • The 'error-path' will indicate a 'pointing-at' leaf or leaf-list, even if the 'pointed-at' leaf is the data node that is being edited

  • The 'error-tag' will be data-missing even if the edit operation is an attempt to delete or change a 'pointed-at' leaf.

  • The default 'error-message' will be required value instance not found even if the edit operation is an attempt to delete or change a 'pointed-at' leaf.

Example RPC Error Reply

<rpc-reply message-id="6"
 xmlns:my3="urn:yumaworks:params:xml:ns:yang:mytest3"
 xmlns:ncx="http://netconfcentral.org/ns/yuma-ncx"
 xmlns="urn:ietf:params:xml:ns:netconf:base:1.0">
 <rpc-error>
  <error-type>application</error-type>
  <error-tag>data-missing</error-tag>
  <error-severity>error</error-severity>
  <error-app-tag>instance-required</error-app-tag>
  <error-path>/my3:top/my3:two</error-path>
  <error-message xml:lang="en">require-instance test failed</error-message>
  <error-info>
   <error-number>350</error-number>
  </error-info>
 </rpc-error>
</rpc-reply>

SIL or SIL-SA Callback Issues

Note

Symptom: SIL Callback for RPC, Action, EDIT, or GET2 is not working correctly

Callback Crashes With Garbage or NULL Pointer

If the server invokes the callback function, and parameters or fields within a struct are NULL or some invalid value, then could be a build problem.

  • Make sure all old libraries and binary code are removed.

  • Rebuild all code from sources or binary libraries

  • Add the DEBUG=1 make flag to enable symbolic debugging

  • Make sure the correct SIL or SIL-SA Compiler Flags are used for your binary package

Callback Crashes When a VAL macro is Used

The val_value_t struct is often used within YANG instrumentation code.

Check the Base Type

The 'btyp' is a critical field that determines which variant of the v_ union is used. If the wrong macro is used for the basetype, then the code is very likely to crash at that point.

ncx_btype_t btyp = VAL_TYPE(val);

Example: Use 'uint32' macro but the btyp is really NCX_BT_STRING

// WRONG!!! CAUSES CRASH!!!
uint32 num = VAL_UINT32(val);

// BETTER TO CHECK THE TYPE!!!
if (VAL_TYPE(val) == NCX_BT_UINT32) {
    uint32 num = VAL_UINT32(val);
}

Check the Leafref Type

The 'btyp' for a leafref node starts out as NCX_BT_LEAFREF.

  • If the 'btyp' is set to this value then the VAL_STRING() macro must be used to access the preliminary value.

  • To access the real value in this case, the 'val_convert_leafref' function can be used

if (VAL_TYPE(val) == NCX_BT_LEAFREF) {
    val_value_t *realval = val_convert_leafref(val);
    // ...
}
val_value_t *val_convert_leafref(const val_value_t *val)

Convert a value of type NCX_BT_LEAFREF to the value that the final leafref is pointing at.

Parameters:

val -- value to convert; this must be an NCX_BT_LEAFREF

Returns:

malloced val_value_t that must be freed with val_free_value the val->btyp and val->typdef and val->v fields will be set based on the real type; the val->obj will still point at the original object