Benchmark Application Example

The Benchmark application allows for measuring throughput, packet error rate, latency, and CPU utilization in different scenarios. The application exposes a CLI interface for efficient test management. Additionally, it supports several Bluetooth features:

• advertisement interval and other parameters,
• scanning and connection to multiple BLE devices,
• sending payload over BLE using the Nordic UART Service (NUS).

Thanks to these features, it is possible to measure the performance impact of the multiprotocol solution.

Note

For measurement purposes, use at least two boards when running this application.

See Testing multiprotocol scenarios for testing scenarios that use multiprotocol solutions and are based on this Benchmark Application example.

Description

The Benchmark application provides two CLI interfaces:

• UART, which uses UART communication and SEGGER J-Link as a USB-UART converter.
• RTT, which uses the SEGGER J-Link RTT transport.

A Thread application automatically creates or joins a network on channel 11.

In a Zigbee appllication, the creation and joining process must be performed manually. By default, the new Zigbee network is created on channel 16. For both types of applications, CLI commands can be run using a terminal application like PuTTY, configured according to the following CLI terminal settings.

Measured parameters

The following table lists the parameters measured by the application.

Test modes

The application supports the following modes:

• Unidirectional – the application sends packets in one direction and tries to saturate the wireless traffic.
  
  
  Benchmark Unidirectional test mode
  
• ACK – the application sends a packet and waits for a short acknowledgment from the peer before sending the next packet. If no acknowledgment is received:
  • In a Thread application, the ACK timeout value is used for triggering the timeout event of the waiting for acknowledgment state.
  • In a Zigbee application, the APS ACK timeout mechanism (9 seconds) is used.
    
    
    Benchmark ACK test mode
    
• Echo – the application sends a packet and waits for an echo from the peer before sending the next packet. If no echo is received, the ACK time-out value is used for triggering the time-out event of the waiting for the echo state.
  
  
  Benchmark Echo test mode
  

Test configuration commands

You can use the commands from the following table to configure your tests.

Bluetooth configuration commands

Use the commands from the following table to configure and run the BLE features supported by this example.

Bluetooth advertising configuration

Use the commands from the following table to configure the BLE advertising during the tests:

Bluetooth scanning configuration

Use the commands from the following table to configure the BLE scanning during the tests:

Bluetooth connection configuration

Use the commands from the following table to configure the BLE connection during the tests:

Note

The command ble conn dle [value] sets the maximum data length that can be sent in a single packet (L2CAP and ATT headers plus payload). The actual ATT payload that can be sent in a single frame is smaller by L2CAP and ATT header sizes (7 octets). This value must be set before a connection is established for both central and peripheral devices. Maximum value is 251.

Bluetooth ping configuration

Use the commands from the following table to send diagnostic packets to the connected device.

Bluetooth flood configuration

Use the commands from the following table to configure and send diagnostic packets continuously to the connected device. The flood commands can only be used by the peripheral device.

CPU utilization commands

The CPU utilization can be started and stopped multiple times. This allows measuring the average utilization from many non-continuous time periods. The clear command removes stored data.

Use the commands from the following table to configure the CPU utilization measurement.

OpenThread commands (Thread-only)

The Thread variant of the Bechmark Application supports OpenThread CLI commands. For the complete list of supported commands, see OpenThread CLI reference.

Execute these commands in the Benchmark Application with the following formula:

ot <OpenThread command>

Running commands remotely on discovered peers (Thread-only)

You can also issue commands on the discovered peers. The command response is then forwarded to the origin of a remote command. This allows for efficient test management, especially when the test consists of multiple nodes.

Use the following command template to send commands on the discovered peers:

• remote [peer_num] [cmd] – Send command cmd to peer peer_num.

Example:

remote 0 test info

Setup

You can find the source code and the makefile in the following folders:

• <InstallFolder>/examples/multiprotocol/benchmark/{thread,zigbee}
• <InstallFolder>/components/softdevice/s140

Configuring the boards (Thread-only)

1. Prepare two nRF52840 Development Kits.
2. Flash the SoftDevice to both boards according to the instructions in Running examples.
3. Program the example by using one of the following options:
   • Default instructions:
     1. Build and flash the example according to the instructions in Building examples.
   • Precompiled hex file:
     1. Run the following commands on both boards to flash the application:

        $ cd <InstallFolder>/examples/multiprotocol/benchmark/thread/hex
        $ nrfjprog -f NRF52 --sectorerase --program nrf52840_xxaa_s140_pca10056.hex

4. Reset the boards.
5. Start a terminal emulator like PuTTY and connect to the used COM port with the following UART settings:
   • Baud rate: 115200
   • 8 data bits
   • 1 stop bit
   • No parity
   • HW flow control: RTS/CTS
6. Run the following commands from the CLI to start Thread stack:

   ot ifconfig up
   ot thread start

You can now start Running benchmark tests using the Thread protocol.

You can enable autostart of Thread network by changing .autostart_disable from true to false in thread_protocol_api.c.

Configuring the boards (Zigbee-only)

1. Prepare two nRF52840 Development Kits.
2. Flash the SoftDevice to both boards according to the instructions in Running examples.
3. Program the example by using one of the following options:
   • Default instructions:
     1. Build and flash the example according to the instructions in Building examples.
   • Precompiled hex file:
     1. Run the following commands on both boards to flash the application:

        $ cd <InstallFolder>/examples/multiprotocol/benchmark/zigbee/hex
        $ nrfjprog -f NRF52 --sectorerase --program nrf52840_xxaa_s140_pca10056.hex

4. Reset the boards.
5. Start a terminal emulator like PuTTY and connect to the used COM port with the following UART settings:
   • Baud rate: 115200
   • 8 data bits
   • 1 stop bit
   • No parity
   • HW flow control: RTS/CTS
6. Choose one of the boards and configure it as a Zigbee coordinator:

   bdb role zc

7. Run the following command from the CLI coordinator node to start a new Zigbee network:

   bdb start

8. Connect the second board to the newly created network:

   bdb start

You can now start Running benchmark tests using the Zigbee protocol.

Running benchmark tests

1. Choose one of the boards and run the test info command.
2. Run the test peer discover command to connect to the second board.
3. Run the test start command and observe the logs.

You will be able to observe results similar to the following output:

    === Test Finished ===
Test duration: 72912ms

Average CPU utilization:
    Local: 20.32%
    Remote: 23.50%

Without retransmissions:
    PER: 0.00%
    Throughput: 43 kbps

With retransmissions:
    PER: 0.00%
    Throughput: 43 kbps

MAC PER: 0.12%

Raw data:
    Config:
        Length: 79
        ACK timeout: 200ms
        Count: 5000
        Mode: ACK
    Status:
        Test in progress: False
        Reset counters: False
        ACKs lost: 0
        Waiting for ACKs: 0
        Packets left count: 0
        Frame number: 5001
    Local:
        CPU utilization: 20.32%
        Duration: 72912ms
        App counters:
            Bytes received: 0B
            Packets received: 0
            RX error: 0
            RX total: 5006
        Mac counters:
            TX error: 5
            TX total: 5011
    Remote:
        CPU utilization: 23.50%
        Duration: 72917ms
        App counters:
            Bytes received: 405000B
            Packets received: 5000
            RX error: 0
            RX total: 5002
        Mac counters:
            TX error: 8
            TX total: 5014

Done