Managing Multiple Versions of the Zephyr RTOS

Written by Mohammed Billoo, Embedded Linux Consultant, MAB Labs Embedded Solutions

Motivation

Embedded software engineers typically use a single variant of their toolchain and the underlying Real-Time Operating System (RTOS) in the application running on their IoT device. However, as updates to the toolchain and RTOS are pushed out, it is beneficial for engineers to stay up to date with these releases. They can quickly identify regressions in their application due to these updates and address them immediately. If engineers don’t stay up to date with these updates, the alternative is usually painful. Eventually, the application will require a feature or major bug fix only available in a recent version of the toolchain or RTOS. Since there haven’t been regular updates to the application with releases of the toolchain or RTOS, the process will be costly and lengthy. Thus, engineers (and their respective managers) should stay regularly updated with the latest toolchain and underlying RTOS driving their application. Unfortunately, due to business constraints, it may not be possible for engineers to stay up-to-date with the toolchain and RTOS on a regular basis. 

In that case, two issues usually manifest when there is no choice but to perform a major update. The first is compile-time bugs due to differences in the underlying API calls. These are usually straightforward to resolve. The second is run-time bugs due to logical differences in a particular subsystem. These are more difficult to solve. One strategy I employ to help resolve run-time issues due to a major update of the RTOS is to perform a “bisection.”  This involves updating the RTOS from the starting version to the latest version in roughly half increments and determining where the regression first manifested.

Setting Up Multiple Versions of Zephyr

While this process may be straightforward, it requires extra steps when working on a project using The Zephyr Project RTOS. This is because the Zephyr RTOS has strict versioning requirements on the “Zephyr SDK” (https://docs.zephyrproject.org/latest/develop/toolchains/zephyr_sdk.html), which is the toolchain used to compile the application and RTOS source code. In this blog post, I’ll demonstrate a process that worked for me on a recent project where I needed to migrate from Zephyr 2.7 to Zephyr 3.2. This wasn’t too bad of a migration since there are only three releases between 2.7 and 3.2 (specifically, 3.0 to 3.2). Thus, in this case, I started by jumping directly to Zephyr 3.2.

In previous blog posts, I have demonstrated using the nRF Connect SDK (https://www.nordicsemi.com/Products/Development-software/nrf-connect-sdk) in Windows when getting started with The Zephyr Project RTOS. However, in this blog post, we will use Linux and the underlying West tools since moving across different versions of Zephyr becomes easier. 

First, we can create a checkout for Zephyr 2.7 using the following invocation for West:

$> west init -m https://github.com/zephyrproject-rtos/zephyr --mr v2.7.0 zephyr_2.7 
$> cd zephyr_2.7 
$> west update

We can also create a similar checkout for Zephyr 3.2:

$> west init -m https://github.com/zephyrproject-rtos/zephyr --mr v3.2.0 zephyr_3.2
$> cd zephyr_3.2
$> west update

Setting up Multiple Versions of the Zephyr SDK

It is not sufficient to only have the Zephyr codebase. We will also need the SDK to cross-compile the codebase to run on the final device. We will need to retrieve the SDK that was used at the time when Zephyr 2.7 was used, which happened to be version 0.14.0. The SDK can be installed by executing the following commands:

$> wget https://github.com/zephyrproject-rtos/sdk-ng/releases/download/v0.14.0/zephyr-sdk-0.14.0_linux-x86_64.tar.gz
$> tar xzvf zephyr-sdk-0.14.0_linux-x86_64.tar.gz
$> cd zephyr-sdk-0.14.0_linux-x86_64
$> ./setup.sh

Ensure to enter ‘y’ when prompted to “Register Zephyr SDK CMake package.” We have reproduced the original development environment with Zephyr 2.7 and Zephyr SDK 0.14.0. Since Zephyr SDK 0.14.0 is not compatible with Zephyr 3.2, well also have to download and install the latest Zephyr SDK (which was 0.16.1 at the time of this blog post). We can follow the same steps above and replace “0.14.0” with “0.16.1”:

$> wget https://github.com/zephyrproject-rtos/sdk-ng/releases/download/v0.16.1/zephyr-sdk-0.16.1_linux-x86_64.tar.gz
$> tar xzvf zephyr-sdk-0.16.1_linux-x86_64.tar.gz
$> cd zephyr-sdk-0.16.1_linux-x86_64
$> ./setup.sh

Again, remember to enter ‘y’ when prompted to “Register Zephyr SDK CMake package.”

Switching Between Zephyr Versions

We can emulate a sample application based on Zephyr 2.7 by copying the hello_world sample:

$> cp -r zephyr_2.7/zephyr/samples/hello_world ./app

And we can instruct West to use Zephyr 2.7 by changing into the “zephyr_2.7” directory and configuring the environment for Zephyr 2.7:

$> cd zephyr_2.7/zephyr
$> source ./zephyr-env.sh

We can confirm that Zephyr 2.7 is selected by building the sample application and reviewing the first lines of the output (which are highlighted in red below):

$> west build -p always -b nrf52840dk_nrf52840 app
-- west build: making build dir /home/mab/zephyr-test/build pristine
-- west build: generating a build system
Including boilerplate (Zephyr base): /home/mab/zephyr-test/zephyr_2.7/zephyr/cmake/app/boilerplate.cmake
-- Application: /home/mab/zephyr-test/app
-- Zephyr version: 2.7.0 (/home/mab/zephyr-test/zephyr_2.7/zephyr), build: zephyr-v2.7.0
-- Found Python3: /usr/bin/python3.10 (found suitable exact version "3.10.12") found components: Interpreter 
-- Found west (found suitable version "1.1.0", minimum required is "0.7.1")
-- Board: nrf52840dk_nrf52840
-- Cache files will be written to: /home/mab/.cache/zephyr
-- ZEPHYR_TOOLCHAIN_VARIANT not set, trying to locate Zephyr SDK
-- Using toolchain: zephyr 0.16.1 (/home/mab/zephyr-toolchains/zephyr-sdk-0.16.1)

However, we can also see that West used the newer toolchain (also highlighted in red above), which is not what we wanted. We can rectify this by setting the “ZEPHYR_SDK_INSTALL_DIR” environment variable to the installation directory of Zephyr SDK 0.14.0. I use the “pwd” command in a bash terminal along with backticks (the symbol next to the ~ on my keyboard) as an easy way to do this as shown below:

$> cd ~/zephyr-toolchains/zephyr-sdk-0.14.0
$> export ZEPHYR_SDK_INSTALL_DIR=`pwd`

If we build our application again, we can see now that West is using the desired toolchain:

$> west build -p always -b nrf52840dk_nrf52840 app
-- west build: generating a build system
Including boilerplate (Zephyr base): /home/mab/zephyr-test/zephyr_2.7/zephyr/cmake/app/boilerplate.cmake
-- Application: /home/mab/zephyr-test/app
-- Zephyr version: 2.7.0 (/home/mab/zephyr-test/zephyr_2.7/zephyr), build: zephyr-v2.7.0
-- Found Python3: /usr/bin/python3.10 (found suitable exact version "3.10.12") found components: Interpreter 
-- Found west (found suitable version "1.1.0", minimum required is "0.7.1")
-- Board: nrf52840dk_nrf52840
-- Cache files will be written to: /home/mab/.cache/zephyr
-- Using toolchain: zephyr 0.14.0 (/home/mab/zephyr-toolchains/zephyr-sdk-0.14.0)

If we wish to use Zephyr 3.2 now, we simply need to set up the environment for Zephyr 3.2:

$> cd zephyr_3.2/zephyr
$> source ./zephyr-env.sh

We can confirm that West is using Zephyr 3.2 by looking at the first few lines of the output:

$> west build -p always -b nrf52840dk_nrf52840 app
-- west build: making build dir /home/mab/zephyr-test/build pristine
-- west build: generating a build system
Loading Zephyr default modules (Zephyr base).
-- Application: /home/mab/zephyr-west/app
-- Found Python3: /usr/bin/python3.10 (found suitable exact version "3.10.12") found components: Interpreter 
-- Cache files will be written to: /home/mab/.cache/zephyr
-- Zephyr version: 3.2.0 (/home/mab/zephyr-west/zephyr_3.2/zephyr)
-- Found west (found suitable version "1.1.0", minimum required is "0.7.1")
-- Board: nrf52840dk_nrf52840
-- Found host-tools: zephyr 0.16.1 (/home/mab/zephyr-toolchains/zephyr-sdk-0.16.1)
-- Found toolchain: zephyr 0.16.1 (/home/mab/zephyr-toolchains/zephyr-sdk-0.16.1)

However, we see above that Zephyr SDK 0.16.1 even though we set the “ZEPHYR_SDK_INSTALL_DIR” environment variable to point to Zephyr SDK 0.14.0. This is because newer versions of Zephyr have intelligence built in to use only compatible SDK versions. We can test this by removing the CMake registration of Zephyr SDK 0.16.1 and rebuilding our application. The CMake registrations are located under ~/.cmake/packages/Zephyr-sdk. This directory contains what look to be hashes (which reference each Zephyr SDK version):

$> ls ~/.cmake/packages/Zephyr-sdk/
4a74ba8c64ac4f47d90dfa3cfab63530  cb7c790e9983325865bce4c513a0fa1f

Since it’s difficult for me to remember hashes, I usually remove all of the entries in the directory and simply re-run the setup script only for Zephyr SDK 0.14.0:

$> rm ~/.cmake/packages/Zephyr-sdk/*
$> cd ~/zephyr-toolchains/zephyr-sdk-0.14.0/
$> ./setup.sh

If we now try to build our application against Zephyr 3.2, we can observe the intelligence contained in recent versions of Zephyr to ensure that a compatible SDK exists:

$> west build -p always -b nrf52840dk_nrf52840 app
-- west build: making build dir /home/mab/zephyr-test/build pristine
-- west build: generating a build system
Loading Zephyr default modules (Zephyr base).
-- Application: /home/mab/zephyr-west/app
-- Found Python3: /usr/bin/python3.10 (found suitable exact version "3.10.12") found components: Interpreter 
-- Cache files will be written to: /home/mab/.cache/zephyr
-- Zephyr version: 3.2.0 (/home/mab/zephyr-west/zephyr_3.2/zephyr)
-- Found west (found suitable version "1.1.0", minimum required is "0.7.1")
-- Board: nrf52840dk_nrf52840
CMake Error at /home/mab/zephyr-test/zephyr_3.2/zephyr/cmake/modules/FindZephyr-sdk.cmake:56 (find_package):
  Could not find a configuration file for package "Zephyr-sdk" that is
  compatible with requested version "0.15".

As shown previously, we can rectify this by simply re-installing Zephyr SDK 0.16.1. 

Summary and Next Steps

In this blog post, we showed how multiple versions of Zephyr and SDK can be maintained on the same development machine. However, if we try to build the application originally based on Zephyr 2.7 against Zephyr 3.2 we can see the compile time issues that were mentioned at the beginning:

$> west build -p always -b nrf52840dk_nrf52840 app
.
.
.
/home/mab/zephyr-test/app/src/main.c:7:10: fatal error: zephyr.h: No such file or directory
    7 | #include <zephyr.h>
      |          ^~~~~~~~~~

The error above exists because the location of the header file changed in Zephyr 3.2. We would have to go through the source files and address all compile-time and run-time issues.

In the next blog post, I will demonstrate a personal best practice to account for these types of compile-time errors (and even run-time bugs). Ultimately, a single version of Zephyr has to be used to compile the application when released. But, it can be valuable to quickly switch between Zephyr versions during development.