This blog was written by Mike Szczys of Golioth and was originally published in the blog section of the Golioth website.
Here is an abstract of the blog “Zephyr: What modules should you add to a manifest allow list?“
Zephyr provides a large set of modules, but not every module is necessary for your project. Knowing which specific modules to include in your manifest allow list can save time and reduce unnecessary file cloning. While Zephyr’s west meta-tool and its package management system allow manifest files to inherit from other files, this can result in the cloning of unused packages.
Mike Szczys explains the process of checking your build directory to identify necessary modules, using a trial-and-error approach to fine-tune your manifest allow list.
Read the original blog below:
Zephyr: What modules should you add to a manifest allow list?
Zephyr has all of the bells and whistles. Your project only needs a handful of them. But which handful? To be fair, you can build with every possible module in your local tree and only the necessary bits will be pulled in. But wouldn’t it be nice to know exactly which modules need to be added to a manifest allow list? Answer that question and your users won’t be stuck cloning tons of unnecessary files. That could save time on each build, which really adds up over the course of a project’s life.
The west meta-tool used by Zephyr includes a package management system based on manifest files, often called west.yml. Part of the power of this system is that manifest files may inherit other manifest files. The downside to this is that you may be cloning a large number of packages your project will never use. Limit this by using an allow-list in your manifest. But what packages do you need to add to your allow list?
There is no answer to this question
Let’s be up-front about this: there is no definitive answer to this question.
Your project needs to allow all of the modules it uses. Sometimes that means modules that are enabled for some builds and disabled for others. For instance, the Golioth Firmware SDK includes example apps that will build for Espressif, Nordic, and NXP processors. Each have their own HAL but only one of them is used in any given build. You can’t really programmatically generate a modules list in a case like this; you just need to know these packages are needed, even if currently not in the build.
Even without an automated tool, I’ve had to answer this question for myself and I have some pointers on how to approach the problem.
The low-hanging fruit: check your build directory
The first thing you need to do is make sure your project builds without an allow list. That all files inherited from Zephyr or from NCS (Nordic’s Zephyr-based nRF Connect SDK) will be included from the build.
manifest: projects: - name: zephyr revision: v3.7.0 url: https://github.com/zephyrproject-rtos/zephyr west-commands: scripts/west-commands.yml import: true
This manifest will include dozens of modules available from the upstream Zephyr repository. There isn’t actually anything wrong with that. You clone the modules once and they live on your hard drive. But, it does take a long time to clone all of them and it will occupy several gigabytes of space. And it’s a good practice to know exactly which packages are actually in use. So let’s try to limit what is cloned in the future.
The build/modules directory from a Zephyr app
Above is a listing of the build/modules directory from a Zephyr application. All of these modules were scanned during the build process, but almost none of them have any object files that will be used in the build.
├── hal_rpi_pico │ ├── CMakeFiles │ └── cmake_install.cmake ├── hal_silabs │ ├── CMakeFiles │ └── cmake_install.cmake ├── hal_st │ ├── CMakeFiles │ └── cmake_install.cmake ├── hal_telink │ ├── CMakeFiles │ └── cmake_install.cmake
In fact, we can use this to help us find the modules that are actually at work in a project. Here’s a one-liner you can run from the build/modules directory to get a list of modules we know are needed for this build:
➜ find . -type f -not -name "cmake_install.cmake" | cut -d/ -f2 | uniq mbedtls golioth-firmware-sdk zcbor hal_nxp
Let’s add this these modules to an allow-list and move to the next step.
manifest:
projects:
- name: zephyr
revision: v3.7.0
url: https://github.com/zephyrproject-rtos/zephyr
west-commands: scripts/west-commands.yml
import:
name-allowlist:
- mbedtls
- zcbor
- hal_nxp
The trial-and-error step
Okay, the easy part is behind us. Now it’s time to figure things out the hard way. Begin by removing your module sources. These are usually in a modules directory that is a sibling of the zephyr directory where the Zephyr tree is stored. Check carefully that you do not have any uncommitted changes in these modules before removing them from your local storage. (I’ve learned this the hard way.)
Next, add an allow-list with the modules we found in the previous section. Run west update to clone the modules. This should happen rather quickly as we’ve greatly narrowed down what will be checked out. Try to build your application. If it fails, we need to divine which module was missing from the build and add that to the allow-list.
warning: HAS_CMSIS_CORE (defined at modules/cmsis/Kconfig:7) has direct dependencies 0 with value n, but is currently being y-selected by the following symbols: - CPU_CORTEX_M (defined at arch/arm/core/Kconfig:6), with value y, direct dependencies ARM (value: y), and select condition ARM (value: y)
Part the build error is pointing to a modules/cmsis. If you look in the west.yml from the Zephyr tree you’ll see there is indeed a module named cmsis. We can add to our allow list, run `west update`, and then rebuild.
Guess what? That was it… the project now builds! Here’s what my entire manifest looks like:
manifest:
projects:
- name: zephyr
revision: v3.7.0
url: https://github.com/zephyrproject-rtos/zephyr
west-commands: scripts/west-commands.yml
import:
name-allowlist:
- mbedtls
- zcbor
- hal_nxp
- cmsis
self:
path: modules/lib/golioth-firmware-sdk
west-commands: scripts/west-commands.yml
userdata:
patches_dirs:
- patches/west-zephyr
Note that the golioth-firrmware-sdk was one of the modules our search of the build directory turned up. But since that module is being added explicitly in this manifest file, it doesn’t need to be on the allow-list for the inherited Zephyr manifest.
Take control of your manifest with allow lists
Knowing exactly what libraries are being used in your build is part of good project management. Since manifest files let you target libraries and modules with version tags or commit hashes, this locks your project to a known-working state. I’m a huge advocate of this and gave an entire talk about Zephyr manifest files at the Embedded Open Source Summit.
Limiting your manifest files to libraries you are explicitly using helps you understand when upstream dependencies change. It may be a bit of a hassle to go through this process the first time, but doing so is a basic form of vetting your build and your product will be better for it.