Building with Zephyr RTOS: Remote Processing, SoC Design, and Production Challenges

The Embedded Open Source Summit took place from April 16-18 in Seattle, Washington, alongside the Open Source Summit North America. The Zephyr Developer Summit was part of the EOSS, aimed at developers using or considering Zephyr in embedded products. This year, we concentrated on supporting topics relevant to Zephyr users, upstream contributors, and maintainers.

More than 860 individuals from 721 organizations across 46 countries registered for the EOSS conference. The Zephyr track featured over 50 technical sessions, both in-person and on-demand, focusing on users, upstream developers, and maintainers.

Videos from the Zephyr Developer Summit are available on the Zephyr Project YouTube channel. We will highlight a few videos each week in a blog for easy access.

Today we are featuring:

Remote Processor Framework in Zephyr RTOS – Vaishnav Mohandas Achath, Texas Instruments India Pvt. Ltd.

As the demand for collaborative systems in IoT and Edge AI grows, heterogeneous multicore processors (such as ARMv8, ARMv7 R/M cores), DSPs, and hardware accelerators are becoming integral to practical use cases. This talk, delivered by Vaishnav Mohandas Achath of TI, explores the proposal for a Remote Processor subsystem in the Zephyr RTOS, addressing the complexities of distributed and heterogeneous computing in embedded systems.

Current approaches to managing remote cores in systems with multiple processors often rely on external systems such as Linux or JTAG, limiting efficiency and startup times. Introducing a Remote Processor framework into Zephyr would streamline the process, allowing Zephyr to manage the powering on, firmware loading, and powering off of remote processors directly. This would enable faster system startup and offer new possibilities for Zephyr’s deployment in production environments with heterogeneous multicore architectures.

The presentation covers the motivations behind creating this framework, existing methods for managing remote processors in Linux and U-Boot, and how these solutions fall short in certain system-level use cases. It delves into the implementation of both host and client modes for the remote processor framework in Zephyr, drawing inspiration from the mature frameworks in Linux. Vaishnav provides technical details of functions like loading, starting, stopping, and preparing remote processors, along with the challenges posed by cross-architecture systems.

A key part of the proposal involves improving Zephyr’s client-mode capabilities, including enhancing support for graceful shutdowns and memory negotiations between host and client processors. The session also includes a demonstration of the remote processor framework running on a Texas Instruments AM62x platform, showcasing the loading and execution of remote processors using Zephyr as the host.

By introducing this framework, Zephyr could offer a more efficient, lightweight alternative to existing solutions like U-Boot, providing a flexible approach to power management, real-time operation, and heterogeneous system startup in embedded applications. This proposal is a work-in-progress aimed at gathering community feedback before the development of a formal RFC and submission process. The ultimate goal is to enable seamless interaction between diverse cores within a system and expand Zephyr’s use in distributed, multi-core processing environments.

SoC Development: From ROM to Application – Nadav Cohen Zukerman, Autotalks

This talk outlines the two-year journey of developing a System on Chip (SoC) from scratch using Zephyr RTOS. Nadav Cohen Zukerman of Autotalks shares insights into the development process, challenges encountered, and lessons learned during the creation of a next-generation V2X (Vehicle-to-Everything) SoC, spanning two tape outs, two SoC bringups, and the development of additional features.

The SoC boot flow, which uses Zephyr from the ROM bootloader to the fully operational application, is a core focus of the presentation. The talk explains the design and implementation of the boot stages, highlighting how Zephyr supports not just the application layer, but also early boot processes. The decision to use Zephyr throughout the boot flow allowed for modular design, reuse of drivers, and simplified image-loading functionality. Challenges such as implementing secure boot, managing various memory architectures, and optimizing for performance are discussed in detail.

Pre-silicon development was a critical phase, using hardware emulators, simulators, and software simulators to overcome the lack of physical hardware. Debugging tools like digital wave analysis were used to detect hardware and software bugs. Specific examples of issues, such as cache invalidation bugs and challenges with memory configuration, are highlighted, including how collaboration with the Zephyr community and Synopsis support helped resolve them.

The SoC includes features such as a secure boot flow, high-performance memory utilization, and an architecture designed for high throughput in V2X use cases. The talk concludes with a summary of the SoC’s tapeouts, upcoming deployments, and Zephyr’s integral role in this life-saving technology.

This session provides valuable insights for developers working on complex SoCs, illustrating how Zephyr can be used across the entire development process – from boot to application – within a high-performance, real-time environment.

Why NXP is Doubling Down on Zephyr Project – Brendon Slade, NXP

In this talk, Brendon Slade from NXP explores the company’s deepening investment in the Zephyr Project and the reasons behind it. NXP has been involved with Zephyr since its inception in 2016, but recent market trends, including the rise of open source software in the microcontroller space, have driven the company to expand its commitment. As the demand for smart, connected, secure, and efficient devices grows, Zephyr’s alignment with NXP’s strategic goals in security, safety, and connectivity has made it a key platform for NXP’s future developments.

Brendon discusses the challenges faced by companies during the COVID-19 pandemic, especially around supply chain disruptions, and how Zephyr’s flexibility helps mitigate such risks. Customers now seek a unified codebase that can be easily reconfigured across a range of devices, from low-end microcontrollers to high-performance processors, without requiring extensive rework.

NXP’s broad product portfolio, including low-power MCUs and high-performance crossover processors like the i.MX RT series, benefits from Zephyr’s adaptability. Brendon highlights customer use cases where Zephyr enables seamless transitions between devices and simplifies product development. Additionally, Zephyr’s open source nature, long-term support, and robust security features make it an attractive choice as companies face increasing regulatory pressures.

Moving forward, NXP is making Zephyr enablement mainstream across its product lines, ensuring new devices are Zephyr-ready from launch. The company is also focusing on improving the developer experience through tools like VS Code extensions and enhanced debugging support. With Zephyr, NXP aims to provide flexible, scalable solutions that help customers accelerate their time-to-market while maintaining the security and reliability required in today’s connected world.

Taking Your Hardware to Production with Zephyr RTOS – Chris Gammell, Golioth

In this talk, Chris Gammell from Golioth delves into the journey of taking hardware from initial prototypes to full-scale production using the Zephyr RTOS. He emphasizes that while bench development with development boards is a crucial starting point, transitioning to a final product ready for mass production introduces a new set of challenges. The talk explores how Zephyr’s features and ecosystem equip developers to build durable and scalable products efficiently.

Chris outlines the various stages of product development, using Golioth’s mascot, Echo, to represent each phase:

Early Prototyping on Development Boards: Validating the overall idea by leveraging Zephyr’s extensive support for various boards and sensors. Utilizing board files, Kconfig, and device tree overlays to customize development boards. Employing tools like Menuconfig for configuration management and Zephyr’s shell for interacting with sensors directly. This stage focuses on rapid iteration and validation of core functionalities.
First Pilot Production (10 Units): Transitioning to custom hardware to validate specific chipsets and peripherals intended for the final product. Introducing debugging tools like RTT and Ozone for better visibility as hardware becomes more constrained. Implementing MCUboot for secure bootloading and considering firmware updates from the outset. Managing board revisions within Zephyr to handle hardware changes across different versions.
Scaling Up Production (100 Units): Preparing for larger production runs by setting up test infrastructure and provisioning processes. Utilizing Zephyr’s testing tools like Twister, PTT, and device simulation to automate testing and ensure reliability. Incorporating secure provisioning methods, such as loading certificates at the factory, to enhance security and streamline device activation.
Optimizing for Mass Production (1,000+ Units): Addressing challenges associated with mass production, such as cost optimization, power management, and security enhancements. Utilizing Zephyr’s power management subsystem and pin control system to reduce power consumption and extend battery life. Implementing security features like custom keys in MCUboot and leveraging public key infrastructure (PKI) to secure firmware updates and device communications. Optimizing data usage by employing efficient serialization protocols like CBOR and dynamically adjusting log levels to reduce operational costs.
Maintenance and Scaling in the Field (10,000+ Units): Managing devices post-deployment, especially when dealing with large fleets. Leveraging Zephyr’s settings subsystem for remote configuration and updates, and utilizing remote procedure calls to control devices over the air. Monitoring device health using watchdogs and crash dumps, and employing exception reporting to track and address issues at scale. Extending device capabilities through dynamic updates, such as pushing new machine learning models or firmware extensions using LLVM.

Key Takeaways:

Zephyr Ecosystem Advantages: Offers extensive support for various hardware platforms and peripherals, enabling rapid prototyping and flexibility in design choices.
Development Tools: Tools like Menuconfig, RTT, Ozone, and Zephyr’s shell facilitate efficient development, debugging, and configuration management.
Production Readiness: Features like MCUboot and board revision management prepare devices for secure bootloading and handle hardware changes smoothly.
Testing and Automation: Zephyr’s testing tools ensure reliability and reduce manual effort, which is critical when scaling up production.
Power Management and Security: Built-in subsystems help optimize power usage and enhance security, essential for battery-powered and connected devices.
Fleet Management: Zephyr’s settings subsystem and remote procedure calls aid in managing devices at scale, allowing for updates and monitoring in the field.
Ongoing Maintenance: Emphasizes the need for automation, exception reporting, and extensibility to maintain device fleets effectively over time.
By integrating these practices and leveraging Zephyr’s robust ecosystem, developers can bridge the gap between prototype and production, ensuring their products are ready for the challenges of the real world

Watch the rest of the Zephyr Developer Summit videos here. The schedule and links to the slides can be found here. Photos from the EOSS can be found here.

For more information about the 2024 event, stay tuned by subscribing to the Zephyr quarterly newsletter or connect with us on @ZephyrIoT, Zephyr Project LinkedIn or the Zephyr Discord Channel to talk with community and TSC members.