The Embedded Open Source Summit took place from April 16-18 in Seattle, Washington, alongside the Open Source Summit North America. The Zephyr Developer Summit was part of the EOSS, aimed at developers using or considering Zephyr in embedded products. This year, we concentrated on supporting topics relevant to Zephyr users, upstream contributors, and maintainers.
More than 860 individuals from 721 organizations across 46 countries registered for the EOSS conference. The Zephyr track featured over 50 technical sessions, both in-person and on-demand, focusing on users, upstream developers, and maintainers.
Videos from the Zephyr Developer Summit are available on the Zephyr Project YouTube channel. We will highlight a few videos each week in a blog for easy access.
Today we are featuring:
- Ticket Spinlocks in Zephyr RTOS: What, Where & Why – Evgenii Paltsev, Synopsys
- Unreasonably Efficient AR: Using Zephyr to Achieve More with Simple Hardware – Aedan Cullen, Independent
- Userspace in Zephyr: Overview and Practicality – Daniel Leung, Intel Corporation
- Using Picolibc in Embedded Systems – Keith Packard, Amazon
Ticket Spinlocks in Zephyr RTOS: What, Where & Why – Evgenii Paltsev, Synopsys
At the Zephyr Developer Summit, Evgenii Paltsev from Synopsys introduced a key development for improving the performance and fairness of multicore systems in Zephyr RTOS: ticket spinlocks. As Zephyr RTOS has grown in adoption, particularly for Symmetric Multi-Processing (SMP) systems, its original spinlock implementation began to show limitations, particularly around fairness and core access to shared resources.
Traditional spinlocks in Zephyr, implemented since its early SMP support in version 1.11, rely on a single atomic variable that acts as a lock flag. While simple and efficient, this approach does not guarantee fairness. In heavy multicore systems, certain cores could dominate the lock, leading to other cores being repeatedly denied access. This issue, called lock starvation, was especially problematic during stress tests on new multicore processors with up to 12 cores. The lack of fairness resulted in inconsistent behavior, system delays, and failed stress tests.
To address this issue, Zephyr adopted ticket spinlocks, a concept first implemented in the Linux kernel. Ticket spinlocks solve the fairness problem by assigning a “ticket” to each core requesting a lock. Cores are served in order, ensuring that every core gets a fair chance to acquire the lock. This eliminates the possibility of any core being starved of access.
Ticket spinlocks operate with two atomic counters: a ticket counter and an owner counter. When a core requests a spinlock, it increments the ticket counter and waits until its ticket matches the owner counter. Once the current owner releases the lock, the next core in line (with the next ticket number) can acquire the lock. This guarantees a first-come, first-served order for lock acquisition, resolving the fairness issues inherent in traditional spinlocks.
Evgenii presented results from testing ticket spinlocks in both simulated environments and on real hardware. On simulators like Synopsys NSim and real hardware such as a quad-core RISC-V board, ticket spinlocks showed marked improvements in fairness. Unlike the original spinlocks, ticket spinlocks evenly distributed lock access across all cores, preventing any core from being repeatedly blocked.
However, when running in more complex environments, such as virtualized systems or systems under heavy load, some challenges remained. For instance, in highly virtualized setups, the scheduling of guest OS threads on the host CPU can still introduce delays. This is due to external factors like task scheduling on the host, rather than an issue with the ticket spinlock algorithm itself.
While ticket spinlocks offer clear advantages in terms of fairness, there are some potential downsides. The new implementation introduces extra overhead from additional atomic operations and memory barriers. Although the performance impact is minor, it may vary depending on the architecture and workload.
Moreover, there are certain edge cases, such as when Zephyr is running in a virtualized environment or on hardware without cache coherence, where ticket spinlocks may not fully solve all issues. In such cases, hardware-specific solutions like hardware-managed spinlocks might be required.
Despite these challenges, ticket spinlocks have already been enabled for some platforms in Zephyr and are being actively tested. Over time, as these spinlocks mature and are tested across more platforms, they may become the default synchronization mechanism in Zephyr RTOS.
Ticket spinlocks represent a significant improvement in handling synchronization across multicore systems in Zephyr RTOS. By addressing the fairness issues of traditional spinlocks, they offer a more reliable and predictable mechanism for managing lock access, ensuring that no core is starved of resources. While still experimental, ticket spinlocks are a promising step forward and are likely to play a key role in the future of Zephyr’s multicore performance.
Unreasonably Efficient AR: Using Zephyr to Achieve More with Simple Hardware – Aedan Cullen, Independent
At the Zephyr Developer Summit, Aedan Cullen presented his ongoing work on building low-power augmented reality (AR) devices using Zephyr RTOS, showcasing how efficient software can push simple hardware beyond its traditional limits. His project focuses on developing lightweight, wearable AR devices with near-eye displays, challenging the norm of relying on powerful multicore systems running Linux for multimedia tasks.
Aedan’s design leverages the NXP i.MX RT500 crossover microcontroller, integrating sensors, a high-density OLED microdisplay, and Wi-Fi connectivity. By using Zephyr, Aedan was able to streamline development and minimize power consumption while maintaining functionality. Zephyr’s build system and extensive driver library provided an efficient environment for prototyping, while community support enabled fast progress despite hardware and software challenges.
Aedan highlighted the challenges of working with niche hardware like OLED microdisplays, the intricacies of configuring peripherals, and the benefits of using specialized hardware features such as the Tensilica Fusion DSP for offloading processing tasks. His experience shows the potential of RTOS-based systems to support multimedia applications, typically reserved for more complex platforms, inspiring others to explore ambitious hardware projects with limited resources.
Userspace in Zephyr: Overview and Practicality – Daniel Leung, Intel Corporation
At the Zephyr Developer Summit, Daniel Leung from Intel presented an insightful talk on userspace in Zephyr RTOS. The discussion covered the core concepts, advantages, and considerations for using userspace in Zephyr, along with its limitations and practical applications.
Userspace in Zephyr is a method of segregating user-level code from kernel-level functions, ensuring that user threads operate with restricted permissions to safeguard the system. This approach limits access to critical system resources, providing an extra layer of security and error prevention. Key concepts such as memory domains, memory partitions, and kernel objects were discussed, with Leung explaining how these mechanisms help manage data sharing and isolation between user threads.
The session also delved into system calls (syscalls), which allow user threads to access kernel services and hardware, detailing the process and overhead involved. Daniel emphasized that using userspace adds complexity in terms of memory management and syscall execution, but offers benefits such as enhanced system robustness and fault isolation.
Despite the benefits, userspace is not suited for all systems. Hardware must support memory protection units (MPU) or memory management units (MMU) for userspace to function. Additionally, the performance overhead and increased memory requirements might not make sense for simple or resource-constrained applications.
In conclusion, userspace in Zephyr provides a robust framework for isolating code and protecting critical system functions, making it an attractive option for complex, secure, or multi-threaded applications.
Using Picolibc in Embedded Systems – Keith Packard, Amazon
At the Zephyr Developer Summit, Keith Packard from Amazon presented a deep dive into Picolibc, a C library tailored for embedded systems. Picolibc is designed for low-memory environments, offering full C17 and POSIX 2008 compatibility while ensuring minimal memory usage. Its lightweight nature makes it ideal for safety-critical applications such as rocketry and satellite systems where dynamic memory allocation (like malloc) is avoided to prevent unexpected behavior.
The talk provided an overview of Picolibc’s design goals, including standards conformance, robust architecture support, and an emphasis on minimizing footprint. Keith highlighted the importance of eliminating hidden memory allocations and ensuring that the library behaves predictably, especially in space-bound and other critical systems where failures are not an option.
Keith also discussed the testing infrastructure used to validate Picolibc on various platforms, including bare-metal test frameworks and emulators. He explained how Picolibc has been integrated into multiple real-time operating systems (RTOS) like Zephyr, FreeRTOS, and RIOT, demonstrating its versatility and ease of use across different environments.
As the default C library for Zephyr, Picolibc provides efficient memory management options, including various printf and scanf levels that can be configured to suit specific application needs. While the library is standards-compliant and optimized for small systems, it also supports features like advanced floating-point operations and extended character set handling.
Keith concluded with insights into the future of Picolibc, including potential enhancements to its math library and ongoing efforts to ensure robust functionality through extensive testing. With its focus on minimal footprint, high performance, and comprehensive standards compliance, Picolibc is an excellent choice for developers working on embedded systems with limited resources.
Watch the rest of the Zephyr Developer Summit videos here. The schedule and links to the slides can be found here. Photos from the EOSS can be found here.
For more information about the 2024 event, stay tuned by subscribing to the Zephyr quarterly newsletter or connect with us on @ZephyrIoT, Zephyr Project LinkedIn or the Zephyr Discord Channel to talk with community and TSC members.