Written by Joel Stapleton, Technical Product Manager, Nordic Semiconductor, and Zephyr Governing Board Chair
In January 2020, NCC Group notified the Zephyr Project of a number of security issues found as part of their independent research into the security posture of Zephyr. NCC Group, a global expert in cyber security and risk mitigation, initiated independent research into the Zephyr RTOS in response to growing client interest in the Project. They noted that they found Zephyr to be a mature, and a highly active and growing project with increasing market share. The report, which came out in May 2020, outlines the issues discovered in detail and acknowledges the proactive work of the Security Committee to fix issues and follow-up on recommendations of the report. This blog aims to explain how the Zephyr Project offers an IoT solution, beyond an RTOS kernel and source code when it comes to securing end-products, and what the NCC Group’s report showed the Project and how it responded.
Since its launch in February 2016, the Zephyr Project has been steered by this vision:
“The Zephyr Project strives to deliver the best-in-class RTOS for connected, resource-constrained devices, built to be secure and safe.”
This vision was created to challenge the status-quo of proprietary or commercial kernels, and commercially-governed open projects that exist today. NCC Group’s report noted that their work to date has found IoT devices are typically built using a “hodgepodge” of chipset vendor board support packages, bootloaders, SDKs, and an RTOS kernel. In contrast, the Zephyr solution is unique as it is vendor-neutral, with a scope from multi-architecture board support packages, to cloud connectivity for IoT products. All source code is available and within the Project Continuous Integration (CI) framework. The Project brings together a community of experts to participate on all aspects of the solution, from the standards to adopt, policies and processes to follow, and methodologies for build, test, maintenance, distribution and incident response. The aim is to make a solution that developers can trust for the lifecycle of their products.
Zephyr has experienced significant growth in contributors, users, and the number of end-products made with Zephyr. The Zephyr RTOS is becoming established as a robust IoT solution for resource constrained devices where Linux is not an option. Product and service providers in the IoT space are increasingly working with Zephyr as its maturity and user base increases.
This was the case for NCC Group according to Technical Director Jeremy Boone: “NCC Group serves as a strategic security advisor to many companies that manufacture Internet-of-Things devices. Recently we have received an increasing number of queries regarding Zephyr. Our clients were primarily concerned with gaining an understanding of Zephyr’s overall security posture, and to better understand the factors that must be considered when designing a secure IoT device that is based on Zephyr. In order to better serve our clients, we decided to invest in a significant research effort to acquire a deep understanding of the Zephyr architecture.”
The Zephyr Project has always recognized the importance of the Security Posture of the solution which goes beyond the correctness of encryption implementations or the minimization of vulnerabilities. Security Posture incorporates the application of best practices for secure development and design, and the readiness and ability to notify and respond to security incidents.
The Project established a Security Committee and the role of Security Architect from the beginning. The Security committee have put in place the relevant policies and procedures followed by the project, and selected industry frameworks such as the Linux Foundation’s (LF) Core Infrastructure Initiative (CII) Best Practices badging program for which the project has maintained gold status over the last few years. The Zephyr Project is a CVE Numbering Authority (CNA) and adds vulnerabilities to the U.S. National Vulnerability Database (NVD) by being able to assign Common Vulnerability and Exposure (CVE) numbers and create notifications directly from the Project.
The first Zephyr Long Term Support (LTS) release of Zephyr (14.0) in April 2019 was the realization of Zephyr’s strategy to provide LTS releases for applications that require highly stable code bases with bug and security patch support for long periods. Most recently in 2020, Zephyr’s vulnerability and embargo policies were revised and a system put in place to allow product makers, not participating as a member, to receive Security Vulnerability Notifications at the start of embargo periods so risk mitigation and corrective measures can be taken by users for their customers.
So when NCC Group investigated the Zephyr RTOS and MCUBoot bootloader, what happened and how did we do?
NCC Group informed the Zephyr Project of their research findings and intention to publish a report detailing vulnerabilities that were discovered. An embargo period was agreed and the Security Response process within Zephyr came into action. NCC Group reported 26 issues to Zephyr. Within those 26 issues, 2 were considered critical vulnerabilities, 2 high risk vulnerabilities, with the remainder medium, low or informational. 1 low risk issue was reported for MCUBoot, although it was noted that no issues that could undermine the secure boot function were found.
The first step the Security Response team took after the issues were reported was to capture the information in the Zephyr Project bug tracking Jira system to begin the issue management lifecycle, limiting access to information on a need-to-know basis. The team then triaged the issues to self-assess the severity and set a priority. At this time, CVE numbers were created (without publishing details) for all critical, high and medium risk issues.
At that time, Zephyr did not have a registry for notification of vulnerabilities to users who are not members represented in the response team. That registry is now in place giving project members and registered product makers need-to-know access to information when CVE numbers are created so those affected can take action.
Following CVE number creation, issues were assigned for fixes. Zephyr has Maintainer roles for major sub-systems which may be assigned and/or delegated to those with the deepest knowledge of the affected code. Once fixes were proposed, they went through a key step in the life cycle of all Zephyr contributions – the Pull Request (PR) & Review. This step allowed peers to assess the fix for quality and correctness. Once approved, the PRs were merged resulting in code being changed in Zephyr to resolve the issues for the next Zephyr release, v2.2.
8 issues in total were prioritized by the Project and fixed in time for v2.2 of Zephyr (March 2020) which was already in the release process at the time. The issues were still under embargo, so only those with a need-to-know had a link between the vulnerability and the PR for each fix.
While a further 5 lower priority issues were resolved in the Zephyr Master branch in preparation for Zephyr v2.3, the security committee further assessed the fixes that should be back-ported to Zephyr v2.2, and the LTS release (v1.14) to maintain those releases for users. 4 fixes were back-ported to create v2.2.1 in May 2020, and 7 fixes for LTS release to create 1.14.2 in April 2020 (Note: some issues were not affecting this release).
On May 26, the embargo was lifted and NCC Group published the report. CVE entries were updated with details of the vulnerabilities.
The response effort involved people from many of our Project members, and individual contributors, to collaborate to manage the issues. NCC Group was able to observe the response of the project prior to publication, and the Zephyr Project is pleased to direct our community to their findings.
It is an old adage that no software is bug free. Likewise, no software is completely secure or vulnerability free. The pursuit of securing an IoT solution is to reduce the risk of vulnerability inclusion by explicitly planning for secure design, security hardening of critical attack surfaces, and ensuring an issue management system is in place, including notification of findings so users are able to take mitigating actions early, and stay ahead of attackers. The work of NCC Group and our community has resulted in security vulnerabilities being fixed, an evaluation of our processes, and measurement of our response capabilities to make Zephyr better today than ever.
Many thanks to the work done by NCC Group; and many thanks to the Zephyr community that have worked to strengthen Zephyr and responded to these issues for the benefit of all users.
For more details about Zephyr’s security initiatives, please visit https://www.zephyrproject.org/security/. Or, feel free to join the Zephyr Project Slack and ask questions and participate in the discussions.
