Written by Kevin Townsend, a Zephyr contributor and Senior Embedded Engineer at Linaro
The Zephyr Scientific Library (zscilib) is an attempt to provide a set of functions useful for scientific computing, data analysis and data manipulation in the context of resource constrained embedded hardware devices.
It is written entirely in C, and while the main development target for the library is the Zephyr Project, it tries to be as portable as possible, and a standalone reference project is included to use this library in non-Zephyr-based projects.
The technical details of the library are easy enough to read looking at the repo. I thought the reason WHY the (admittedly niche!) library exists at all might make a more interesting or relatable story to read, though.
tl;dr
Zscilib is a story of going out for a loaf of bread, and coming back with the deed to the bakery. I’m still hungry, but … “Holy crap, look at all these toys and levers and possibilities! I can make bread forever, now! Once I learn how, anyway”. Something most engineers will understand, and most hapless bystanders will sigh in perpetual disbelieve at.
It was all going so well …
It was a given that my life goals didn’t involve writing a small scientific computing library for embedded devices.
I wasn’t a good math student. I affectionately like to refer to myself as a pretend-gineer. But I’m pragmatic, patient, willing to learn, and at one moment all I wanted in life was to know: “How can I take colour data from an RGB sensor, run it through a 3×3 constant correlation matrix to convert it to an XYZ tristimulus, do some other math, get an approximation of correlated colour temperature (CCT), and go home?” (Note: this was pre-Covid. Home is now officially overrated.)
For the curious: http://www.brucelindbloom.com/index.html?Eqn_RGB_to_XYZ.html
Sounds like what any engineer-type does every day of the week, right? Take a bunch of numbers, torture them a bit (basic multiplication here, strictly PG13!), get said results, claim requisite honour and glory, boom!
And yet, here I am, two years later, writing this blog post to try to explain what zscilib (unexpectedly) is, wondering how it all went happily wrong with eigenvectors, and QR decomposition, and Gram-Schmidt, and householder transformation, and why on God’s green earth do I even know what a Hessenberg matrix is!?!
… until it wasn’t
Pretend-gineer or not, I managed to put the code together to take my RGB vector, mistreat it just a bit with linear algebra, and I got my colour temperature value out. Eureka! I had what I needed, and I felt smug and smarter for my efforts. For a moment, anyway.
The problem was that comparing the output values to a known-good source, they sucked. They had a huge margin of error.
Naturally, my always-ready Finger-of-Judgement crooked itself back upon me (over-menacingly, I felt), causing me to believe that I had made an error in my calculations. That’s what being a pretend-gineer is all about, of course: perpetual self-doubt. That, and hoarding free snacks knowing they’ll figure it out eventually and finally fire you. Enjoy what you can while you can!
So, I went back, verified everything, used various online tools and calculators to compare my results with theirs, learned more than I should have about colour theory and various CIE standards and conversion algorithms, and finally, satisfied my math was good …
… the values still sucked.
In retrospect, the reason is predictable. A simple correlation matrix is great at correlating similar results, but pretty useless when you deviate from the conditions it was generated in. It just wasn’t large enough to be useful except in conditions overly similar to where it was captured. Widening the net by using samples from very different sources in a tiny 3×3 correlation matrix also leads to a huge margin of error, because it’s too broad. You just can’t win. The problem wasn’t the calculations, it was the approach.
I had my moment of enlightenment, I bent that Finger-of-Judgment back upon itself, and the solution seemed obvious. I just needed a better correlation matrix with more samples in it!
ProLifeTip: The solution is always MORE!
Alas, that was also the problem. There’s a reason every application note from every RGB sensor manufacturer tries to sell you on the RGB to XYZ via a 3×3 correlation matrix approach. The math is easy. A three element RGB vector multiplied by a 3×3 correlation matrix conveniently gives you a three element vector out. Perfect! Except, the results suck.
Multiplying your 3-element vector input by, say, a much larger 18×3 matrix is what you want, then, right?! Except, that doesn’t work since now you get 18 values out, and an XYZ tristimulus has `tri` in the name for a reason. And it’s not “keep trying”.
When the Going Gets Weird …
I was disappointed by all that effort for mediocre results, and to hit a dead-end, but that’s just how it works sometimes. Good engineering is more about persistence than brilliance for me, so I went back and did some more reading.
I learned a LOT more than I wanted to know about colour science, and darker corners of linear algebra, and all kinds of specialised vocabulary, but I kept stumbling across the same stale approaches. But I was learning, so it was fine.
The Eureka moment finally came in an obscure paper from a small university where four magic letters appeared at the bottom of one page : the authors described how they were correlating RGB inputs with a much larger constant correlation matrix, using something called the PINV.
Some people might be familiar with the pseudoinverse, or Moore-Penrose inverse, but I most certainly wasn’t.
Normally you can only calculate the inverse a square matrix, so you can’t just convert an 18×3 matrix to a 3×18 matrix. The pseudoinverse, however, allows you to do just that, and was the missing piece in this epic quest to go from RGB to CCT, with non-crummy results.
So, with the Sword of PINV firmly in hand, I could finally chop down to size the largest of correlation matrices!\
The problem is that the math is complex. And poorly explained. And I still kind of suck at math. And every example I could find for it (and there weren’t that many) was buried in impenetrable jumbles of code that were painful (for me) to gain any insight from.
But, I knew this was the answer. So I started reading some more, and googling, and poking and prodding, and harassing my smarter friends, and after finally giving up on finding something sane and in C and small enough to run on a tiny embedded processor with limited flash and SRAM, I decided to just write the damn thing myself. Which was the birth of zscilib as a project.
I needed structs and functions to handle vectors, and then matrices, and the ratsnest that is the pseudoinverse. That, woefully, leads you to eigenvectors, and eigenvalues, and QR decomposition, and Singular Value Decomposition (of early Google fame!), and lots of other things I’d happily avoided until then. AND, it all had to run on tiny, embedded processors with tens of KBs or memory, maybe a few hundred KBs of flash, and execution speeds often measured in tens of MHz. But I actually love the challenge of making things work on the smallest of processors, so it was nice itch to scratch, and personally satisfying.
I ended up hiring someone for the summer to help with some of the work (thanks Marti!), and finally had what I needed, and was pretty proud of the resulting vector and matrix code that could run on constrained embedded devices, since that wasn’t something that existed before in quite that way (to my knowledge), and seemed like a useful base for any number of things.
Alas, and after all that work (and now a bit of my own money!), even the mighty Sword of PINV was unable to pierce the RGB sensor’s Cloak of Invulnerability. It turns out, RGB sensors just suck as a means to determine colour temperature. The spectral response of the individual diodes are far too wide, and they overlap a lot with each other, and colour temperature is a function of spectral density, so to get a decent colour temperature approximation, you really need a larger number of narrow samples from the full spectral distribution of the light you’re measuring. It was never going to happen.
If that makes no sense, don’t worry about it. It just means … I wish I had known 6 months earlier that I was pursuing an inevitable dead end, but it is what it is, and you just need to accept sometimes that experimentation, even taken to the extreme, sometimes leads to failure, and that failure sometimes leads to new opportunities.
The End of the Beginning
At this point, zscilib was already born, though. I had written a fairy exhaustive set of functions to do increasingly obscure calculations, and tailored to embedded devices. I was pretty happy with it, and with the work Marti and me had done over one summer.
It was always a project purely for myself, but I published it on Github, made it public, and started a new job that meant I had to put it aside to move on to new problems.
I keep wanting to come back to it, though, and really miss it, and hope it can get some love and use from other people who have similar problems, since that was the point of publishing of sharing it.
I’m sure there are terrible assumptions and design choices in it, but there’s also a lot of effort that went into testing every function, documenting them, and being as exhaustive and meticulous as possible.
I still have an itch for sensors, and light in general, and I’ve been working on more promising avenues using linear light sensors that can be used to accurately capture a wide range of spectral data, which can then be accurately converted to colour temperature (the way it should be done!), but I hope zscilib sees some use outside my own narrow, obsessive use case, and I’d love for anyone else to love it to.
Interested ?
If you’re interested in scientific computing on low power embedded (or edge) devices, contributors and users are always welcome, and you can have a look at the work that’s already been done with the following links:
– Source Code: https://github.com/zscilib/zscilib
– API Documentation: https://zscilib.github.io/zscilib/
– Current Feature: https://github.com/zscilib/zscilib#current-features
– Issues if you want to contribute!: https://github.com/zscilib/zscilib/issues
About Me
Kevin Townsend works as a Senior Embedded Engineer at Linaro, working on a number of open source projects such as Zephyr RTOS and Trusted-Firmware M. You can usually find him on the Zephyr Slack, or reach out on Github (@microbuilder).