You have a dozen systems and aim to port software to all of them. How to you make sure that your porting efforts work on all systems? Let's play portability whack-a-mole!

This section of the portability series introduces how developing minci helped us keep on top of portability across all systems. Building minci was a straight-forward choice based upon our needs and the complexity of other available tools, but either way you're going to need some sort of machinery to verify that your continued portability efforts doesn't cause fallout on other systems.

This is especially true with umbrella systems, such as Linux distributions or SunOS derivatives, where portability measures for a target umbrella cause fallout for specific systems in that umbrella.

why continuous integration

With the continued development and integration of oconfigure underway, I soon ran into problems where one portability measure (e.g., adding _DEFAULT_SOURCE on a glibc Linux for including crypt(3)) would cause fallout on other systems. In this example, because _XOPEN_SOURCE had already been defined, and conflicted with the new define.

I needed a tool for making sure commits on the current focus were not causing fallout on other already-ported systems. With only one other system, I could keep both open in terminals. As that number grew, it became impractical. This is where CI (continuous integration) tools are handy for verifying…

  • compilation: that the source code compiles and links (catches missing functions and required libraries, feature tests, …);
  • regressions: run-time testing of the system in motion (catches alignment issues, different function behaviour, …); and
  • distribution: assets bundled with the distributed system are sufficient for building, regressions, and installation (checks forgotten assets).

Searching for CI tools produced lots of useless information. It's a buzzword and results were heavily skewed toward heavy-weight vendors. I'm sure there are many high-quality, light-weight systems hidden in the results, but I couldn't find any.

In the end, it seemed like a much simpler task to simply write one myself. At heart a CI tool consists of a test runner, which verifies for a single host; and the report server, which manages the reports of test runners. A test runner for BSD.lv tools would need only:

  1. freshen sources
  2. configure sources (run configure)
  3. build and test regressions and distribution (run make with various rules)
  4. report success or failure to the server

The server would need only:

  1. collect reports
  2. display reports

In the end, this was accomplished with a shell script, an openradtool data model, some driver code, and some nice CSS.

building a continuous integratationator

I ended (but am starting here) with the report server. The report server handles the set of test reports, which it must accept and produce. Each report consists of:

  • what software was tested
  • what system it was run on
  • when stages of freshing, configuring, and building finished (or not)
  • identify of the tester

For simplicity, I combined the accept and produce functionality into one tool backed by openradtool. Reports are accepted over a simple HTTP POST and produced in HTML5 from an HTTP GET from the same CGI resource.

Interaction of runners, sources, and the report server.

The data model is described in db.ort and the back-end driver in main.c. The driver both accepts reports and formats them in HTML. The rest is just CSS.

For the test runner, I opted for a simple POSIX shell script, minci.sh. So far this has worked fine on all target systems, with some snags along the way in portable idioms for UNIX tools, e.g., head -n1 vs. head -1 or echo -n vs. printf. I have the HTTP POST component be handled by curl, which is available on all systems I use—often as part of the base system.

Test runner sequence.

The BSD.lv tools are all mirrored from a local CVS repository to GitHub, making it easy to fetch sources from one location. The reason for using git(1) instead of cvs(1) via anoncvs is the concept of a source tree identified by the last commit. This way, if any two source trees have the same last-commit identifier, they're identical. CVS doesn't have this concept, so tracking the same source trees is more difficult. The minci.sh test runner uses this identifier to check whether sources need to be updated and re-checked.

The test runner runs periodically via crontab(1). This introduces a delay in testing sources, but this is at best a minor inconvenience. Most heavy-weight continuous integration tools are triggered on each commit, but I considered this unnecessary complexity.

Instead of relying on complicated usernames and passwords, the test runner identifies itself with an email address and shared secret token (hashed in the POST submission) in its configuration file. Very straightforward and very effective. Adding a new tester to the database is as simple as generating a token and assigning that to an e-mail address.

local test-runner configuration

Since the BSD.lv tools have been fitted to use pkg-config(1) for detecting libraries, it's easy for the test runner user to override system libraries, which might be out of date, with those installed locally. The user need only set PKG_CONFIG_PATH to where the package specification files are found. This comes particularly in handy for downloading LibreSSL or newer version of OpenSSL. For systems like kcaldav, I was also able to use this for local versions of kcgi when it wasn't available from a system's third-party package manager.

At present, there's no automated way to deploy dependencies for the test runners: they must be manually installed. As I introduce more inter-depending systems (like kcaldav), this will probably change.

The last feature added to the test runner was an auto-update, which works much in the same way as the other parts checking for source freshness. Auto-updating isn't a long-term feature, but is perfect for in situ updates as I flesh out the continuous integration tool itself.

results and future work

With continuous integration in place on all platforms, I'm free to make changes to individual repositories knowing that within an hour or so, I can see whether the changes have affected on platforms.

You can see the results for the BSD.lv tools in action at kristaps.bsd.lv/cgi-bin/minci.cgi.

One unexpected (but excellent) result of minci is that it puts more pressure on me to provide regression tests. This is specific to the SPARC64 hardware: it demonstrates that standard operation of my tools don't have hidden alignment issues. Since starting this continuous integration, I've added dozens of regression tests to oconfigure, kcgi, and even kcaldav.

For future work, I'll add a notification tool for the report server. The idea is to store a token of the last check time, run a check on reports since then that have passed or failed, then pass these along to a script such as hotplugd(8) or other tools do. This way, I needn't actually check the reports dashboard, and just wait for an e-mail if updates have failed on any systems.