At the and of 2016 I had the pleasure to attend the 11th Latin American Conference on Pattern Languages of Programs, a.k.a SugarLoaf PLoP. PLoP is a series of conferences on Patterns (as in “Design Patterns”), a subject that I appreciate a lot. Each of the PLoP conferences but the original main “big” conference has a funny name. SugarLoaf PLoP is called that way because its very first edition was held in Rio de Janeiro, so the organizers named it after a very famous mountain in Rio. The name stuck even though a long time has passed since it was held in Rio for the last time. 2016 was actually the first time SugarLoaf PLoP was held outside of Brazil, finally justifying the “Latin American” part of its name.
I was presenting a paper I wrote on patterns for testing Debian packages. The Debian project funded my travel expenses through the generous donations of its supporters. PLoP’s are very fun conferences with a relaxed atmosphere, and is amazing how many smart (and interesting!) people gather together for them.
My paper is titled “Patterns for Writing As-Installed Tests for Debian Packages”, and has the following abstract:
Large software ecosystems, such as GNU/Linux distributions, demand a large amount of effort to make sure all of its components work correctly invidually, and also integrate correctly with each other to form a coherent system. Automated Quality Assurance techniques can prevent issues from reaching end users. This paper presents a pattern language originated in the Debian project for automated software testing in production-like environments. Such environments are closer in similarity to the environment where software will be actually deployed and used, as opposed to the development environment under which developers and regular Continuous Integration mechanisms usually test software products. The pattern language covers the handling of issues arising from the difference between development and production-like environments, as well as solutions for writing new, exclusive tests for as-installed functional tests. Even though the patterns are documented here in the context of the Debian project, they can also be generalized to other contexts.
In practical terms, the paper documents a set of patterns I have noticed in the last few years, when I have been pushing the Debian Continous Integration project. It should be an interesting read for people interested in the testing of Debian packages in their installed form, as done with autopkgtest. It should also be useful for people from other distributions interested in the subject, as the issues are not really Debian-specific.
I have recently finished the final version of the paper, which should be published in the ACM Digital Library at any point now. You can download a copy of the paper in PDF. Source is also available, if you are into markdown, LaTeX, makefiles and this sort of thing.
If everything goes according to plan, I should be presenting a talk on this at the next Debconf in Montreal.
debci 1.4 was released just a few days ago. Among general improvements, I would like to highlight:
- pretty much every place in the web UI that mentions a PASS or a FAIL also displays the tested package version. This was suggested to me on IRC by Holger
- I also tried to workaround an instability when setting up the LXC containers used for the tests, where the test bed process setup would finish without failure even though some steps in the middle of it failed. This caused the very final step for the debci-specific setup to fail, so there was no
debciuser inside the container, which caused tests to fail because that user was missing. Before that was fixed I was always keeping an eye on this issue, fixing the issue by hand, and re-triggering the affected packages by hand, so as far I can tell there is no package whose status has been permanently affected by this.
- Last, but not least, this release brings an interesting contribution by Gordon Ball, which is keeping track of different failure states. debci will now let you know whether a currently failing package has always failed, has passed in a previous version, or if the same version that is currently failing has previously passed.
ci.debian.net has been upgraded to
debci 1.4 just after that. At the same time I have also upgraded
autopkgtest to their latest versions, available in jessie-backports. This means that it is now safe for Debian packages to assume the changes in autopkgtest 4.0 are available, in special the
$AUTOPKGTEST_* environment variables.
In other news, for several weeks there were had issues with tests not being scheduled when they should have. I was just assuming that the issue was due to the existing test scheduler,
debci-batch, being broken. Today I was working on a new implementation that is going to be a lot faster, I started to hit a similar issue on my local tests, and finally realized what was wrong. The fact is that
debci-batch stores the timestamp of the last time a package has been scheduled to run, and it there are no test result after that timestamp, it assumes the package is still in the queue to be tested, and does not schedule it again. It turns out that a few weeks ago, during maintainance work, I had cleared the queue, discarding all jobs that were there, but forgot to reset those timestamps, so when
debci-batch came around again, it checked the timestamp of the last request and did not make new requests because there was no test result after that timestamp! I cleared all those timestamps, and the system should now go back to normal.
That is it for now. I you want to contribute to the Debian CI project and want to get in touch, you can pop up on the
#debci channel on the OFTC IRC network, or mail the autopkgtest-devel mailing list.
Earlier today I was handling a reproducibility bug and decided I had to try a reproducibility test by myself. I tried reprotest, but I was being hit by a disorderfs issue and I was not sure whether the problem was with reprotest or not (at this point I cannot reproduce that anymore).
So I decided to hack a simple script to that, and it works. I even included it in devscripts after writing a manpage. Of course reprotest is more complete, extensible, and supports arbitrary virtualization backends for doing the more dangerous/destructive variations (such as changing the hostname and other things that require root) but for quick tests
debrepro does the job.
$ debrepro # builds current directory $ debrepro /path/to/sourcepackage # builds package there $ gbp-buildpackage --git-builder=debrepro # can be used with vcs wrappers as well
debrepro will do two builds with a few variations between them, including $USER, $PATH, timezone, locale, umask, current time, and will even build under disorderfs if available. Build path variation is also performed because by definition the builds are done in different directories. If diffoscope is installed, it will be used for deep comparison of non-matching binaries.
If you are interested and don’t want to build devscripts from source or wait for the next release, you can just grab the script, save it as “debrepro” somewhere on your $PATH and make it executable.
As of yesterday, I am the new maintainer of pristine-tar. As it is the case for most of Joey Hess’ creations, it is an extremely useful tool, and used in a very large number of Debian packages which are maintained in git.
My first upload was most of a terrain recognition nature: I did some housekeeping tasks, such as making the build idempotent and making sure all binaries are built with security hardening flags, and wrote a few automated test cases to serve as build-time and run-time regression test suite. No functional changes have been made.
As Joey explained when he orphaned it, there are a few technical challenges involved in making sure pristine-tar stays useful in the future. Although I did read some of the code, I am not particularly familiar with the internals yet, and will be more than happy to get co-maintainers. If you are interested, please get in touch. The source git repository is right there.
Debian Ruby Sprint 2016 - day 5: More Reproducible Builds, Retrospective, and A Little Bit of Tourism5 de Março de 2016, 16:21
Earlier today I was made aware by Holger of the results of our reproducibility efforts during the sprint. I would like to thank Lunar for pinging us about the issue, and Holger for pointing me to updated results. The figure below depicts a stacked area chart where the X axis is time and the green area is reproducible packages. Red is packages that fail to build, and Orange are unreproducible packages
I was able to book accommodation for the sprint attendees very close to both my place and the sprint venue, what was very useful but also had this downside of them not being able to see much of city. As the final day of the sprint was getting closer, we decided to have a different lunch to allow them to see one of the most famous local landmarks, the botanical gardens.
So we headed down to the botanical gardens, grabbed a few items for lunch at the park coffee shop, and set out to visit this very beautiful place. I have to say that there is the place were I usually take every visitor I have. We were joined by Gioavani who had just arrived for the the MiniDebconf on the following weekend.
The final lists of accomplishments of the day was again very impressive
- r10k 2.1.1-2
- run massive update on team repositories
- bump Standards-Version
- fix Vcs-* fields
- drop version in gem2deb build-dependency
- set debhelper compatibility level to 9
- update the default ruby-tests.rake
- day 4 report
- uploaded ruby-faraday-middleware-multi-json 0.0.6-2
- uploaded ruby-powerpack 0.1.1-2
- uploaded ruby-contracts 0.13.0-1
- uploaded ruby-chef-config 12.7.2-1 (NEW)
- uploaded ruby-foreigner #808530 and asked it removal from the NEW queue (was already ROMed)
- filled for RM ruby-opengraph-parser (#816752)
- new how-can-i-help version developed and uploaded
- uploaded ruby-romkan to unstable (from exp)
- uploaded ruby-rinku to unstable (from exp)
- uploaded ruby-ole to unstable (from exp)
- uploaded ruby-net-ldap to unstable (from exp)
- uploaded ruby-rack-mobile-detect to unstable (from exp)
- uploaded gem2deb 0.28 to help with reproducible builds: filenames are now sorted
- uploaded rails 2:126.96.36.199-2 with packaging improvements
- run unit tests during the build and on CI
- apply upstream patch to fix ActiveRecord breakage under Ruby 2.3
- pushed a ton of tags for existing uploads
- merged improvements to the team master repository
- review/cleanup the contents of the repository
- improved helper scripts to automate the workflow (upload, build, new-upstream, etc)
- followed up on ruby2.3 transition, filed #816698 against subversion because of ftbfs on mips, mipsel
- put ruby-cocoon into a better state
- uploaded ruby-plist
- gem2deb: gem2tgz will now create foo.gemspec (easier to patch) instead of metadata.yml
- gemwatch: ditto
- close #794139 jekyll bug (unreproducible)
- close #798934 ruby-ffi-rzmq bug (unreproducible)
- closed ftbfs #816586 #800057 #784699 as unreproducible
- reassigned #760952 #680297 to ruby2.3 (from ruby2.2)
- investigated how to list packages with non-buildd-binary uploads
- ScottK has removed ruby2.1 from unstable!
By the end of the afternoon I asked everyone to fill out a simple retrospective list, what we can use later to make future sprints better and better. Below are the results we got.
What was good:
- restricted room hours actually made for a nice rhythm (did not apply for a long time…)
- very good food
- very cheap food!
- longer period makes the effort of travel more worthwhile
- many participants and longer sprint than usual allowing more work to be done
- good preparation with clear goals, make the sprint usefull
- patience with the less experienced participants
- RFS very fast
- Antônio is an excellent host
- You all are so helpful
- great dinners
What could be better:
- room too close to the street, too much vehicle noise, but sometimes nice music
- more coffee ^W meat
- could know more portugues so ordering food would have been easier
- debian infra could have not been down during the sprint
The night ended at Bar do Alemão (“The German’s Bar”). Both their beer and their food are very good, but I don’t have enough elements to vouch for their authenticity. :) We were joined by Giovani (who we also met earlier in the botanic gardens), and by Paulo and Daniel who are organizing the MiniDebconf.
And that is the end of this year’s Debian Ruby team sprint. I hope we do it all over again next year.