As the day 4 of the Debian Ruby team sprint in Curitiba unfolded, we have now fixed a total of more than 70 build failure bugs, managed to almost finish the Ruby 2.3 transition to be good to migrate into testing, and bootstrapped some documentation that will help new contributors get up to speed with Ruby packaging faster.
We have also requested the removal of several packages that are either severely outdated, abandoned upstream, beyond repair, utterly wrong, or in some cases, all of the above.
The full list of work items finished yesterday is:
- filed for RM ruby-rubysl-test-unit (no rdeps, duplicates ruby-test-unit)
- filed for RM ruby-literati (no rdeps left)
- uploaded ruby-certificate-authority 0.1.6-2
- uploaded ruby-pdf-reader 1.4 (Closes: FTBFS #795763)
- uploaded ruby-http 1.0.2-1 (Closes: #795752)
- raise severity of FTBFS bugs with ruby2.3
- day 3 report
- NMU pcs and upload to DELAYED/2 to remove dependency on ruby-monkey-lib and build-dependency on ruby2.1-dev
- upload ruby-parslet (Closes: #795046)
- initial documentation of packaging workflow with updated helper scripts in the team repo – https://wiki.debian.org/Teams/Ruby/Workflow
- ask for the removal of ruby-rspec-longrun
- ruby-opengraph-parser – upstream bug; upstream unresponsive; asked uploader about removal (no rdeps)
- fix ruby-celluloid-io for ruby2.3
- fixed ruby-buff-extension to work on ruby2.3
- uploaded newer ruby-varia-model upstream to work with newer ruby-hashie
- ask for the removal of mdpress
- uploaded ruby-buff-config, ruby-semverse, ruby-buff-ruby-engine, ruby-buff-ignore, ruby-buff-shell-out to remove spork
- filed for RM spork (obsolete, broken)
- filed for RM ruby-gsl on failing archs
- upload ruby-solve (Closes: #816359, thanks zeha!)
- update ruby-standalone to work properly with ruby2.3 (needs to remove the rake binary)
- upload ruby-memfs
- ruby-rack (1.6.4-3) (ROM ruby-memcache-client)
- ruby-parslet 1.7.1-1 #795046
- upload ruby-listen
- upload ruby-clockwork
- upload ruby-bio with a patch to avoid transient FTBFS because of undefined class names
- fixed ruby-fakeweb ftbfs
- cleaned up old repositories on git.d.o
- investigated PET status (appears to somewhat work, but mostly decaying)
- uploaded ruby-ogginfo
- upload ruby-libxml (agan!)
- upload racc 1.4.14-1
- upload ruby-sidekiq-cron
- how-can-i-help updated
- racc 1.4.14-1 (not ruby-racc ;-))
We also managed to flirt with 2 capital sins. For those who care about these things, which I don’t (but I still care about you), I guess 2 out of 7 still means we are good? :-)
I few people that I will not name complained that they hadn’t had enough steak on the previous night, so we set out to visit a traditional all-you-can-eat Brazilian steakhouse (“churrascaria”). I made a reservation at Jardins Grill and there you have gluttony. I am pretty sure that “not enough steak” wasn’t an issue last night. You can see how happy, despite being full to almost the point of being sick, everyone was.
A disjunct set of people, who I will also not name, were very disappointed to find out that the ruby-tinder package has absolutely nothing to do with Tinder but were still very active on the later. Maybe Friday night we will have to split the group into a lust-free family party and a Tinder party.
Debian Ruby Sprint 2016 - day 3: Ruby 2.3 in unstable, Reproducible Builds, and Data Structures for Dinner Booths3 de Março de 2016, 13:42
Day 3 was again a full of useful work. Since the beginning of the sprint, we were able to fix more than 50 FTBFS¹ bugs, alongside general quality improvements in the packages.
¹ in the Debian jargon, FTBFS means that a package “fails to build from source”, which in Debian is a critical bug because users need to be able to produce binary packages from their source code to fully exercise the free software principles.
An important milestone that was also achieved on day 3 was the upload of ruby-defaults 1:2.3.0+1, making
ruby2.3 the new default version of Ruby. That is the version that will shipped in the next Debian release, codenamed stretch. This is the culmination of a joint effort between the Ruby team and Debian Release Team that involves rebuilding a little more than 130 packages that use the Ruby C API to make sure everything will just work on upgrades, both from the previous stable release, and from earlier snapshots of the current development release.
Another small change that will have a big impact for Debian and for free software was an improvement to gem2deb that fixes a reproducibility issue in Ruby packages and will help currently more than 100 Ruby packages become reproducible.
The full list of items that have been worked on is this:
- sponsor ruby-github-linguist. Closes: #815908. Thanks balasankar
- rebuild remaining failing packages – 55 remaining!
- uploaded ruby-em-synchrony (FTBFS #800029)
- day 2 report blog post
- ruby-certificate-authority-0.1.6-1 (#816506)
- filed for ruby-grack removal
- uploaded ruby-em-synchrony
- uploaded ruby-mathml (FTBS #800029)
- uploaded new upstream version of ruby-redis which claims better ruby2.2 support
- uploaded pry (0.10.3-2), hoping it will build correctly on the buildd
- uploaded ruby-hashery
- rails security issues: CVE-2016-2098, CVE-2016-2097
- fixed in unstable on 2:22.214.171.124-1
- contacted security team for jessie update
- uploaded ruby-ruby-engine, ruby-ruby-version, ruby-rspec-pending-for to NEW
- updated ruby-hashie to new upstream version
- ruby-remcached (fix FTBFS)
- made the switch to ruby2.3 in unstable *********
- filed https://bugs.ruby-lang.org/issues/12137 as ruby-awesome-print triggers this
- uploaded ruby-rspec 3.4
- uploaded ruby-rspec-rails 3.4
- uploaded ruby-grack 2.0.2-2 (fix FTBFS)
- ruby-memcache-client (#797256) => RoM ruby-memcache-client (#816538) + https://github.com/rack/rack/issues/1025
- upload ruby-acts-as-lists (Closes: FTBS #816482)
- ruby-beautify: fixed and uploaded
- filed for ruby-foreigner removal (+ fixed diaspora dependency)
- uploaded newer ruby-origin to fix FTBFS
- uploaded ruby-responders
- gem2deb hacking
- dh-make-ruby will not overwrite debian/copyright anymore
- force generated gemspecs to use date from debian/changelog. fixes https://tests.reproducible-builds.org/issues/unstable/timestamps_in_gemspec_files_issue.html
- uploaded version 0.27 to unstable
- make ruby-riddle tests run and work during build
- ruby-flexmock 2.0.4-1 -> uploaded
- RoM ruby-memcache-client (done #816538)
- upload nanoc 4.1.4 and fix tests for ruby2.3
- Upload new upstream version of ruby-test-after-commit
- uploaded newer upstream for ruby-origin fixing FTBFS bug
The day ended at Outback, where we had an amount of beer that led us to formulate what we will now call the One-Sided Dinner Booth Problem.
In a party arranged like above, when the people closest to wall need to go alleviate themselves of some beer, you basically have to perform a removal from the bottom of a stack, which requires popping all the elements at the top. When they come back, you have to options:
- place them back at their original position, in which case
- the people who stayed have to either stay standing or get up all over again (not sure what is worse)
- the odds are that the next person to leave will require less effort from the others
- shift everyone to places against the wall and place the one coming back at the very end of the bench away from the wall, in which case
- the people who stayed are perhaps less annoyed, but you have to rearrange plates and pints
- it is less likely that the next time will require less effort from the others
The One-Sided Dinner Booth Problem is finding the optimal data structure and algorithm for this situation. It is postulated that this is an NP-complete problem, and that only probabilistic solutions are cost-effective.
Day 1 ended with dinner at a Yamato, my preferred Japanese restaurant in the city. Curitiba has a very large Japanese community, and lots of Japanese restaurants. Yamato, however, is the only one were you will stumble upon senior Japanese people, probably first or second generation immigrants, what I guess says something about its authenticity.
Right after breaking for lunch, but before actually going out, we made what so far is official group photo (I might try again as the shot was not a really good one).
Of course the most interesting part was the actual work that was done, and day 2 list is not less impressive than the day before:
- ruby-albino 1.3.3-4 (#813644) @sbadia
- #816256: ruby-versionomy: FTBFS: `require’: cannot load such file — blockenspiel/unmixer_mri (LoadError)
- investiage ruby2.1 removal
- upload ruby-beautify
- upload ruby-aws-sdk
- made ruby-blockenspiel arch:all again
- filed RC bug against pcs depending on ruby2.1-dev
- ruby-dev in experimental now supports `pkg-config ruby` so packages can use pkg-config to build against the current default Ruby
- ruby-libxml (some tests skipped)
- fixed FTBFS #804794 in subtle delaying ruby2.1 removal (now in DELAYED/2)
- mailed debian-ruby and ask people to test ruby2.3 as default
- #816162: ruby-zentest: FTBFS: Failure: TestZenTest#test_testcase9
- ruby-sshkit 1.9.0~rc1-1 (795118)
- capistrano 3.4.0-1 + ruby-sshkit 1.9.0~rc1-1 => http://paste.debian.net/plain/410699
- filed for ruby-patron removal
- updated and uploaded ruby-pygments.rb updated + patch for tests (fix #812940)
- discussed dependency cycle resolution
- will keep cycle between ruby2.3 and rake (as they really need each other)
- will break cycle from pure ruby packages to ruby when they are depended on by ruby2.3
- for same package set, enable Multi-Arch: foreign when needed
- uploaded ruby-power-assert (M-A foreign, break ruby cycle)
- uploaded ruby-did-you-mean (M-A foreign, break ruby cycle)
- uploaded ruby-minitest (M-A foreign, break ruby cycle)
- uploaded ruby-test-unit (M-A foreign, break ruby cycle)
- uploaded rake (M-A foreign)
- uploaded ruby-net-telnet (M-A foreign)
- discussed removal of libruby metapackage, filed bug against dh-make-drupal (only rdep) #816417
- uploaded yard (Closes: #812809)
- rerun failed builds
- closed #798539 (not applicable on current version of ruby-webmock)
- closed #816321 fixed by ruby2.3 upload
- uploaded ruby-cliver (by @lucasmoura)
- uploaded ruby-celluloid-supervision ( closes: #810770)
- uploaded ruby-celluloid-fsm ( closes: #815107)
- uploaded ruby-raindrops
- investigated status of dhelp, ruby-mysql again
- ruby-http-form-data (1.0.1+gemwatch-1) @sbadia
- nmu ohcount (DELAYED/2)
- updated trocla to newest upstream, fixing #816257
- reassign #816358 from ruby-safe-yaml to ruby2.3
- close #816120 (transient bug in ruby-default-value-for, waiting for ruby-sqlite3 binNMU)
- upload updated ruby-rc4
- uploaded new upstream version of pry, fixing FTBFS with ruby2.3
- ruby-rb-inotify , required to new ruby-listen( need ruby-rb-inotify >= 0.9.7)
- applied patch from upstream to ruby2.3 for #816358
- uploaded new upstream version for coderay supporting ruby2.3
On Monday Cédric told us that he and Sebastien had brought a bottle of French wine and some smelly French cheeses, and suggested that in the best Debian tradition we should have a Mini Cheese and Wine Party™. Sure thing! Luckily there is a farmer’s market 2 blocks from home on Tuesdays mornings, where I usually buy my fruits, vegetables, and cheese & friends, so the timing was perfect. I went shopping early in the morning, and bought a few things, and was back before it was the time to go to UTFPR. After the day-long hacking session we stopped by another store nearby to buy a few extra bottles of wine and other snacks.
At night, in my place, I ended up playing cheese master.
There was enough food that at the end we were all very full.
And with the spokesperson task of the day done, off to hacking I am!
This year’s Debian Ruby team sprint started today here at Curitiba. Everyone arrived fine, and we started working at the meeting room we have booked for the week at Curitiba campus of the Federal Technical University of Paraná. The room is at the “Department of Business and Community Relations”, what makes a lot of sense! :-)
The day started with a quick setup, with a simple 8-port switch, and a couple of power strips. It tooks us a few minutes to figure what was blocked or not on the corporate network, and almost everyone who needs connections that are usually blocked in such environments already had their VPN setups so we were able to get started right after that.
We are taking notes live on mozilla’s piblic etherpad site
Today we accomplished quite a lot:
- analyzed the pending issues for the ruby2.3 transition issues, categorizing the missing packages into “needs a binNMU now”, “needs a binNMU after switching the default to ruby2.3”, and “broken”.
- ruby-defaults uploaded to experimental switching to ruby2.3 as default, and dropping support for ruby2.2
- ruby-gsl fixed to build against GSL 2.x (was blocking ruby2.3 transition)
- #816253: ruby-fast-gettext: fix FTBFS issue and import a new upstream
- ruby-aws: several issues fixed
- ruby-binding-of-caller: fixed rubygems-integration
- fixed ruby-kakasi-ffi (again)
- made ruby-blockenspiel arch:any again
- ruby-fast-gettext 1.0.0-1 (fix ftbfs 816253)
- ruby-aws-sdk: new upstream, debcheck fix and several bumps
- ruby-fcgi – dropped transitional packages + refreshed packaging with -w
- ruby-sshkit 1.8.0
- ruby-beautify: new upstream and few minor fixes
- asciidoctor (new version sponsored)
- capistrano 3.4.0 (ftbfs #795724, #802734)
- #816254: ruby-packetfu: FTBFS: expected NameError with “uninitialized constant PacketFu::FakePacket”, got #
- updated rake to 10.5.0; making it not include -I/usr/lib/ruby/vendor_ruby when running tests
- triaged an closed open bugs on ruby-httpclient that do not apply anymore.
- updated ruby-httpclient to get rid of warnings in apt-listbugs under Ruby 2.3
- ruby-packetfu (fix ftbfs #816254) by @kanashiro
- investigated extension/rdepends build failing with ruby2.3-only
- applied upstream patch to ruby to fix extension ftbfs when extension uses c++
- ruby-albino 1.3.3-4 (#813644)
- basic user-level testing using ruby2.3 as default:
- chef WORKS mostly; it seems ohai segfaults some times
- rails autopkgtest FIXED Could not find gem ‘binding_of_caller (>= 0.7.2)’, which is required by gem ‘web-console (~> 2.0)’, in any of the sources.
- vagrant WORKS
- redmine autopkgtest WORKS
- apt-listbugs FIXED warnings (e.g. try `apt-listbugs list $pkg`); caused by ruby-httpclient
- nanoc WORKS
- capistrano 3.4.0-1 (#795724, #802734) by @sbadia + @terceiro
It’s been almost 2 years since the Debian Continuous Integration project has been launched, and it has proven to be a useful resource for the development of Debian.
I have previously made a an introductory post, and this this is an update on the latest developments.
Back in early 2014 when Debian CI was launched, there were less than 200 source packages with declared test suite metadata, and using a single worker machine polling the archive for updates and running tests sequentially in an infinite loop (“the simplest thing that could possibly work”) was OK-ish.
Then our community started an incredible, slow and persistent effort to prepare source packages for automated testing, and we now have almost 5,000 of them. The original, over-simplistic design had to be replaced.
The effort of transforming debci in a distributed system was started by Martin Pitt, who did an huge amount of work. In the latest months I was able to complete that work, to a point where I am confident in letting it run (mostly) unatended. We also had lots of contributions to the web UI from Brandon Fairchild, who was a GSOC intern in 2014, and continues to contribute to this date.
All this work culminated in the migration from a single-worker model to a master/workers setup, currently with 10 worker nodes. On busy periods all of those worker nodes will go on for days with full utilization, but even then the turnaround between package upload and a test run is now a lot faster than it used to.
Debian members can inspect the resource usage on those systems, as well as the length of the processing queue, by browsing to the corresponding munin instance (requires authentication via a SSL client certificated issued by sso.debian.org).
The system is currenly being hosted on a Amazon EC2 account sponsored by Amazon.
The setup is fully automated and reproducible. It is not fully (or at all) documented yet, but those interested should feel free to get in touch on IRC (OFTC, #debci)
Testing backend changed from schroot to lxc
Together with the infrastructure updates, we also switched to using lxc instead of schroot as backend. Most test suites should not be affected by this, but the default lxc settings might cause some very specific issues in a few packages. See for example #806542 (“liblinux-prctl-perl: autopkgtest failures: seccomp, capbset”)
Adding support for KVM is also in the plans, and we will get to that at some point.
If you want to learn more on how you can add tests for your package, a good first start is the debci online documentation (which is also available locally if you install `debci`).
You might also be interested in watching the live tutorial (WebM, 469 MB!) that has been presented at Debconf 15 earlier this year, full of tips and real examples from the archive. It would be awesome if someone wanted to transcribe that into a text tutorial ;-)
How to get involved
There are a few ways you can contribute:
autodep8. if you are knowledgeable on a subset of packages that are very similar and can have their tests executed in a similar way, such as “$Language libraries”, you might consider writing a test metadata generator so that each package does not need to declare a debian/tests/control file explicitly, requiring only The `Testsuite:` header in debian/control.
Ruby and Perl are already covered, and there is initial support for NodeJS. Adding support for new types of packages is very easy. See the source repository.
If you manage to add support for your favorite language, please get in touch so we can discuss whitelisting the relavant packages in ci.debian.net so that they will get their tests executed even before being uploaded with the proper `Testsuite:` control field.
autopkgtest. autopkgtest is responsible for actually running your tests, and you can use it to reproduce test runs locally.
debci. debci is the system running in ci.debian.net (version 1.0, currently in testing, is exactly what is running up there, minus a version number and a changelog entry).
It can also be used to have private clones of ci.debian.net, e.g. for derivatives or internal Debian-related development. See for example the Ubuntu autopkgtest site.
Getting in touch
For maintainer queries and general discussion:
- mailing list: email@example.com
- IRC: #debian-qa on OFTC. Feel free to highlight `terceiro`
For the development of debci/autopkgtest/autodep8
- mailing list: firstname.lastname@example.org
- IRC: #debci on OFTC