In this blog I share technical information, write about projects I'm involved with, or about cool/fun/interesting stuff I find.

Upgrades to Jessie, Ruby 2.2 transition, and chef update

2 de Julho de 2015, por Antonio Terceiro - 22 comentários

Last month I started to track all the small Debian-related things that I do. My initial motivation was to be concious about how often I spend short periods of time working on Debian. Sometimes it’s during lunch breaks, weekends, first thing in the morning before regular work, after I am done for the day with regular work, or even during regular work, since I do have the chance of doing Debian work as part of my regular work occasionally.

Now that I have this information, I need to do something with it. So this is probably the first of monthly updates I will post about my Debian work. Hopefully it won’t be the last.

Upgrades to Jessie

I (finally) upgraded my two servers to Jessie. The first one, my home server, is a Utilite which is a quite nice ARM box. It is silent and consumes very little power. The only problem I had with it is that the vendor-provided kernel is too old, so I couldn’t upgrade udev, and therefore couldn’t switch to systemd. I had to force systemv for now, until I can manage to upgrade the kernel and configure uboot to properly boot the official Debian kernel.

On my VPS things are way better. I was able to upgrade nicely, and it is now running a stock Jessie system.

fixed https on ci.debian.net

pabs had let me know on IRC of an issue with the TLS certificate for ci.debian.net, which took me a few iterations to get right. It was missing the intermediate certificates, and is now fixed. You can now enjoy Debian CI under https .

Ruby 2.2 transition

I was able to start the Ruby 2.2 transition, which has the goal of switch to Ruby 2.2 on unstable. The first step was updating ruby-defaults adding support to build Ruby packgaes for both Ruby 2.1 and Ruby 2.2. This was followed by updates to gem2deb (0.18, 0.18.1, 0.18.2, and 0.18.3) and rubygems-integration . At this point, after a few rebuild requests only 50 out of 137 packages need to be looked at; some of them just use the default Ruby, so a rebuild once we switch the default will be enough to make it use Ruby 2.2, while others, specially Ruby libraries, will still need porting work or other fixes.

Updated the Chef stack

Bringing chef to the very latest upstream release into unstable was quite some work.

I had to update:

  • ruby-columnize (0.9.0-1)
  • ruby-mime-types (2.6.1-1)
  • ruby-mixlib-log 1.6.0-1
  • ruby-mixlib-shellout (2.1.0-1)
  • ruby-mixlib-cli (1.5.0-1)
  • ruby-mixlib-config (2.2.1-1)
  • ruby-mixlib-authentication (1.3.0-2)
  • ohai (8.4.0-1)
  • chef-zero (4.2.2-1)
  • ruby-specinfra (2.35.1-1)
  • ruby-serverspec (2.18.0-1)
  • chef (12.3.0-1)
  • ruby-highline (1.7.2-1)
  • ruby-safe-yaml (1.0.4-1)

In the middle I also had to package a new dependency, ruby-ffi-yajl, which was very quickly ACCEPTED thanks to the awesome work of the ftp-master team.

Random bits

  • Sponsored a upload of redir by Lucas Kanashiro
  • chake, a tool that I wrote for managing servers with chef but without a central chef server, got ACCEPTED into the official Debian archive.
  • vagrant-lxc , a vagrant plugin for using lxc as backend and lxc containters as development environments, was also ACCEPTED into unstable.
  • I got the deprecated ruby-rack1.4 package removed from Debian


Computação Brasil #27: Software Livre

24 de Junho de 2015, por Antonio Terceiro - 0sem comentários ainda

A Computação Brasil é revista publicada pela Sociedade Brasileira de Computação (SBC), e publica edições temáticas cobrindo assuntos importantes na Computação. A ultima edição, de número 27, é dedicada ao Software Livre.

Citando o editorial, escrito pelo professor Paulo Roberto Freire Cunha, presidente da SBC:

Software Livre é hoje um assunto importantíssimo para toda a sociedade, ultrapassando a questão ideológica e tornando-se um ecossistema complexo e de interesse global, que inclui pesquisa científica, educação, tecnologia, segurança, licença de uso e políticas públicas.

Um ponto fundamental é o impacto que ele gera no mercado, atuando hoje como um facilitador para milhares de startups em todo o mundo. Por isso, tem um papel essencial num país como o Brasil, onde o sucesso do empreendedorismo e da inovação tecnológica está diretamente ligado à redução de custos, à criatividade e à flexibilidade.

Para destacar o assunto, a revista traz diversos especialistas que mostrarão as mais recentes análises e pesquisas sobre Software Livre, com o propósito de estimular o leitor a fazer uma reflexão sobre o futuro do desenvolvimento de plataformas com código aberto e suas aplicações em diferentes frentes.

Eu e o Paulo escrevemos juntos 2 textos para esta edição. O primeiro, com co-autoria da professora Christina Chavez, que foi minha orientadora de doutorado, é sobre Controle de Qualidade em projetos de software livre. O segundo, em autoria com o Rodrigo Maia, é sobre a reformulação do Software Público Brasileiro, projeto no qual trabalhamos juntos desde a metade e 2014.

A versão online da revista é meio tosca, se como eu você não tiver Flash instalado (não tenho idéia de como seja a versão com Flash). Mas a versão em PDF é bem razoável.



rrg: visualize the require hierarchy in Ruby projects

20 de Março de 2015, por Antonio Terceiro - 22 comentários

Yesterday I was hacking on some Ruby code and getting a weird error which I thought was caused by mutually recursive require statements (i.e. A requires B, and B requires A). Later I realized that this is not an issue in Ruby, since the intepreter keeps track of what has already been required and will not enter a loop. But during the investigation I came up with something that turned out to be useful.

rrg will read the source code of a Ruby project and generate a graph based on the require statements in the code; nodes represent the source files and an arrow from A to B means that A contains a `require ‘B’` statement.

From the README:

Just run rrg at the root of your project. rrg will parse the code inside lib/, and generate a graph description in the Graphviz format. You can pipe the output to Graphviz directly, or store it in a file and process it to generate an image.

If you call rrgv instead, it will automatically process the graph with Graphviz, generate a PNG image, and open it.

Let’s see some examples. First the classical “analysing itself” example, the require graph for rrg itself:

Not very interesting, since all of the logic is currently in the main binary and not on library code. But 1) when I do the refactorings I want to, there will be more library code and 2) while writing this I implemented also parsing scripts in bin/.

Now chake which is a slightly larger project:

An even larger (but still not that big) project, gem2deb:

Note that these visualizations may not be accurate representations of the actual source code. In Ruby, nothing stops one from implementing class A::B in lib/x/y.rb, but most reasonable code will make sure that filenames and the classes namespaces actually match.

If you are working on a sane codebase, though, visualizing graphs like this helps understand the general structure of the code and perceive possible improvements. The gem2deb graph gave me some ideas already and I didn’t even paid much attention to it yet.



rmail: reviving upstream maintaince

15 de Fevereiro de 2015, por Antonio Terceiro - 33 comentários

It is always fun to write new stuff, and be able to show off that shiny new piece of code that just come out of your brilliance and/or restless effort. But the world does not spin based just on shiny things; for free software to continue making the world work, we also need the dusty, and maybe and little rusty, things that keep our systems together. Someone needs to make sure the rust does not take over, and that these venerable but useful pieces of code keep it together as the ecosystem around them evolves. As you know, Someone is probably the busiest person there is, so often you will have to take Someone’s job for yourself.

rmail is a Ruby library able to parse, modify, and generate MIME mail messages. While handling transitions of Ruby interpreters in Debian, it was one of the packages we always had to fix for new Ruby versions, to the point where the Debian package has accumulated quite a few patches. The situation became ridiculous.

It was considered to maybe drop it from the Debian archive, but dropping it would mean either also dropping feed2imap and sup or porting both to other mail library.

Since doing this type of port is always painful, I decided instead to do something about the sorry state in which rmail was on the upstream side.

The reasons why it was not properly maintained upstream does not matter: people lose interest, move on to other projects, are not active users anymore; that is normal in free software projects, and instead of blaming upstream maintainers in any way we need to thank them for writing us free software in the first place, and step up to fix the stuff we use.

I got in touch with the people listed as owner for the package on rubygems.org, and got owner permission, which means I can now publish new versions myself.

With that, I cloned the repository where the original author had imported the latest code uploaded to rubygems and had started to receive contributions, but that repository was inactive for more than one year. It had already got some contributions from the sup developers which never made it in a new rmail release, so the sup people started using their own fork called “rmail-sup”.

Already in my repository, I have imported all the patches that still made sense from the Debian repository, did a bunch of updates, mainly to modernize the build system, and did a 1.1.0 release to rubygems.org. This release is pretty much compatible with 1.0.0, but since I did not test it with Ruby versions older than than one in my work laptop (2.1.5), I bumped the minor version number as warning to prospective users still on older Ruby versions.

In this release, the test suite passes 100% clean, what always gives my mind a lot of comfort:


$ rake
/usr/bin/ruby2.1 -I"lib:." -I"/usr/lib/ruby/vendor_ruby" "/usr/lib/ruby/vendor_ruby/rake/rake_test_loader.rb" "test/test*.rb"
Loaded suite /usr/lib/ruby/vendor_ruby/rake/rake_test_loader
Started
...............................................................................
...............................................................................
........

Finished in 2.096916712 seconds.

166 tests, 24213 assertions, 0 failures, 0 errors, 0 pendings, 0 omissions, 0 notifications
100% passed

79.16 tests/s, 11546.95 assertions/s

And in the new release I have just uploaded to the Debian experimental suite (1.1.0-1), I was able to drop all of the patches and just use the upstream source as is.

So that’s it: if you use rmail for anything, consider testing version 1.1.0-1 from Debian experimental, or 1.1.0 from rubygems.org if you into that, and report any bugs to the [github repository](https://github.com/terceiro/rmail). My only commitment for now is keep it working, but if you want to add new features I will definitively review and merge them.



DebConf 14: Community, Debian CI, Ruby, Redmine, and Noosfero

1 de Setembro de 2014, por Antonio Terceiro

This time, for personal reasons I wasn’t able to attend the full DebConf, which started on the Saturday August 22nd. I arrived at Portland on the Tuesday the 26th by noon, at the 4th of the conference. Even though I would like to arrive earlier, the loss was alleviated by the work of the amazing DebConf video team. I was able to follow remotely most of the sessions I would like to attend if I were there already.

As I will say to everyone, DebConf is for sure the best conference I have ever attended. The technical and philosophical discussions that take place in talks, BoF sessions or even unplanned ad-hoc gathering are deep. The hacking moments where you have a chance to pair with fellow developers, with whom you usually only have contact remotely via IRC or email, are precious.

That is all great. But definitively, catching up with old friends, and making new ones, is what makes DebConf so special. Your old friends are your old friends, and meeting them again after so much time is always a pleasure. New friendships will already start with a powerful bond, which is being part of the Debian community.

Being only 4 hours behind my home time zone, jetlag wasn’t a big problem during the day. However, I was waking up too early in the morning and consequently getting tired very early at night, so I mostly didn’t go out after hacklabs were closed at 10PM.

Despite all of the discussion, being in the audience for several talks, other social interactions and whatnot, during this DebConf I have managed to do quite some useful work.

debci and the Debian Continuous Integration project

I gave a talk where I discussed past, present, and future of debci and the Debian Continuous Integration project. The slides are available, as well as the video recording. One thing I want you to take away is that there is a difference between debci and the Debian Continuous Integration project:

  • debci is a platform for Continuous Integration specifically tailored for the Debian repository and similar ones. If you work on a Debian derivative, or otherwise provides Debian packages in a repository, you can use debci to run tests for your stuff.
    • a (very) few thinks in debci, though, are currently hardcoded for Debian. Other projects using it would be a nice and needed peer pressure to get rid of those.
  • Debian Continuous Integration is Debian’s instance of debci, which currently runs tests for all packages in the unstable distribution that provide autopkgtest support. It will be expanded in the future to run tests on other suites and architectures.

A few days before DebConf, Cédric Boutillier managed to extract gem2deb-test-runner from gem2deb, so that autopkgtest tests can be run against any Ruby package that has tests by running gem2deb-test-runner --autopkgtest. gem2deb-test-runner will do the right thing, make sure that the tests don’t use code from the source package, but instead run them against the installed package.

Then, right after my talk I was glad to discover that the Perl team is also working on a similar tool that will automate running tests for their packages against the installed package. We agreed that they will send me a whitelist of packages in which we could just call that tool and have it do The Right Thing.

We might be talking here about getting autopkgtest support (and consequentially continuous integration) for free for almost 2000 4000 packages. The missing bits for this to happen are:

  • making debci use a whitelist of packages that, while not having the appropriate Testsuite: autopkgtest field in the Sources file, could be assumed to have autopkgtest support by calling the right tool (gem2deb-test-runner for Ruby, or the Perl team’s new tool for Perl packages).
  • make the autopkgtest test runner assume a corresponding, implicit, debian/tests/control when it not exists in those packages

During a few days I have mentored Lucas Kanashiro, who also attended DebConf, on writing a patch to add support for email notifications in debci so maintainers can be pro-actively notified of status changes (pass/fail, fail/pass) in their packages.

I have also started hacking on the support for distributed workers, based on the initial work by Martin Pitt:

  • updated the amqp branch against the code in the master branch.
  • added a debci enqueue command that can be used to force test runs for packages given on the command line.
  • I sent a patch for librabbitmq that adds support for limiting the number of messages the server will send to a connected client. With this patch applied, the debci workers were modified to request being sent only 1 message at a time, so late workers will start to receive packages to process as soon as they are up. Without this, a single connected worker would receive all messages right away, while a second worker that comes up 1 second later would sit idle until new packages are queued for testing.

Ruby

I had some discusion with Christian about making Rubygems install to $HOME by default when the user is not root. We discussed a few implementation options, and while I don’t have a solution yet, we have a better understanding of the potential pitfalls.

The Ruby BoF session on Friday produced a few interesting discussions. Some take away point include, but are not limited to:

  • Since the last DebConf, we were able to remove all obsolete Ruby interpreters, and now only have Ruby 2.1 in unstable. Ruby 2.1 will be the default version in Debian 8 (jessie).
  • There is user interest is being able to run the interpreter from Debian, but install everything else from Rubygems.
  • We are lacking in all the documentation-related goals for jessie that were proposed at the previous DebConf.

Redmine

I was able to make Redmine work with the Rails 4 stack we currently have in unstable/testing. This required using a snapshot of the still unreleased version 3.0 based on the rails-4.1 branch in the upstream Subversion repository as source.

I am a little nervous about using a upstream snapshot, though. According to the "roadmap of the project ":http://www.redmine.org/projects/redmine/roadmap the only purpose of the 3.0 release will be to upgrade to Rails 4, but before that happens there should be a 2.6.0 release that is also not released yet. 3.0 should be equivalent to that 2.6.0 version both feature-wise and, specially, bug-wise. The only problem is that we don’t know what that 2.6.0 looks like yet. According to the roadmap it seems there is not much left in term of features for 2.6.0, though.

The updated package is not in unstable yet, but will be soon. It needs more testing, and a good update to the documentation. Those interested in helping to test Redmine on jessie before the freeze please get in touch with me.

Noosfero

I gave a lighting talk on Noosfero, a platform for social networking websites I am upstream for. It is a Rails appplication licensed under the AGPLv3, and there are packages for wheezy. You can checkout the slides I used. Video recording is not available yet, but should be soon.

That’s it. I am looking forward to DebConf 15 at Heidelberg. :-)