Seja bem vind@, se você é um debiano (um baiano que usa debian) faça parte de nossa comunidade!


DebConf 14: Community, Debian CI, Ruby, Redmine, and Noosfero

1 de Setembro de 2014, por Desconhecido - 0sem comentários ainda

This time, for personal reasons I wasn’t able to attend the full DebConf, which started on the Saturday August 22nd. I arrived at Portland on the Tuesday the 26th by noon, at the 4th of the conference. Even though I would like to arrive earlier, the loss was alleviated by the work of the amazing DebConf video team. I was able to follow remotely most of the sessions I would like to attend if I were there already.

As I will say to everyone, DebConf is for sure the best conference I have ever attended. The technical and philosophical discussions that take place in talks, BoF sessions or even unplanned ad-hoc gathering are deep. The hacking moments where you have a chance to pair with fellow developers, with whom you usually only have contact remotely via IRC or email, are precious.

That is all great. But definitively, catching up with old friends, and making new ones, is what makes DebConf so special. Your old friends are your old friends, and meeting them again after so much time is always a pleasure. New friendships will already start with a powerful bond, which is being part of the Debian community.

Being only 4 hours behind my home time zone, jetlag wasn’t a big problem during the day. However, I was waking up too early in the morning and consequently getting tired very early at night, so I mostly didn’t go out after hacklabs were closed at 10PM.

Despite all of the discussion, being in the audience for several talks, other social interactions and whatnot, during this DebConf I have managed to do quite some useful work.

debci and the Debian Continuous Integration project

I gave a talk where I discussed past, present, and future of debci and the Debian Continuous Integration project. The slides are available, as well as the video recording. One thing I want you to take away is that there is a difference between debci and the Debian Continuous Integration project:

  • debci is a platform for Continuous Integration specifically tailored for the Debian repository and similar ones. If you work on a Debian derivative, or otherwise provides Debian packages in a repository, you can use debci to run tests for your stuff.
    • a (very) few thinks in debci, though, are currently hardcoded for Debian. Other projects using it would be a nice and needed peer pressure to get rid of those.
  • Debian Continuous Integration is Debian’s instance of debci, which currently runs tests for all packages in the unstable distribution that provide autopkgtest support. It will be expanded in the future to run tests on other suites and architectures.

A few days before DebConf, Cédric Boutillier managed to extract gem2deb-test-runner from gem2deb, so that autopkgtest tests can be run against any Ruby package that has tests by running gem2deb-test-runner --autopkgtest. gem2deb-test-runner will do the right thing, make sure that the tests don’t use code from the source package, but instead run them against the installed package.

Then, right after my talk I was glad to discover that the Perl team is also working on a similar tool that will automate running tests for their packages against the installed package. We agreed that they will send me a whitelist of packages in which we could just call that tool and have it do The Right Thing.

We might be talking here about getting autopkgtest support (and consequentially continuous integration) for free for almost 2000 packages. The missing bits for this to happen are:

  • making debci use a whitelist of packages that, while not having the appropriate Testsuite: autopkgtest field in the Sources file, could be assumed to have autopkgtest support by calling the right tool (gem2deb-test-runner for Ruby, or the Perl team’s new tool for Perl packages).
  • make the autopkgtest test runner assume a corresponding, implicit, debian/tests/control when it not exists in those packages

During a few days I have mentored Lucas Kanashiro, who also attended DebConf, on writing a patch to add support for email notifications in debci so maintainers can be pro-actively notified of status changes (pass/fail, fail/pass) in their packages.

I have also started hacking on the support for distributed workers, based on the initial work by Martin Pitt:

  • updated the amqp branch against the code in the master branch.
  • added a debci enqueue command that can be used to force test runs for packages given on the command line.
  • I sent a patch for librabbitmq that adds support for limiting the number of messages the server will send to a connected client. With this patch applied, the debci workers were modified to request being sent only 1 message at a time, so late workers will start to receive packages to process as soon as they are up. Without this, a single connected worker would receive all messages right away, while a second worker that comes up 1 second later would sit idle until new packages are queued for testing.

Ruby

I had some discusion with Christian about making Rubygems install to $HOME by default when the user is not root. We discussed a few implementation options, and while I don’t have a solution yet, we have a better understanding of the potential pitfalls.

The Ruby BoF session on Friday produced a few interesting discussions. Some take away point include, but are not limited to:

  • Since the last DebConf, we were able to remove all obsolete Ruby interpreters, and now only have Ruby 2.1 in unstable. Ruby 2.1 will be the default version in Debian 8 (jessie).
  • There is user interest is being able to run the interpreter from Debian, but install everything else from Rubygems.
  • We are lacking in all the documentation-related goals for jessie that were proposed at the previous DebConf.

Redmine

I was able to make Redmine work with the Rails 4 stack we currently have in unstable/testing. This required using a snapshot of the still unreleased version 3.0 based on the rails-4.1 branch in the upstream Subversion repository as source.

I am a little nervous about using a upstream snapshot, though. According to the "roadmap of the project ":http://www.redmine.org/projects/redmine/roadmap the only purpose of the 3.0 release will be to upgrade to Rails 4, but before that happens there should be a 2.6.0 release that is also not released yet. 3.0 should be equivalent to that 2.6.0 version both feature-wise and, specially, bug-wise. The only problem is that we don’t know what that 2.6.0 looks like yet. According to the roadmap it seems there is not much left in term of features for 2.6.0, though.

The updated package is not in unstable yet, but will be soon. It needs more testing, and a good update to the documentation. Those interested in helping to test Redmine on jessie before the freeze please get in touch with me.

Noosfero

I gave a lighting talk on Noosfero, a platform for social networking websites I am upstream for. It is a Rails appplication licensed under the AGPLv3, and there are packages for wheezy. You can checkout the slides I used. Video recording is not available yet, but should be soon.

That’s it. I am looking forward to DebConf 15 at Heidelberg. :-)



An introduction to the Debian Continuous Integration project

1 de Junho de 2014, por Desconhecido - 0sem comentários ainda

Debian is a big system. At the time of writing, looking at my local package list caches tells me that the unstable suite contains 21306 source packages, and 42867 binary packages on amd64. Between these 42867 binary packages, there is an unthinkable number of inter-package dependencies. For example the dependency graph of the ruby packages contains other 20-something packages.

A new version of any of these packages can potentially break some functionality in the ruby package.

And that dependency graph is very small. Looking at the dependency graph for, say, the rails package will make your eyes bleed. I tried it here, and GraphViz needed a PNG image with 7653×10003 pixels to draw it. It ain’t pretty. Installing rails on a clean Debian system will pull in another 109 packages as part of the dependency chain. Again, as new versions of those packages are uploaded the archive, there is a probability that a backwards-incompatible change, or even a bug fix which was being worked around, might make some funcionality in rails stop working. Even if that probability is low for each package in the dependency chain, with enough packages the probability of any of them causing problems for rails is quite high.

And still the rails dependency chain is not that big. libreoffice will pull in another 264 packages. gnome will pull in 1311 dependencies, and kde-full 1320 (!).

With a system this big, problems will arrive, and that’s a fact of life. As developers, what we can do is try to spot these problems as early as possible, and fixing them in time to make a solid release with the high quality Debian is known for.

While automated testing is not the proverbial Silver Bullet of Software Engineering, it is an effective way of finding regressions.

Back in 2006, Ian Jackson started the development of autopkgtest as a tool to test Debian packages in their installed form (as opposed to testing packages using their source tree).

In 2011, the autopkgtest test suite format was proposed as a standard for the Debian project, in what we now know as the DEP-8 specification.

Since then, some maintainers such as myself started experimenting with DEP-8 tests in their packages. There was an expectation in the air that someday, someone would run those tests for the entire archive, and that would be a precious source of QA information.

Durign the holiday break last year, I decided to give it a shot. I initially called the codebase dep8. Later I renamed it to debci, since it could potentially also run other other types of test suites in the future. Since early January, ci.debian.net run an instance of debci for the Debian Project.

The Debian continuous Integration will trigger tests at most 4 times a day, 3 hours after each dinstall run. It will update a local APT cache and look for packages that declare a DEP-8 test suite. Each package with a test suite will then have its test suite executed if there was any change in its dependency chain since the last test run. Existing test results are published at ci.debian.net every hour, and at the end of each batch a “global status” is updated.

Maintainers can subscribe to a per package Atom feed to keep up with their package test results. People interested in the overall status can subscribe to a global Atom feed of events.

Since the introduction of Debian CI in mid-January 2014, we have seen an amazing increase in the number of packages with test suites. We had little less than 200 packages with test suites back then, against around 350 now (early June 2014). The ratio of packages passing passing their test suite has also improved a lot, going from less than 50% to more than 75%.

There is documentation available, including a FAQ for package maintainers with further information about how the system works, how to declare test suites in their packages and how to reproduce test runs locally. Also available is development information about debci itself, to those inclined to help improve the system.

This is just the beginning. debci is under a good rate of development, and you can expect to see a constant flux of improvements. In special, I would like to mention a few people who are giving amazing contributions to the project:

  • Martin Pitt has been working on improving debci to support parallel and distributed workers. Being the current autopkgtest maintainer, Martin also already got some bug fixes and fixes into autopkgtest motivated by debci use cases.
  • Brandon Fairchild is a GSOC student working on improving the debci web interface to provide more useful information, display information for multiple suites and architectures, plus making the UI work even without Javascript enabled.
  • Lucas Kanashiro is another GSOC student, who is working on investigating patterns among packages that fail their test suites, so that we can figure out how we can fix them, or if there are classes of failures that are actually caused by problems in the debci infrastructure.


how-can-i-help: oportunidades de contribuição com o Debian

11 de Fevereiro de 2014, por Desconhecido - 0sem comentários ainda

Já faz um tempo desde o meu último post, e pra esse novo vou retomar o assunto do anterior — como começar a colaborar com o Debian, inspirado num post recente do Stefano.

O how-can-i-help é um programa que te mostra oportunidades de colaboração relacionadas aos pacotes que estão instalados no seu sistema: pacotes que precisam de novos mantenedores, pacotes que estão com bugs críticos, pacotes que estão com bugs críticos e vão ser removidos da testing, etc.

Outro tipo de possível contribuição que o how-can-i-help vai listar e que vai ser bastante útil pra quem quer começar são bugs com a tag “gift”, ou seja bugs que o mantenedor do pacote marcou como fáceis para serem resolvidos por novos colaboradores.

Por padrão o how-can-i-help vai rodar toda vez que você instalar um pacote, mas também você também pode rodar ele digitando how-can-i-help manualmente no terminal.

Então, se você quer começar a contribuir com o Debian, você pode começar agora mesmo com:


  $ apt-get install how-can-i-help

OBS: o how-can-i-help só estão disponível a partir do Debian jessie (próximo release), ou seja, você precisa estar usando jessie(testing) ou sid(unstable).



Criando um mapa com dados importados do OpenStreetMap

7 de Novembro de 2013, por Desconhecido - 0sem comentários ainda

Eu mapeei todas as estações de compartilhamento de bicicletas do BikeSalvador e queria fazer um mapa que mostrasse essas estações. Quando havia apenas cinco estações, após mapear no OSM, eu criei um arquivo GeoJSON usando o geojson.io. Porém agora já são 19 estações. Seria um retrabalho enorme mapear as estações no OSM e depois criar o geojson manualmente.

Felizmente conheci o osmfilter, um software para filtrar os dados do OpenStreetMap. Combinando o osmfilter com o geojson.io é possível facilmente extrair alguns dados do OpenStreetMap e apresentar essa informação em um mapa personalizado. Então vamos às instruções de como fazer isso.

O primeiro passo é baixar os dados do OSM. Se você quiser trabalhar com a base de dados de todo o Brasil, você pode baixar dos servidores do GeoFabrik. Como eu necessitava apenas dos dados de uma cidade, utilizei o editor JOSM, fiz o download dos dados e salvei em um arquivo .osm no meu computador.

Download dos dados no JOSM

Agora nós precisamos utilizar o osmfilter. Veja as instruções de instalação na página osmfilter no Wiki do OpenStreetMap. O comando que eu utilizei para filtrar as estações de compartilhamento de bicicletas de Salvador foi:

./osmfilter salvador.osm --keep="amenity=bicycle_rental" > bikesalvador.osm

Você pode combinar mais de um filtro em um único comando. Por exemplo, se você quiser filtrar todos os restaurantes italianos, você poderia utilizar --keep="amenity=restaurant and cuisine=italian".

Aqui entra o geojson.io. Acesse o site, clique no botão Open e importe o arquivo gerado pelo osmfilter.

geojson.io interface

Após isso, você vai ver todos os dados filtrados sobre o mapa, inclusive os metadados. Se parte dos metadados não te interessar, você pode remover uma ou mais colunas.

Você vai precisar de uma conta no GitHub para salvar seu arquivo GeoJSON. Depois de salvar, o GitHub irá te fornecer uma página com o seu GeoJSON e também um código para que você possa incluir o mapa que você criou em uma página web. O mapa gerado pelo geojson.io é esse abaixo:

Quem quiser criar um mapa ainda melhor, recomendo ler esse tutorial de como adicionar uma camada GeoJSON ao Leaflet.



Criando um aplicativo móvel HTML5 em menos de 24 horas

23 de Outubro de 2013, por Desconhecido - 0sem comentários ainda

Screenshot do Fui pra Final!

Tudo começou quando meu amigo e colega de trabalho César Velame me propôs fazer um aplicativo que calculasse a nota necessária na prova final para um aluno ser aprovado. Como a UFRB utiliza pesos diferentes entre a prova final e a média da disciplina, muita gente tem dificuldade em calcular a nota necessária para ser aprovado. O intuito do aplicativo foi facilitar a vida desses estudantes.

Começamos a fazer na manhã de segunda-feira. Utilizamos jQuery Mobile, que eu já tinha alguma experiência, pois utilizei no Clips.tk. Acabei não me lembrando que na homepage do jQuery Mobile há uma ferramenta que permite gerar a estrutura básica da página HTML. Isso teria nos poupado algum tempo, mas valeu o aprendizado. jQuery Mobile é fantástico e bem fácil de usar!

Aprendemos também um pouco de javascript para calcular a nota necessária na prova final e fazer isso ser exibido na página. No final do dia, disponibilizamos a primeira versão do aplicativo em fuiprafinal.com.br. Na manhã seguinte, corrigimos alguns erros e fizemos um layout mais atraente. Eu também escrevi algumas frases engraçadinhas para o app mostrar de acordo com a situação do estudante!

Ainda estamos melhorando o Fui pra final! e em breve vamos disponibilizar no Marketplace do Firefox OS e testar a geração de aplicativos nativos para Android e outras plataformas através do PhoneGap.

Essa experiência foi muito interessante, pois foi uma maneira divertida e prática de aprender novas ferramentas. O fato de estar disponibilizando algo para as pessoas nos estimula a programar mais e com melhor qualidade. O código fonte do aplicativo está disponível no GitHub.