Gustavo Noronha (kov): WebKitGTK+ testing

Most times when I blog about WebKitGTK+ it’s to talk about new features, and their usage in Epiphany. This time I’d like to tell those who care more about our test infrastructure. Like I said in my previous post, testing is something we take very seriously in WebKit land. It would be hard to get such a complex project, with such diversity of platforms moving forward without automated testing.

Apple hosts a buildbot master that controls a whole lot of build slaves, for many platforms. Today we added the fourth WebKitGTK+ build slave to the family: 64bits release. This makes up for 4 build slaves in total, 2 release bots sponsored by Collabora, and 2 debug bots sponsored by Igalia. These bots build WebKitGTK+, and then run what we call the “layout tests”. I use quotes there because the name is a bit misleading. Despite being HTML/JavaScript tests, they cover a whole lot of functionality, and tests for regressions in many areas, including security, crashes, animations, media playing, DOM behaviour, and javascript API behaviour. WebKitGTK+ bots currently run 6397 tests, which represent about half the available tests.

Our bots are also, as of today, the first ones to run platform-specific API tests. Almost a year ago we started writing small tests based on glib’s GTestSuite, and they have been very valuable in helping us make sure our API expectations aren’t breaking (at least unknowingly), and to be able to test things that would be very hard to have Layout Tests for. So, yay! Thanks to everyone involved.

Back to layout tests, now: the other half is currently skipped because of one of three main reasons:

• We suck, and the test fails for real, either because we are missing implementation of something it uses (say, JavaScript isolated worlds, which has been recently added), or because our implementation is wrong
• The test is a render dump, and we did not generate results for it yet
• We lack functionality to run the test in our DumpRenderTree implementation

The first one is the worst of all. It means we have broken functionality, or lack web compatibility. The second one is less bad, we can usually trust layout, and rendering to be good because most of the rendering code is shared (thought there are exceptions, of course). Render dumps are a special way of representing the render tree as text, and we need to generate our own results because of differences in things such as font sizes. The third one is also pretty bad - it means we cannot test some features; DumpRenderTree is an application that uses the port’s API to run the tests, and provide additional JavaScript API through JavaScriptCore.

If you feel like helping WebKitGTK+, choosing a bunch of these (specially non-render-dump) skipped tests to make them pass could likely be a good first step =).

Lucas Rocha: Resigning from the Board

I’ve been thinking a lot about what to do about my participation on GNOME Foundation’s Board of Directors in face of the fact that my wife and I are expecting our first baby in February. After careful consideration, I decided to resign. I want to be fully focused on my family (especially in the first few months) without feeling bad for doing Board stuff while I could be with my family or not doing Board stuff while I’m spending time with my family.

So, considering my priorities now, I feel that I would be more useful to the Board by stepping down and letting another person with a lot of energy to take my position on Board for the rest of this mandate. The Board has decided to appoint Jorge Castro to replace me. He’s highly motivated and I’m sure he’ll give good contributions as a member of the Board.

I’ve had a great time in my two and a half years on Foundation Board. It was a great way to support the community in several ways and to work with some very nice people. The GNOME Foundation is getting better everyday and I feel very proud to be part of it.

Gustavo Noronha (kov): WebKitGTK+, and the Page Cache

So, one of the things I get to do during work hours for Collabora is to contribute code, and do maintenance tasks for WebKitGTK+, and have been doing so since early last year, working on all kinds of things, from improving the network backend to handle the real-world web, to fixing scrolling problems, while reviewing patches from the many awesome developers who have been joining us (more on that later =D).

One of the big features I have worked on this past month or so, along with Xan Lopez is the Page Cache. The page cache is a feature of web browsers that makes going back, and forward between pages in the same view very fast. It’s better explained in this post, but to summarize, the idea is that instead of destroying all the work you have done since downloading the resources, and having to reparse/rebuild the structures the view uses to display the page from the cached resources, you hit pause on the page, and store the whole thing as is, and when coming back to it, you just hit play. You can see in the video two instances of Epiphany, one with the page cache enabled, one with it disabled. Easy to see which was has it enabled. Thanks to KiBi for the suggestion regarding a page that shows this easily =D.

<video controls="controls" height="302" src="http://kov.eti.br/media/webkit/page-cache.ogv" width="450"></video>

We initially thought we had this feature enabled, since our initialization functions (that exists since before the current maintainers were involved) did setup the number of desired pages in the cache, but during the hackfest we held in December we found out we were fooled all this time. Enabling the page cache does make going back faster, but also made lots of things become unstable and crash.

Since then, we have been working on figuring out all the problems, and fixing them, using help from adventurous users of in-development software ;D. I believe we’re now at a point in which I can happily declare the GTK+ port has a working page cache in trunk! If you’re interested in the nasty details, bear with me!

Let me go back in time a bit, and show you what problems we had. First, some background: the GTK+ port deviates a lot from the other ports when it comes to scrolling. This is because, when designing this part of the port, Holger Freyther had a very nice idea in mind: that the WebView should be a first-class citizen GTK+ scrollable widget. Meaning it would use GTK+’s adjustments for scrolling, and be able to interact with any parent scrolling widget, be it a GtkScrolledWindow, or a MokoFingerScroll.

We cannot just throw away all the rest of the scrolling code in WebCore, though, that deals with all the details related to interacting with the DOM, and JavaScript code. This means our WebView contains adjustments that need to be set, and unset on our port’s version of WebKit’s own representation of the view, called the FrameView, to interact with it, and to get updates on the bounds of content, and such. For every load, in the non-page-cache case, a new FrameView is created, the previous one is destroyed - this means we need to set the adjustments on every load.

The problem starts when you have the page cache enabled, because the code path used to do what is called “commit” the load of a cached page (that is, start replacing the content that is currently being displayed by the one that should now be displayed) is completely different, and we were not setting the adjustments on this new view, so we started with that.

But all was not well. We were still having weird behaviour with scrollbars disappearing, and becoming the wrong size, and worse, crashes when “back” was hit. We then started investigating in more detail how it is that the page cache does its magic, to try and figure out the source of all evil.

It turns out that when you leave a page that can be cached, the existing FrameView is no longer destroyed - it is stored as is in a CachedFrame to be restored if you go back, and a new one is created for the new page. This was having the undesired effect of having the adjustment be set in more than one FrameView at once, causing all kinds of (predictable, after we knew for real what was going on) unwanted effects. Thus, we reworked the code to make sure the adjustments are only ever set in one FrameView at once, making sure they are unset when the FrameView is being frozen, and reset when it’s being restored from the page cache.

Last, but not least, it was discovered that going back to a page that contained resources with data: URIs (such as Google results pages which contain a small number of image hits) also caused a crash. This was because our network backend was not storing the data: URI in the ResourceResponse objects it fed into WebCore. The page cache relies on those responses to recreate the requests it uses to artificially replay the load when restoring the page from the page cache, so we fixed that as well[http://trac.webkit.org/changeset/54786].

What can be taken from all this? Building browsers is a lot of hard work! I can’t think how we could deal with this level of complexity without the awesome testing suite of WebKit. The good news is all of those issues I talked about in this post are now covered by the automate tests that run as part of the normal buildbot cycle in our bots, so we’re covered for the future, at least for these specific problems =D.

Hugo Doria: Convertendo um repositório git para mercurial

Hoje eu precisei migrar um projeto do git para o mercurial. Achei que ia dar um trabalho enorme, mas , felizmente, o processo é bem simples. Segue um passo-a-passo rápido:

Para fazer a migração é preciso da extensão "convert" do mercurial, mas ela vem desabilitada por padrão. Para reverter isso adicione as seguintes linhas ao arquivo ~/.hgrc :

[extensions]
hgext.convert=

Agora para importar seu código faça:

$ hg convert seu-repositório-git

Exemplo:

$ hg convert ~/projetos/meu-git

Isso vai criar um novo diretório, chamado "meu-git-hg". Se preferir, você pode definir o diretório para onde o código será convertido. Para fazer isso basta rodar o comando dessa forma:

$ hg convert seu-repositório-git meu-novo-repositório-mercurial

Exemplo:

$ hg convert ~/projetos/meu-git ~/projetos/meu-hg

Ao tentar listar o conteúdo deste diretório você verá que ele está vazio. Para que seus arquivos originais apareçam neste é preciso fazer o seguinte:

$ cd meu-git-hg

$ hg checkout

Pronto! Seus arquivos aparecerão novamente e já estarão sob o mercurial. :-)

Maiores informações: http://mercurial.selenic.com/wiki/ConvertExtension