Among the many things I missed (Spotify is number 1 in that list btw), I could have done with a countdown timer. Contrary to what the interfaces of most Android timer apps would have you believe, it’s not rocket-science to write a usable timer, so I figured this would be a decent entry-point into writing mobile web applications. For the most part, this would just be your average web-page, but I did want it to feel ‘native’, so I started looking at the new building blocks site that documents the FirefoxOS shared resources. I had elaborate plans for tabs and headers and such, but turns out, all I really needed was the button style. The site doesn’t make hugely clear that you’ll actually need to check out the shared resources yourself, which can be found on GitHub.

Writing the app was easy, except perhaps for getting things to align vertically (for which I used the nested div/”display: table-cell; vertical-alignment: middle;” trick), but it was a bit harder when I wanted to use some of the new APIs. In particular, I wanted the timer to continue to work when the app is closed, and I wanted it to alert you only when you aren’t looking at it. This required use of the Alarm API, the Notifications API and the Page Visibility API.

The page visibility API was pretty self-explanatory, and I had no issues using it. I use this to know when the app is put into the background (which, handily, always happens before closing it. I think). When the page gets hidden, I use the Alarm API to set an alarm for when the current timer is due to elapse to wake up the application. I found this particularly hard to use as the documentation is very poor (though it turns out the code you need is quite short). Finally, I use the Notifications API to spawn a notification if the app isn’t visible when the timer elapses. Notifications were reasonably easy to use, but I’ve yet to figure out how to map clicking on a notification to raising my application – I don’t really know what I’m doing wrong here, any help is appreciated! Update: Thanks to Thanos Lefteris in the comments below, this now works – activating the notification will bring you back to the app.

The last hurdle was deploying to an actual device, as opposed to the simulator. Apparently the simulator has a deploy-to-device feature, but this wasn’t appearing for me and it would mean having to fire up my Linux VM (I have my reasons) anyway, as there are currently no Windows drivers for the Geeksphone devices available. I obviously don’t want to submit this to the Firefox marketplace yet, as I’ve barely tested it. I have my own VPS, so ideally I could just upload the app to a directory, add a meta tag in the header and try it out on the device, but unfortunately it isn’t as easy as that.

Getting it to work well as a web-page is a good first step, and to do that you’ll want to add a meta viewport tag. Getting the app to install itself from that page was easy to do, but difficult to find out about. I think the process for this is harder than it needs to be and quite poorly documented, but basically, you want this in your app:

if (navigator.mozApps) {
  var request = navigator.mozApps.getSelf();
  request.onsuccess = function() {
    if (!this.result) {
      request = navigator.mozApps.install(location.protocol + "//" + location.host + location.pathname + "manifest.webapp");
      request.onerror = function() {
        console.log("Install failed: " + this.error.name);
      };
    }
  };
}

And you want all paths in your manifest and appcache manifest to be absolute (you can assume the host, but you can’t have paths relative to the directory the files are in). This last part makes deployment very awkward, assuming you don’t want to have all of your app assets in the root directory of your server and you don’t want to setup vhosts for every app. You also need to make sure your server has the webapp mimetype setup. Mozilla has a great online app validation tool that can help you debug problems in this process.

And we’re done! (Ctrl+Shift+M to toggle responsive design mode in Firefox)

Visiting the page will offer to install the app for you on a device that supports app installation (i.e. a Firefox OS device). Not bad for a night’s work! Feel free to laugh at my n00b source and tell me how terrible it is in the comments

Avoid reflow

When I mentioned this at the conference, I prefaced it with a quick explanation of how rendering a web page works. It’s probably worth reiterating this. After network and such have happened and the DOM tree has been created, this tree gets translated into what we call the frame tree. This tree is similar to the DOM tree, but it’s structured in a way that better represents how the page will be drawn. This tree is then iterated over and the size and position of these frames are calculated. The act of calculating these positions and sizes is referred to as reflow. Once reflow is done, we translate the frame tree into a display list (other engines may skip this step, but it’s unimportant), then we draw the display list into layers. Where possible, we keep layers around and only redraw parts that have changed/newly become visible.

Really, reflow is actually quite fast, or at least it can be, but it often forces things to be redrawn (and drawing is often slow). Reflow happens when the size or position of things changes in such a way that dependent positions and sizes of elements need to be recalculated. Reflow usually isn’t something that will happen to the entire page at once, but incautious structuring of the page can result in this. There are quite a few things you can do to help avoid large reflows; set widths and heights to absolute values where possible, don’t reposition or resize things, don’t unnecessarily change the style of things. Obviously these things can’t always be avoided, but it’s worth thinking if there are other ways to achieve the result you want that don’t force reflow. If positions of things must be changed, consider using a CSS translate transform, for example – transforms don’t usually cause reflow.

If you absolutely have to do something that will trigger reflow, it’s important to be careful how you access properties in JavaScript. Reflow will be delayed as long as possible, so that if multiple things happen in quick succession that would cause reflow, only a single reflow actually needs to happen. If you access a property that relies on the frame tree being up to date though, this will force reflow. Practically, it’s worth trying to batch DOM changes and style changes, and to make sure that any property reads happen outside of these blocks. Interleaving reads and writes can end up forcing multiple reflows per page-draw, and the cost of reflow can add up quickly.

Avoid drawing

This sounds silly, but you should really only make the browser do as little drawing as is absolutely necessary. Most of the time, drawing will happen on reflow, when new content appears on the screen and when style changes. Some practical advice to avoid this would be to avoid making DOM changes near the root of the tree, avoid changing the size of things and avoid changing text (text drawing is especially slow). While repositioning doesn’t always force redrawing, you can ensure this by repositioning using CSS translate transforms instead of top/left/bottom/right style properties. Especially avoid causing redraws to happen while the user is scrolling. Browsers try their hardest to keep up the refresh rate while scrolling, but there are limits on memory bandwidth (especially on mobile), so every little helps.

Thinking of things that are slow to draw, radial gradients are very slow. This is really just a bug that we should fix, but if you must use CSS radial gradients, try not to change them, or put them in the background of elements that frequently change.

Avoid unnecessary layers

One of the reasons scrolling can be fast at all on mobile is that we reduce the page to a series of layers, and we keep redrawing on these layers down to a minimum. When we need to redraw the page, we just paste these layers that have already been drawn. While the GPU is pretty great at this, there are limits. Specifically, there is a limit to the amount of pixels that can be drawn on the screen in a certain time (fill-rate) – when you draw to the same pixel multiple times, this is called overdraw, and counts towards the fill-rate. Having lots of overlapping layers often causes lots of overdraw, and can cause a frame’s maximum fill-rate to be exceeded.

This is all well and good, but how does one avoid layers at a high level? It’s worth being vaguely aware of what causes stacking contexts to be created. While layers usually don’t exactly correspond to stacking contexts, trying to reduce stacking contexts will often end up reducing the number of resulting layers, and is a reasonable exercise. Even simpler, anything with position: fixed, background-attachment: fixed or any kind of CSS transformed element will likely end up with its own layer, and anything with its own layer will likely force a layer for anything below it and anything above it. So if it’s not necessary, avoid those if possible.

What can we do at the platform level to mitigate this? Firefox already culls areas of a layer that are made inaccessible by occluding layers (at least to some extent), but this won’t work if any of the layers end up with transforms, or aren’t opaque. We could be smarter with culling for opaque, transformed layers, and we could likely do a better job of determining when a layer is opaque. I’m pretty sure we could be smarter about the culling we already do too.

Avoid blending

Another thing that slows down drawing is blending. This is when the visual result of an operation relies on what’s already there. This requires the GPU (or CPU) to read what’s already there and perform a calculation on the result, which is of course slower than just writing directly to the buffer. Blending also doesn’t interact well with deferred rendering GPUs, which are popular on mobile.

This alone isn’t so bad, but combining it with text rendering is not so great. If you have text that isn’t on a static, opaque background, that text will be rendered twice (on desktop at least). First we render it on white, then on black, and we use those two buffers to maintain sub-pixel anti-aliasing as the background varies. This is much slower than normal, and also uses up more memory. On mobile, we store opaque layers in 16-bit colour, but translucent layers are stored in 32-bit colour, doubling the memory requirement of a non-opaque layer.

We could be smarter about this. At the very least, we could use multi-texturing and store non-opaque layers as a 16-bit colour + 8-bit alpha, cutting the memory requirement by a quarter and likely making it faster to draw. Even then, this will still be more expensive than just drawing an opaque layer, so when possible, make sure any text is on top of a static, opaque background when possible.

Avoid overflow scrolling

The way we make scrolling fast on mobile, and I believe the way it’s fast in other browsers too, is that we render a much larger area than is visible on the screen and we do that asynchronously to the user scrolling. This works as the relationship between time and size of drawing is not linear (on the whole, the more you draw, the cheaper it is per pixel). We only do this for the content document, however (not strictly true, I think there are situations where whole-page scrollable elements that aren’t the body can take advantage of this, but it’s best not to rely on that). This means that any element that isn’t the body that is scrollable can’t take advantage of this, and will redraw synchronously with scrolling. For small, simple elements, this doesn’t tend to be a problem, but if your entire page is in an iframe that covers most or all of the viewport, scrolling performance will likely suffer.

On desktop, currently, drawing is synchronous and we don’t buffer area around the page like on mobile, so this advice doesn’t apply there. But on mobile, do your best to avoid using iframes or having elements that have overflow that aren’t the body. If you’re using overflow to achieve a two-panel layout, or something like this, consider using position:fixed and margins instead. If both panels must scroll, consider making the largest panel the body and using overflow scrolling in the smaller one.

I hope we’ll do something clever to fix this sometime, it’s been at the back of my mind for quite a while, but I don’t think scrolling on sub-elements of the page can ever really be as good as the body without considerable memory cost.

Take advantage of the platform

This post sounds all doom and gloom, but I’m purposefully highlighting what we aren’t yet good at. There are a lot of things we are good at (or reasonable, at least), and having a fast page need not necessarily be viewed as lots of things to avoid, so much as lots of things to do.

Although computing power continues to increase, the trend now is to bolt on more cores and more hardware threads, and the speed increase of individual cores tends to be more modest. This affects how we improve performance at the application level. Performance increases, more often than not, are about being smarter about when we do work, and to do things concurrently, more than just finding faster algorithms and micro-optimisation.

This relates to the asynchronous scrolling mentioned above, where we do the same amount of work, but at a more opportune time, and in a way that better takes advantage of the resources available. There are other optimisations that are similar with regards to video decoding/drawing, CSS animations/transitions and WebGL buffer swapping. A frequently occurring question at EdgeConf was whether it would be sensible to add ‘hints’, or expose more internals to web developers so that they can instrument pages to provide the best performance. On the whole, hints are a bad idea, as they expose platform details that are liable to change or be obsoleted, but I think a lot of control is already given by current standards.

On a practical level, take advantage of CSS animations and transitions instead of doing JavaScript property animation, take advantage of requestAnimationFrame instead of setTimeout, and if you find you need even more control, why not drop down to raw GL via WebGL, or use Canvas?

I hope some of this is useful to someone. I’ll try to write similar posts if I find out more, or there are significant platform changes in the future. I deliberately haven’t mentioned profiling tools, as there are people far more qualified to write about them than I am. That said, there’s a wiki page about the built-in Firefox profiler, some nice documentation on Opera’s debugging tools and Chrome’s tools look really great too.

p.s. I’ll be talking on a performance panel at EdgeConf this Saturday. Though it’s fully booked, I think tickets occasionally become available again, so might be worth keeping an eye on. They’ll be much cleverer people than me knocking about, but I’ll be doing my best to answer your platform performance related questions.

So back from layout into graphics again! For the last few weeks, I’ve been working with Benoit Girard on getting progressive tile rendering finished and turned on by default in Firefox for Android. The results so far are very promising! First, a bit of background (feel free to skip to the end if you just want the results).

You may be aware that we use a multi-threaded application model for Firefox for Android. The UI runs in one thread and Gecko, which does the downloading and rendering of the page, runs in another. This is a bit of a simplification, but for all intents and purposes, that’s how it works. We do this so that we can maintain interactive performance – something of paramount important with a touch-screen. We render a larger area than you see on the screen, so that when you scroll, we can respond immediately without having to wait for Gecko to render more. We try to tell Gecko to render the most relevant area next and we hope that it returns in time so that the appearance is seamless.

There are two problems with this as it stands, though. If the work takes too long, you’ll be staring at a blank area (well, this isn’t quite true either, we do a low-resolution render of the entire page and use that as a backing in this worst-case scenario – but that often doesn’t work quite right and is a performance issue in and of itself…) The second problem is that if a page is made up of many layers, or updates large parts of itself as you scroll, uploading that work to the graphics unit can take a significant amount of time. During this time, the page will appear to ‘hang’, as unfortunately, you can’t upload data to the GPU and continue to use it to draw things (this isn’t true in every single case, but again, for our purposes, it is).

Progressive rendering tries to spread this load by breaking up that work into several smaller tiles, and processing them one-by-one, where appropriate. This helps us mitigate those pauses that may happen for particularly complex/animated pages. Alongside this work, we also add the ability for a render to be cancelled. This is good for the situation that a page takes so long to render that by the time it’s finished, what it rendered is no longer useful. Currently, because a render is done all at once, if it takes too long, we can waste precious cycles on irrelevant data. As well as splitting up this work, and allowing it to be cancelled, we also try to do it in the most intelligent order – render areas that the user can see that were previously blank first, and if that area intersects with more than one tile, make sure to do it in the order that maintains visual coherence the best.

A cherry on the top (which is still very much work-in-progress, but I hope to complete it soon), is that splitting this work up into tiles makes it easy to apply nice transitions to make the pathological cases not look so bad. With that said, how’s about some video evidence? Here’s an almost-Nightly (an extra patch or two that haven’t quite hit central), with the screenshot layer disabled so you can see what can happen in a pathological case:

And here’s the same code, with progressive tile rendering turned on and a work-in-progress fading patch applied.

This page is a particularly slow page to render due to the large radial gradient in the background (another issue which will eventually be fixed), so it helps to highlight how this work can help. For a fast-to-render page that we have no problems with, this work doesn’t have such an obvious effect (though scrolling will still be smoother). I hope the results speak for themselves

]]>

One of the perks of being a Virgin Media customer (beyond getting my name wrong and constant up-sell harassment) is that I got cheap, early-access Eurogamer Expo tickets! This was my first Eurogamer Expo, though I’m no stranager to ECTS or ATEI/EAG. The setup was quite good – perhaps a bit smaller than I expected, but nice to see a games show that’s actually aimed at gamers. I was always amused at the hoops you had to jump through to get tickets for ECTS and ATEI; more so when you actually visit the events and realise the majority of people there are gamers who have jumped through those same hoops. Good to see that the games industry, finally, after several years, got wise.

There was a fair amount on show. Lots of soon and not-so-soon to be released games, the WiiU, a surprising and pleasing amount of indie content and various bits and bobs. The WiiU was certainly the main attraction, but was managed terribly and was extremely disappointing. While most of the company reps were great and very helpful, a couple of Nintendo’s were oddly aggressive and patronising. I don’t think anyone at Eurogamer needs to be told how to play WiiU mini-games, or have buttons on their controllers pressed for them. The decision to dedicate three entire kiosks in the WiiU section to a video panorama viewer was baffling too. It’s almost as if no one at Nintendo has picked up a smart-phone in the last 5 years or so – this isn’t astounding stuff. Wonderful 101 seemed quite fun, but not as fun as I was expecting. The rest of the WiiU content was very disappointing. Pikmin 3 looking bland and boring was especially upsetting. It’s ironic that playing on the console has secured my decision not to buy it on release. I could easily write about how disappointing the WiiU was for a lot longer, but I just don’t care enough.

What was pleasantly surprising was how good Sony’s presence and content was. Reps were polite and helpful, not getting in the way where they weren’t needed and turning up when they were. Much like a good waiter. They had plenty of kiosks and space, and queues were minimal (not due to lack of interest, mind). Playstation All-stars Battle Royale, though clearly a Smash Bros. rip-off, is actually a very good one. We spent quite a while on it, and it was very enjoyable (possibly more so than Smash Bros. Brawl, but it doesn’t even approach the heights of Melee). The cross-play was especially impressive too, mirroring almost the exact same game frame-for-frame with only minor graphical omissions. Stand-out game of the show had to be When Vikings Attack, though. Incredibly simple concept, but perfect execution and impressive cross-play again. The only disappointment was that it doesn’t have a confirmed release date, but Clever Beans say it will be on PSN before the end of the year. This is definitely day-one purchase material.

Carmageddon definitely deserves a mention. It’s just as much fun as it was all those years ago, and the tablet/smartphone port has been handled perfectly. A shame that there was no demo or footage for the Carmageddon Reincarnation project, but hopefully it made a few more people aware. Also worth mentioning was God of War: Ascension, which although is more of the same, it’s a brilliant same that it’s more of. The multiplayer worked surprisingly well too, though a LAN setup is always going to be more fun than online. There were a few things that I’d have liked to have tried, but queues prevented me – nothing I would deem queue-worthy though. Hitman looked quite impressive, but the whole misogyny thing has put me off. Same goes for Tomb Raider. Dishonoured looked interesting, but not so interesting to queue for. Halo 4 looked like more of the same, though the considerable graphical upgrade certainly doesn’t hurt. Dead or Alive 5 was quite fun, and pleasing to see that they’ve returned to the mechanics of Dead or Alive 2 (clearly the series high). Disappointing amount of guys picking bikini-clad women to fight and leaving the camera aimed at crotch/chest areas; we evened the score a bit by playing as ridiculous-looking guys and aiming at the groin. Yes, I am 12. Disappointed to see that they’ve not included Zack’s weird sports-bra costume. The indie games arcade section is probably worth mentioning in that almost everything in it was terrible and just trading on a quirky look with zero gameplay to back it up. I conclude that there’s still plenty of room for ideas and innovation in the British indie games community.

All in all, a pretty fun event. Slightly disappointing that the industry still hasn’t moved on from the whole booth-babe thing, but it’s definitely far less prevalent than it used to be, so that leaves me with some hope. The graphical standard of console games is astounding, especially given there hasn’t been a hardware refresh in over 5 years. I’ll definitely be returning next year.

]]>

It seems, somehow, for the last few months, I’ve been working on layout. I’m not quite sure how it happened, as anyone who’s spoken to me or follows me on Twitter would know that I have a very healthy fear of the Gecko layout code. I still have that fear, but I’d like to think now that it’s coupled with a tiny amount of understanding; understanding that has, dare I say it, even let me have fun while working on this code!

Those of you that have used browsers on mobile phones (all of you?), especially not the very latest phones, may be familiar with an annoying problem. That is, elements that have position:fixed in their style tend to jump around the place like they’ve had too much coffee. You commonly see this on sites that have a persistent bar at the top or bottom of the page, or floating confirmation notifications, things like this. Brad Frost wrote about this far more eloquently than I could here. This has always annoyed me, especially after learning more about how browsers work. Certainly in Gecko, we have all of the context we need for this not to happen. It also ended up that this problem had been worked on long before I even joined Mozilla last year, so that made it doubly surprising that we suffered from this problem in all releases of Firefox Mobile.

When I first came across this last year, I discovered that the support was already there in the old Firefox Mobile, but disabled by default due to it causing test failures. I was working on other things then, and wasn’t at all acquainted with layout code, so I let it be. Revisiting it for the new, native Firefox Mobile though, these test failures didn’t exist anymore. Enabling this basic support that would let position:fixed elements move and zoom with user input correctly was not too big a deal – just flip an environment variable and write a small amount of support code. This landed in Firefox 15 and is tracked in Bug 607417. Just this is enough for a lot of mobile sites to start using position:fixed (I’m looking at you, Twitter and Facebook!).

This wasn’t enough for me though. Around this time, Android 3.x (Honeycomb) tablets had been around for quite a while and the Galaxy Nexus with Android 4.0 (Ice-cream Sandwich) had just come out, both with even better support for position:fixed. Not to mention the iPhone, which has excellent support. A problem with our implementation in Firefox 15, is that anything fixed to the bottom or right of the screen, or anything that doesn’t anchor to the top-left in any way, may become inaccessible after zooming in. In recent versions of the Android stock browser, not only do these remain accessible, but they zoom very smoothly too. Not wanting to be one-upped by what could be considered our main competition, I started to work on more comprehensive position:fixed support. This work was tracked in Bug 758620.

When zooming in our browser, we don’t change how the page is laid out, but fixed position elements are still rendered relative to the viewport. What we want (at least, for now) is for fixed position elements to lay out with respect to this viewport so that they always remain visible, and to transition smoothly between these states. To achieve this, I changed layout so that fixed position elements are laid out to what we call the scroll-port. When we zoom in, we change the scroll-port, otherwise you wouldn’t be able to scroll to the bottom-right of the page, but this only changes scrolling behaviour and nothing else. This change also made it so that fixed-position children of the document would be relayed out when this scrollport was changed. This fixed the inaccessibility problem, but left nasty-looking transitions when zooming in.

To fix the transitions was quite a bit more comprehensive and lead me down a long path of causing and fixing various layout bugs. When a page is rendered, the DOM tree is parsed into a frame tree, which better represents the layout of the page. This frame tree is then parsed into a display-list, which represents how to draw the page. This display-list is then optimised and parsed/drawn into a layer tree, which is the final representation before we composite it. There’s cleverness to make sure that layers aren’t recreated unnecessarily, but that’s another subject for another time. We wanted to be able to annotate the layer tree so that when compositing, we have enough information to determine how to place fixed-position layers when zooming. This involved creating a new display-list item with the information about how the element is fixed (to the top? left? right? bottom?), and also that would guarantee that the items representing this element would end up on their own layer. Once this was done, code in the compositor was added to leverage this information and draw layers in the right place.

This is an area that a lot of browsers have difficulty with, so it was a fun problem to work on. Try out a couple of my test-cases if you’re interested, they expose how different browsers handle this situation, and in the case of a few of them, some bugs We’re still not perfect, but we’re better than we were before – and to my feeling, we’re adequate now. This work landed in Firefox 16.

So is there work left to do? Well, unfortunately, yes. I’ve just finished off support for fixed backgrounds and backgrounds with multiple fixed/non-fixed layers, and this will arrive in Firefox 18. This is tracked in Bug 786502. I also think that the best behaviour would be for fixed position elements to layout to whichever is largest of the CSS viewport or the scroll-port, and for scrolling to be within the CSS viewport and push the edges when you reach them. I’m told this is what happens in IE10 on Windows 8, and is similar (but slightly better) to what gets done in Safari on iOS. I think it’s about time for a break from this particular feature for me, though.

]]>

I’ve been pretty lax on blogging lately, but here’s something that’s troubling me. I haven’t really done any work directly related to Gnome since I started working at Mozilla. Ends up writing browsers is pretty hard, and any recreational programming time I get, I don’t particularly feel inclined to work on Gnome. I have, however, been attending Guadec this week. I haven’t missed one since 2006 and I don’t intend to. What’s troubling me, is that although Mozilla were kind enough to sponsor my presence here (we’re hiring!), Gnome doesn’t seem to be hugely relevant to us anymore. I’d love to be corrected of course, but judging by the amount of effort we’re putting into the Gtk+3 port, themeing and other Linux-related bugs, I’m pretty sure this is the case.

I have some ideas about this, but I’d like to be brief. For now. So, my simple question is, How can Mozilla and Gnome work better together?

[Edit]: Seems my blog’s commenting form is broken. Until it’s fixed, feel free to mail me your comments, I’d love to hear them! (address on the side of the page)

[Edit2]: Comments appear to be working again, but if they fail, do mail me!

]]>

For those that aren’t already aware, many Mozillians gathered last week in Toronto to Maximise Synergy. Seeing as there have been updates for the UX team and the Firefox/Mobile UI team, I think it’d be worth having a similar update for mobile platform. Some fantastic work is going on in this area, and people deserve kudos

• Kartikaya Gupta (aka kats) continues to do sterling work, fixing an inhuman amount of bugs. Most prominently, kats has been working on our display-port ‘strategies’, code that allows us to make the most of our rendering time and, thus, reduce checkerboarding. He’s also been working on our checkerboard measurement, augmenting and refining our tests and test-method to make sure we’re always pushing ourselves. This, alongside much general bug-fixing work.
• Benoit Girard (aka BenWa) is part of the gfx team, all of whom have been helping us out for the past few months, getting our performance back to a competetive level. Benoit’s tiled rendering patches finally landed, which make a huge difference to performance. Benoit has also been working on some useful remote profiling tools, which he blogs about here.
• James Willcox (aka snorp) continues to battle Flash to get us the best Flash experience in an Android browser (ironic, right?) snorp’s work to get more accurate positioning of Flash elements landed, along with his work to replace elements with a snapshot during page movement on Froyo/Gingerbread, completing the experience.
• Brad Lassey (aka blassey) managed to get enough time to do some coding alongside his usual cat-herding responsibilities, and got us a low-res page cache. This means that in the rare situation that our rendering can’t keep up, instead of showing checkerboard, we show a low-resolution rendering of the page. It’s possible that this content can go stale, but we find it provides a much better experience than showing either a blank colour or checkerboard. This works much better than any of us expected it to, and for me at least, was the turning point after which I find our browser much easier to recommend.
• And me? I managed to get my retained tiles patch landed. Currently, we render the page into system-memory tiles, which we then upload to the GPU. My patch keeps a hold of the GPU tiles and metrics for a while after they’re invalidated. Again, when our renderer can’t keep up, it renders these old tiles into the space instead of rendering nothing. This mostly helps quick changes of direction and zooming out after zooming in. I also spent some time fixing up the low-res cache work, then fixing up my fix-ups, as is my custom Hopefully, preliminary fixed-position layer support will also land soon.

Much other work also occured, and sorry for missing anyone out that I surely did. It’s worth mentioning that the gfx team have been a huge help and have been working extra-hard for months now, helping us get to where we are today (and hopefully beyond!) I had a great (and hard) time during my stay and am very much looking forward to our upcoming release

]]>

It’s been way too long since I last blogged, so here’s something to try and get into the mood again. I have a few other things I’d like to write about, but work here at Mozilla is probably the most pressing one at the moment, so the others will have to wait.

Those who have been following Nightly, and/or who attended our talk at FOSDEM would be aware that we’re currently rewriting the Android version of Firefox Mobile. The major part of this change is that we’re now a ‘native’ Android app, rather than a XUL app. Consequently, we fit in and behave much better than the older version of Firefox Mobile, and we get a few perks too; namely start-up performance.

Alongside this new version of Firefox Mobile, we’ve also taken the opportunity to overhaul the platform side of things. Where as the old version was multi-process, we’ve now switched to a multi-threaded application model, with the input, content processing/rendering and UI rendering all taking separate threads. Just recently, we switched from a Java-based view compositor to a native-code off-main-thread-compositor (or OMTC, for short). While previously we drew the entire page into a buffer (with extra bits around the edges) and composited that, we now directly composite the layers that make up the page.

This gives us the speed and ease of using a single process (to some extent), but with the power afforded to us by an asynchronous rendering process. Most of what I said previously about shadow layers still applies, just replace ‘process’ with ‘thread’. Most of the problems still apply too, but the graphics team has put in some phenomenal work in fixing a lot of it.

So we’re now working our figurative nuts off fixing all the bugs that have cropped up, but I do believe we’re on the road to success. This rendering model gives us the power to retain a lot more content than before, and any saved drawing tends to translate into massive benefits on mobile – we appear to be mostly memory-bandwidth limited, so every little helps. What this translates to for users is smooth, 60fps updates, excellent interactive performance, excellent web standards support and a polished, ‘native’ feeling application. You can sample some of this work by downloading a Nightly build, it’s *almost* at the point of being a daily driver (not quite, but almost).

Maybe you want to help? Running a nightly build and providing feedback (either via the built-in methods, or by filing bugs) is a great way to start, and all the mobile developers hang out on IRC too, in #mobile. We’ve also just opened a new office in London, and I do believe we’re hiring, so if you think you’d like to get in on some of this, do contact us!

Update: The built-in method for feedback is far less obvious than I thought… You can access it via the ‘About’ page, (accessible via the settings menu), then following the links ‘Support’, ‘Ask a Question’, and finally ‘Give us feedback’. I think we should make this better, so I’ve filed a bug.

]]>

Perhaps the title of my last blog post seemed a little negative, so I wanted to write on this topic again, on some of the things I’ve learnt since then, and some of the success I’ve had since then too. Failure was probably too strong a word, but better to be too negative than too positive about these things, especially when surrounded by the amazing talent there is at Mozilla…

I finished off previously by saying there are other, easier problems to solve, and I think I’m making some decent progress in those areas. I described before how shadow layers work, and how the chrome process can use GL-accelerated layer compositing, but the content process is always restricted to basic (unaccelerated) layers. This introduces the bottleneck of getting the image data from system memory to video memory. I was probably over-zealous in my previous approach. While asynchronous updates would be great, we could try to minimise those updates first. This is almost certainly something we should be doing anyway.

One of the ways we do this is by something that (confusingly) gets called ‘rotation’, in the source code. I mentioned scrolling before. To reiterate, we render to a larger buffer than is visible on the screen and when panning, we move that buffer and ask the content process to re-render the newly exposed area. We then update again when that’s finished. Hopefully, that happens quickly, but when it doesn’t, you may see some checker-boarding. When the content process re-renders, theoretically it only needs to re-render the newly exposed pixels, as it already has the rest of the page rendered. This could involve copying all the existing pixels upwards (assuming we’re scrolling downwards) and then rendering in the newly exposed area, but instead of doing this, we say ‘the origin of this buffer is now at these coordinates’, and we treat the buffer as if it wrapped around (thus ‘rotation’).

There are problems with this, however. For example, if you were to zoom into a rotated buffer whose rotation coordinates are visible on the screen, you may see a ‘seam’ at that position. Similarly, when re-using the existing pixels on the buffer, if the new scroll coordinates meant that the sample grid is no longer aligned with the previous sample grid, you may see odd artifacts on scaled images and text that was cut-off in a previous render. The following example demonstrates this:

The results of a misaligned sample grid

On the left is the original image (a checkerboard, purposefully chosen as it’s sort-of a worst case scenario), and on the right, the same image with a 1-pixel border added on the left and upper edges. They both have the same, bi-linear scale applied to them, and the border is then cropped on the right image. You can immediately see that the same image does not result, and putting them together draws extra attention to this. This is what happens when you try to combine the results of two sampling image operations that have misaligned sample grids.

The code makes some attempt at separating out situations where this will happen and marking them so that in those situations, rotation doesn’t occur and the entire buffer is re-rendered. I don’t know what assumptions you can make about cairo’s sampling, or indeed how we drive it to draw pages, but certainly this code is over-zealous with marking when resampling will occur. For example, we zoom pages to fit the width of the screen by default. And any zoom operation marks the surface to say it will be resampled. We also update the content process’s scroll co-ordinates every 20 pixels. So, for the overwhelmingly common case, we re-render the entire buffer every 20 pixels. On a dual-core (or more) machine, assuming your cores aren’t saturated, this doesn’t matter so much without hardware-acceleration, as the chrome process oughtn’t be affected by what’s happening in the content process, and when it finishes, it just does a simple page-flip anyway. Unfortunately, this isn’t the case in practice, I guess due to the memory bandwidth required to re-render such a large surface, and perhaps due to non-ideal scheduling (remember, these are guesses. I’ve been terribly lazy when it comes to testing these theories).

Even more unfortunately, this is a terrible hit for GL-accelerated layers, as we don’t do page-swapping, we do synchronous buffer uploads. Also, the default shadow layer size is 200%x300% of the visible area. So let’s say you have 1280×752 pixels visible (as is the case on a 1280×800 Honeycomb tablet), every 20 pixels you scroll you’re doing a synchronous, 9.4mb upload from system memory to ‘video memory’ (I put this in quotes, as I don’t want to go down the path of explaining shared memory architecture and how it ends up working on android. It would be long and I’d probably be wrong). Even worse, most android devices have a maximum texture size of 2048×2048, so we have to tile these textures – so you’re then splitting up these uploads, with texture-binds in-between, making it even slower.

Well, you might then say, “well at least in some cases you’ll still get the benefit of rotation, right?” Well, unfortunately, you’d be wrong. We disable buffer rotation entirely on shadow layers. So we have a number of problems here. I discovered this when I noticed how frequently we were doing whole-buffer updates, both on GL and software. The first thing I thought was to just disable the marking of possibly-resampling surfaces (you can do this by either not setting PAINT_WILL_RESAMPLE in BasicLayers.cpp, or ignoring it in ThebesLayerBuffer.cpp – you’ll notice that it checks MustRetainContent, which returns TRUE for shadow layers). This ought to get you the benefit of rotation, at the expense of some visible artifacing. The bug for enabling buffer rotation is here. But then I ran into this bug, which I fixed. This gets you buffer rotation being used more frequently with software rendering, but when using hardware acceleration, things now appear very broken. Doubly-so if you use tiles.

So next, I investigated why things were broken when using hardware acceleration. First was to alter desktop to use tiles. After doing this, and picking a small tile-size, I noticed that a lot of drawing was then broken. This ended up being this bug, which I fixed. Now more of the screen is visible, but rotation is still broken. This ended up being a two-fold problem. The first being that we don’t handle uploads to GL layers correctly when there’s rotation, and the second being we don’t handle rendering of rotated GL layers when we have tiles. I fix both of these in this bug.

So after hacking away the resampling check and fixing the various rendering bugs that rotation then exposes, you can see the benefit it would get you. Unfortunately, there’s still a lot of work to be done, and even when this works perfectly, it isn’t going to benefit all situations (we could still do with a fast-path for texture upload on android, and asynchronous updates or page-flipping). But on some sites (my own, for example, and my favourite test-site, engadget.com), the difference is pretty big. So four bugs fixed and a deeper knowledge of how layers are put together, I count this one as a success

]]>