Eyes Above The Waves

Robert O'Callahan. Christian. Repatriate Kiwi. Hacker.

Wednesday 14 February 2007

Offline Web Applications

Chris has been working on adding offline support to Zimbra using Firefox's nascent offline capabilities. The goal is to test our APIs in the context of a large complex AJAX application, to make sure they work and are a good fit for the task He's made considerable progress and just posted a nice demo of reading email offline in Zimbra.

Ironically I think it works so well that the demo is not very exciting. You go offline, everything continues to work --- ho hum, move along, nothing to see here. Boring perhaps, but it's exactly the right user experience.

As Chris mentioned, there are four pieces to our offline puzzle:

Note that two of the pieces already have WHATWG specs, and the jar: URI scheme is already a de facto standard used in a variety of products. The only really new part is using <link rel="offline-resource"> to put resources in a persistent "offline cache" that won't accidentally evict your application. That is quite simple and once we're confident we have the right semantics for it, we'll definitely try to get it standardized somewhere. We're about pushing the Web forward, not just Firefox, and I hope other browsers support these APIs ASAP if they prove popular.

Putting these pieces together, Web application authors can add offline support to their applications in a very incremental way. The first thing you do is collect all static resources deployed to the Web server that you'll need offline and arrange for them to be loaded into the user's offline cache. For performance and consistency reasons, and also to minimize the number of tags required, your best bet is to roll them all into a single JAR file and make that the offline resource. (The "start page" also needs to link to itself as an offline-resource to ensure it will be available offline.) Now, when the user visits your page and signals that they want the application to be available offline, the resources will be automatically downloaded and stored. (The exact UI for this is yet to be determined; I've proposed having "bookmark the page" be that signal.) On each subsequent visit to the start page, Firefox will revalidate the cached resources and update them if they've changed on the server. This should be enough to ensure your application can at least get started while the user is offline.

Most Web apps need to talk to some server. When the browser is offline, they can't. It's up to the application developer to detect offline status using the provided APIs and decide how to behave while offline. Some functionality, such as instant messages, should just be disabled. In other cases, where the app would normally retrieve data from the server, it instead might grab some previously preloaded data. This would work well for email, calendar, CMS, and other applications. Of course the data has to be stored somewhere, and cookies won't cut it, which is where WHATWG client-side storage comes in (or another client-side storage solution, such as Flash's, if you prefer). Similarly, instead of sending data to the server, it will have to be queued and sent later when the user gets back online.

This approach to offline Web apps has some major advantages. The user isn't required to download and install anything. In fact, no trust decision is required; the application remains sandboxed exactly as if online. In fact, there's potentially no new UI required at all. The footprint requirements are small. The offline user experience is very smooth; the address in the URL bar remains the same, and the same bookmarks or shortcuts will access the application. Dynamic online/offline switching is supported. It's very incremental; there's no new tier you have to suddenly add, and no new programming model. AJAX applications tend to move logic from the server to the client and this fits right in with offline support.

Of course, this approach is also not perfect. The main issue that I see right now is that some server-based functionality, such as full-text search, may just not perform well in the browser. (However, there may be ways around that, such as uploading an index to the client, or doing the computation in a Java applet loaded from the offline cache.) I'm glad to see other approaches growing up, such as the Dojo Offline Toolkit; hopefully we'll have a range of solutions to fit all needs.

There are some things we're deliberately not doing. People have asked for richer client-side data models such as SQL or some sort of automatic synchronization. I think it's premature to expose APIs for that sort of thing (even though it would be easy for us to expose SQL, since we already embed SQLLite). I'm skeptical that we can choose a synchronization model, or even a SQL dialect, that suits everyone. SQL also has the problem that exposing SQL to untrusted code could have all kinds of difficult-to-forsee consequences. I think if we try to choose a solution too early, we'll choose something inappropriate for most applications and it will become dead weight that everyone works around. For now I think we need to watch and wait and see if application needs start to converge. Of course, Javascript libraries are also an option.

One API that we are thinking about adding is support for script-controlled loading of resources into the offline cache. This needs to be designed in conjunction with a policy for deleting resources from the offline cache, something we haven't settled on yet.

I'm really excited about seeing this taking shape. Thanks to Chris and Dave Camp, who's doing the offline cache work on the Gecko side.


Apostolos Tsakpinis
It may be tempting to add support for SQL or another data retrieval mechanism, but I totally agree with you on your decision to not include one. This should be the job of javascript libraries, period.
I also think that creating an API to add/remove resources from the offline cache is a must. and JARs may be enough for an application like Zimbra, where assets like images don't change regularly, but when you go to other classes of applications not having an API will become a great issue. Having to include tons of link tags or having to create 100 different jar files will be a great hassle. Imagine having an application like phpMyAdmin which supports infinite amount of UI themes.
On the UI side, I think adding some off-line capabiligy sign is worth of considering to promote that feature at least.
I don't understand how "offline web applications" can be so important - the computer world is evolving to an "always connected" principle, isn't it?
What is to prevent website owners who have no intention of making offline apps using as a way of saying "Yes, I'd like my website to load much faster in the user's browser next time. Please store all of my images and JS for an indefinite amount of time, rather than getting evicted by normal cache eviction mechanisms"?
Also, are there per-site limits on the amount of cache they can use?
Robert O'Callahan
Gerv: hence "when the user visits your page and signals that they want the application to be available offline ..."
There are currently no per-site limits on the amount of cache they can use, but I think we should add some.
Diego: the reality is that we're far away from that. And in many situations where they *could* be connected, people will choose not to be because of cost or inconvenience. It varies depending on the kind of user.
John Silvestri
One thing I'm curious about - I trust this won't maintain per session cookies, and will somehow be up to the application to determine whether or not to allow access to the application? (Maybe via a trick of DOM storage?)
i.e. It would be quite undesirable to have Gmail show all of your e-mail if you closed out of the browser, and later, whilst offline, someone went there without authenticating.
And to the question about 'why do we need this?' - I'm pretty sure the answer is 'software as a service.' The big downside that /everyone/ said about things like Google Docs, Gmail, et. al, is that "it's useless if you're offline."
Personally, I think this would be fantastic for certain resources like Google Maps - let's say that you looked at all the roads you'll be taking (i.e. panned & loaded the map tiles), and you hit the road...you might want to refer to the map again, but are offline. I know I've printed more than a few pages that I could have done without, simply because I didn't want to get lost. ;)
*lightbulb goes on*
Offline Web Applications will save the planet! :D
Robert O'Callahan
Applications could provide some simple access control. However, if you want to be able to read email offline, then it has to be stored on your computer. If you give someone access to your user account on your computer, they'll be able to read the data one way or another (or do something worse, like plant a keyboard listener). My advice is: don't do that :-)
Think of it another way: how is this different from a regular desktop email client like Outlook that stores email locally?
John Silvestri
It's no different (except in the eyes of regular users) - I think I failed to make my point clear. With Gmail, you have a choice when you log in - to save your password, or not to. If you opt not to save your password (the default), then when you close your browser, your session cookie is killed, and upon return, you cannot see the e-mail - you are presented with a login screen.
The question I'm posing is this: How does Gmail, or some similar app, know that you are offline and to let you in, as opposed to making it easy to dig through resources without network authentication?
I should suppose that closing the browser would kill the cookie, and Gmail's fancy offline JS would know not to permit access - but I expect that it would require storing something to say what cookie to permit. The simpler answer is probably that not checking 'remember my password' would disable the offline mode.
Robert O'Callahan
> The simpler answer is probably that not checking
> 'remember my password' would disable the offline
> mode.
I think that's a great idea.
Mikael Bergkvist
One question that immidiatly pops up is whether you will support a mobile version of this?
John Ballinger
Hi Robert,
I am trying to find you contact details but it doesnt seem that straight forward. Is it possible you could email me back?
My partner is a journalist at computerworld (.co.nz) and she would like to write an article about firefox and I said that you we meet at Baacamp and she should contact you if that was ok. Can you email or somethings so I can get her to contact you?
Cheers, John Ballinger
Brendan Eich
Diego: I wanted to note that we're far from the always-connected SciFi dreamworld. Even with pretty good coverage, and without getting into a car, one experiences intermittent connectivity. Offline can be a short-lived state!
Mikael: if someone wants to build this stuff into Minimo, and has the chunky phone to run Minimo, it should all work. No one is focusing on that at the moment, but a volunteer or three could run with it. Minimo is not something Mozilla is investing in, because of "structural" problems to do with phones (basically, carriers control software, users don't download in big numbers; screen's too small; phone speed and memory size are too small for most users to run Minimo). But someone could push on the idea.
I really like the idea of static content delivered by JAR - but then why not make it available as a mechanism for online content as well? In the web application space (ie Zimbra class) theres a helluva lot of http handshaking to bring down all the static content - all the images, htm docs, js, css. This is before the app can even be used.
If zimbra could package up their entire app as a versioned jar, theres one version check and one (big) download.
This would really help those class of apps (again Zimbra) that are mainly static htm/css/js web clients, with dynamic data-only calls (eg soap/json with the client binding the data itself).
Thats what I want!
Robert O'Callahan
> why not make it available as a mechanism for
> online content as well?
It is. In fact, it has been for years (in Firefox). However Chris tells me that jar: performance is pretty bad in Firefox 2, much better on trunk/FF3.
Ok - I didn't know that. Thats something I would like to start playing around with. Although I can see if performance wasn't up to par pre FF3 trunk builds, there wouldn't have been much point. But if thats now sorted, it does create new options :-)
Sam Hasler
Maybe you should have to put in your master password to access some offline storage, and offline content could signal how sensitive it is.
how we can create a web page like yahoo in the manner of checking the password.What software we have to use for it.