Eyes Above The Waves

Robert O'Callahan. Christian. Repatriate Kiwi. Hacker.

Friday 13 May 2005

Rendering Web Page To Images

For a long time now, people have been asking for ways to use Gecko to render a Web page to an image. Creating thumbnails of a Web page is one common desire, but there are lots of potential uses, especially if the feature is available to scripts. I have implemented a new DOM API in 1.8/FF 1.1 that makes this possible. It builds on the canvas element that has recently been implemented in Gecko 1.8 and will be enabled by default soon. (My patch hasn't been checked in yet either, so you can't try this at home just yet.)

To demo this API I've implemented a very simple extension that displays a "thumbnail view" of the currently loaded page in your sidebar. Here's a screenshot. Below is the core source code for the extension. The extension itself is no big deal, and I'm hoping the wonderfully imaginative extension developer community will take this and run with it.



<?xml version="1.0"?>

<?xml-stylesheet href="chrome://global/skin" type="text/css"?>

<script type="application/x-javascript" src="chrome://thumbview/content/thumbview.js"/>
<vbox flex="1" id="before"/>
<html:canvas id="canvas"/>
<vbox flex="1" id="after"/>


function update() {
var w = content.innerWidth + content.scrollMaxX;
var h = content.innerHeight + content.scrollMaxY;
if (w > 10000) w = 10000;
if (h > 10000) h = 10000;

var container = document.getElementById("win");
var canvasW = container.boxObject.width;
var scale = canvasW/w;
var canvasH = Math.round(h*scale);

var canvas = document.getElementById("canvas");
canvas.style.width = canvasW+"px";
canvas.style.height = canvasH+"px";
canvas.width = canvasW;
canvas.height = canvasH;
var ctx = canvas.getContext("2d");
ctx.clearRect(0, 0, canvasW, canvasH);
ctx.scale(canvasW/w, canvasH/h);
ctx.drawWindow(content, 0, 0, w, h, "rgb(0,0,0)");

var NavLoadObserver = {
observe: function(aWindow)

function start() {
var obs = Components.classes["@mozilla.org/observer-service;1"].
obs.addObserver(NavLoadObserver, "EndDocumentLoad", false);

window.addEventListener("load", start, false);

Currently the drawWindow function can only be used by "chrome privileged" content, because untrusted Web content could abuse it in various ways. So extension authors and XUL application developers can use it, but normal Web pages cannot.

Update! I overhauled this entry significantly since we may not be adding a method to 'window' after all. The drawWindow method will be there though.


Doug Turner
A feature that has been missing for over 4+ years. Great work Roc!
Great to see this implemented! I'm sure we'll see loads of interesting extension ideas around this...
On a remotely related issue: Any chance you can fix https://bugzilla.mozilla.org/show_bug.cgi?id=204278 (XUL elements cannot be stacked on top of browser element) for Gecko 1.9, too? IMO, that's another bug that, when fixed, might have great impact on extensions... For example, Mouse Gestures might be able to stack a transparent canvas element on top of the browser, and display actual drawn mouse trails instead of inserting hundreds of tiny HTML elements into the web page (which is the only way to do it right now).
* Showing thumbnails of pages on the back & forward dropdown list
* Generating thumbnails of prefetched pages on google and showing them inline (greasmonkey style extension)
* A bugzilla helper extension, that would let you take a "screenshot" of the place in the page you feel is being mis-rendered.
* Bookmark gallery. When hitting ctrl+b the sidebar would not only show the bookmarks, but images of the pages too.
Another thing that I'd like to see is a binary application or an easy to use library that would generate images for you.
This could be a nice addition to gaim for example, which could show a thumbnail of a persons site in the "more information" dialog. Also any website could generate screenshots for their sites (see: http://dev.upian.com/hotlinks/)
Great, now I would like to see a thumbnail in the history, instead of the pagetitle only.
That would be a great feature ... it is easier to remember a visited page by seeing the thumbnail than only the title ... extension idea (I would make it myself if i could) ?
Daniel Glazman
This is a _superb_ work, roc. In a former professional life, I was the CTO of a web agency and I can tell you web agencies are going to _love_ you if they can generate and save (can you save the bitmap right now?) such a thumbview from within the browser, w/o external tool.
Once the core is checked in, I strongly recommend making your extension an official extension to Firefox released by the Mozilla Foundation. This is awesome.
The most immediate thought for an extension that comes to mind: replacing the tab bar with a tab-thumbnail side bar.
Robert O'Callahan
jens.b: that should get fixed in 1.9, but not 1.8.
mvir, glazman: we should be able to do what you want once we have the ability to save the contents of a canvas. The plan is to have a .toDataURL() method that returns a data: URL containing a base64 encoded PNG. We should be able to get that into 1.8.
One thing that's not obvious here is that windows can be transparent, and drawWindow respects/captures the transparency. (Use "rgba(0,0,0,0)" as the background color.)
Another use of drawWindow is that it extends the power of the canvas element by giving you a way to conveniently draw certain kinds of content into a canvas --- e.g., text, or even SVG!
Da Scritch
But ? That means that future versions of Minimo could have ... full zoomable pages as Opera ?
Aooouuuuuuuuhhhhhh !
Robert O'Callahan
no. interactive zooming requires different technology. That's where our cairo work comes in.
This is what I meant:
Colin Coghill
Excellent! This paves the way to automatic site-map generators (to make posters to stick on one's wall).
Stick a spider and thumbnailer onto one of the graph layout packages and much fun can be had
I had a rough one going with KHTML and graphviz once, but gave up because at the time KHTML didn't render well enough. But Gecko doing it would rock!
- Colin
Rory Parle
Wouldn't it be cool to have the whole page in miniature in the sidebar with a rectangular border marking the visible portion of the page (the part that's in the main window). Then you could scroll around the page by moving that rectangular border around the thumbnail.
Robert Accettura
I think i want to use this for reporter in the future.
This looks like it has so much potential for tracking layout regressions.
Awesome work roc!
Any idea when this will be present in Nightlies?
Michael Newton
What I want, on a related note, is some way I can use Gecko to replace html2ps!
christian biesinger
will you add a way to let extension authors know when a page is completely done loading, i.e. when the background images are there as well?
Will this work on a Linux box without an X server? Or do you need an X server to do the pixel rendering? Sorry if this is a naive question, but it's a huge problem when trying to create thumbnails with khtml.
Robert O'Callahan
jed: within a few days, perhaps
michael: someone could write a remote-controllable XUL app for this
biesi: me? no. But I'm sure you could do this yourself :-). (BTW the GECKO_FORCE_PAINT_ONLOAD environment variable forces onload to be blocked by CSS background image loads.)
Nelson: It needs X. At some point we'll land a version of tetron's null-widget code so that you can run apps without X.
Chris McElligott
Wow very nice indeed, if only we could have Omniweb like Tabs for FireFox, that would be really awesome.
Somebody needs to get onto this and fast!
christian biesinger
>variable forces onload to be blocked by CSS
>background image loads.)
last I checked, that environment variable did two things: not only did it it make images block onload, it also dumped each page to a file when done loading... (http://lxr.mozilla.org/seamonkey/source/layout/base/nsDocumentViewer.cpp#1024)
Robert O'Callahan
I know, I'm just saying that that shows how to get CSS background image loads to block onload.
I would like to see this used to improve search. A rendered page gives a lot of context and information than reading plain html. I can see a program which identifies large blocks of color and high contrast areas, and then OCR-ing them to get a better understanding of a website's contents, as well as relative importances.
Or how about audio browsers for the visually impaired using the generated image and knowledge of the underlying HTML to let users navigate the page "visually"? That should work even on websites that were not designed for accessibility.
Roc, do you know if this is in DeerPark Alpha 1?
I was thinking about taking your code and do something interesting with it...
Robert O'Callahan
It is in Deer Park Alpha 1.
How long until spammers combine this technology with OCR, to extract email addresses that are shown as image(s) or generated by javascript? :-(
Jon Hicks
I'm with Chris - I'd love to see a Firefox sidebar extension that allowed Omniweb style tabs
Ted Mielczarek
FYI, this was broken in Deer Park Alpha 1. It should be fixed in the next nightly build. (Thanks biesi!)
the relevant bug
Robert O'Callahan
ChrisH: if you can call drawWindow, then you could just crawl the window's DOM and get the email addresses directly without OCR. There is no new issue here. (Especially because currently Web content can't use this API.)
Robert O'Callahan
Ted: thanks for that! I don't know why it worked in my build. I think it also worked in ben's build...
Erik Arvidsson
I see that you've added a drawWindow. I think it would be pretty useful to be able to draw a DOM Element as well. That would make the canvas a lot more flexible.
Is there a reason why this only works from chrome? Wouldn't a same origin check be sufficient?
Robert O'Callahan
Drawing a DOM element is a bit hard because DOM elements are not necessarily rectangular. Suppose you have a window in a paginated context or with a column, and an inline element that breaks across a page or column boundary, then the different pieces of the element could be anywhere. What do you draw in that case? Yes, it would be useful but it's hard to specify and hard to implement.
In many cases you could just clone the content into its own hidden IFRAME and render that.
Jeremy Dunck
The plan is to have a .toDataURL() method that returns a data: URL containing a base64 encoded PNG.
In Moz/FF, is there a practical limit to the length of data: URL? I know spec says something like 2KB, but I've seen larger, and it's pertinent for Greasemonkey.
Robert O'Callahan
I've made data: URLs that were hundreds of kilobytes long and they worked.
Is there any way to implement this as a php feature in a script?
I'm new to this so forgive me if my question sounds incredibly dumb. Where can I find the drawWindow API/plugins and how do I install it? Can anyone point me to some general guide to do this sort of things?
I have a project that can really use this feature. Thank you in advance!
Robert O'Callahan
Cirius: it runs on the client only.
luping: You just need one of the newest "Deer Park" Firefox builds.
since FireFox 1.1 shows a thumbnail of an image in the tab is it possible for me to specify an image to be used in the tab on my page? i'm not talking about the favicon image, i mean making firefox use an image that displays in the page as the tab icon. so each page will have its own icon.
Robert O'Callahan
You can use a tag in your page to specify a per-page icon.