Saturday, 14 May 2005

Rendering Web Page To Images

For a long time now, people have been asking for ways to use Gecko to render a Web page to an image. Creating thumbnails of a Web page is one common desire, but there are lots of potential uses, especially if the feature is available to scripts. I have implemented a new DOM API in 1.8/FF 1.1 that makes this possible. It builds on the canvas element that has recently been implemented in Gecko 1.8 and will be enabled by default soon. (My patch hasn't been checked in yet either, so you can't try this at home just yet.)

To demo this API I've implemented a very simple extension that displays a "thumbnail view" of the currently loaded page in your sidebar. Here's a screenshot. Below is the core source code for the extension. The extension itself is no big deal, and I'm hoping the wonderfully imaginative extension developer community will take this and run with it.



thumbshot.png


thumbviewSidebar.xul


<?xml version="1.0"?>

<?xml-stylesheet href="chrome://global/skin" type="text/css"?>

<window
windowtype="global:thumbviewSidebar"
onclick="update()"
id="win"
xmlns:html="http://www.w3.org/1999/xhtml"
xmlns="http://www.mozilla.org/keymaster/gatekeeper/there.is.only.xul">
<script type="application/x-javascript" src="chrome://thumbview/content/thumbview.js"/>
<vbox flex="1" id="before"/>
<html:canvas id="canvas"/>
<vbox flex="1" id="after"/>
</window>


thumbview.js


function update() {
var w = content.innerWidth + content.scrollMaxX;
var h = content.innerHeight + content.scrollMaxY;
if (w > 10000) w = 10000;
if (h > 10000) h = 10000;

var container = document.getElementById("win");
var canvasW = container.boxObject.width;
var scale = canvasW/w;
var canvasH = Math.round(h*scale);

var canvas = document.getElementById("canvas");
canvas.style.width = canvasW+"px";
canvas.style.height = canvasH+"px";
canvas.width = canvasW;
canvas.height = canvasH;
var ctx = canvas.getContext("2d");
ctx.clearRect(0, 0, canvasW, canvasH);
ctx.save();
ctx.scale(canvasW/w, canvasH/h);
ctx.drawWindow(content, 0, 0, w, h, "rgb(0,0,0)");
ctx.restore();
}

var NavLoadObserver = {
observe: function(aWindow)
{
update();
}
};

function start() {
var obs = Components.classes["@mozilla.org/observer-service;1"].
getService(Components.interfaces["nsIObserverService"]);
obs.addObserver(NavLoadObserver, "EndDocumentLoad", false);
}

window.addEventListener("load", start, false);


Currently the drawWindow function can only be used by "chrome privileged" content, because untrusted Web content could abuse it in various ways. So extension authors and XUL application developers can use it, but normal Web pages cannot.



Update! I overhauled this entry significantly since we may not be adding a method to 'window' after all. The drawWindow method will be there though.

38 comments:

  1. A feature that has been missing for over 4+ years. Great work Roc!

    ReplyDelete
  2. Great to see this implemented! I'm sure we'll see loads of interesting extension ideas around this...
    On a remotely related issue: Any chance you can fix https://bugzilla.mozilla.org/show_bug.cgi?id=204278 (XUL elements cannot be stacked on top of browser element) for Gecko 1.9, too? IMO, that's another bug that, when fixed, might have great impact on extensions... For example, Mouse Gestures might be able to stack a transparent canvas element on top of the browser, and display actual drawn mouse trails instead of inserting hundreds of tiny HTML elements into the web page (which is the only way to do it right now).

    ReplyDelete
  3. Awsome.
    * Showing thumbnails of pages on the back & forward dropdown list
    * Generating thumbnails of prefetched pages on google and showing them inline (greasmonkey style extension)
    * A bugzilla helper extension, that would let you take a "screenshot" of the place in the page you feel is being mis-rendered.
    * Bookmark gallery. When hitting ctrl+b the sidebar would not only show the bookmarks, but images of the pages too.
    Another thing that I'd like to see is a binary application or an easy to use library that would generate images for you.
    This could be a nice addition to gaim for example, which could show a thumbnail of a persons site in the "more information" dialog. Also any website could generate screenshots for their sites (see: http://dev.upian.com/hotlinks/)

    ReplyDelete
  4. Great, now I would like to see a thumbnail in the history, instead of the pagetitle only.
    That would be a great feature ... it is easier to remember a visited page by seeing the thumbnail than only the title ... extension idea (I would make it myself if i could) ?

    ReplyDelete
  5. Daniel Glazman14 May 2005 09:29

    This is a _superb_ work, roc. In a former professional life, I was the CTO of a web agency and I can tell you web agencies are going to _love_ you if they can generate and save (can you save the bitmap right now?) such a thumbview from within the browser, w/o external tool.
    Once the core is checked in, I strongly recommend making your extension an official extension to Firefox released by the Mozilla Foundation. This is awesome.

    ReplyDelete
  6. The most immediate thought for an extension that comes to mind: replacing the tab bar with a tab-thumbnail side bar.

    ReplyDelete
  7. Robert O'Callahan14 May 2005 11:21

    jens.b: that should get fixed in 1.9, but not 1.8.
    mvir, glazman: we should be able to do what you want once we have the ability to save the contents of a canvas. The plan is to have a .toDataURL() method that returns a data: URL containing a base64 encoded PNG. We should be able to get that into 1.8.
    One thing that's not obvious here is that windows can be transparent, and drawWindow respects/captures the transparency. (Use "rgba(0,0,0,0)" as the background color.)
    Another use of drawWindow is that it extends the power of the canvas element by giving you a way to conveniently draw certain kinds of content into a canvas --- e.g., text, or even SVG!

    ReplyDelete
  8. But ? That means that future versions of Minimo could have ... full zoomable pages as Opera ?
    Aooouuuuuuuuhhhhhh !

    ReplyDelete
  9. Robert O'Callahan14 May 2005 11:45

    no. interactive zooming requires different technology. That's where our cairo work comes in.

    ReplyDelete
  10. This is what I meant:
    http://apple.slashdot.org/apple/04/03/31/0513245.shtml?tid=126&tid=185&tid=95
    http://www.acm.uiuc.edu/macwarriors/projects/trailblazer/

    ReplyDelete
  11. Colin Coghill14 May 2005 13:55

    Excellent! This paves the way to automatic site-map generators (to make posters to stick on one's wall).
    Stick a spider and thumbnailer onto one of the graph layout packages and much fun can be had
    I had a rough one going with KHTML and graphviz once, but gave up because at the time KHTML didn't render well enough. But Gecko doing it would rock!
    - Colin

    ReplyDelete
  12. Wouldn't it be cool to have the whole page in miniature in the sidebar with a rectangular border marking the visible portion of the page (the part that's in the main window). Then you could scroll around the page by moving that rectangular border around the thumbnail.

    ReplyDelete
  13. I think i want to use this for reporter in the future.
    This looks like it has so much potential for tracking layout regressions.
    Awesome work roc!

    ReplyDelete
  14. Any idea when this will be present in Nightlies?

    ReplyDelete
  15. What I want, on a related note, is some way I can use Gecko to replace html2ps!

    ReplyDelete
  16. christian biesinger15 May 2005 13:28

    will you add a way to let extension authors know when a page is completely done loading, i.e. when the background images are there as well?

    ReplyDelete
  17. Will this work on a Linux box without an X server? Or do you need an X server to do the pixel rendering? Sorry if this is a naive question, but it's a huge problem when trying to create thumbnails with khtml.

    ReplyDelete
  18. Robert O'Callahan15 May 2005 22:32

    jed: within a few days, perhaps
    michael: someone could write a remote-controllable XUL app for this
    biesi: me? no. But I'm sure you could do this yourself :-). (BTW the GECKO_FORCE_PAINT_ONLOAD environment variable forces onload to be blocked by CSS background image loads.)
    Nelson: It needs X. At some point we'll land a version of tetron's null-widget code so that you can run apps without X.

    ReplyDelete
  19. Wow very nice indeed, if only we could have Omniweb like Tabs for FireFox, that would be really awesome.
    Somebody needs to get onto this and fast!

    ReplyDelete
  20. christian biesinger17 May 2005 01:15

    >(BTW the GECKO_FORCE_PAINT_ONLOAD environment
    >variable forces onload to be blocked by CSS
    >background image loads.)
    last I checked, that environment variable did two things: not only did it it make images block onload, it also dumped each page to a file when done loading... (http://lxr.mozilla.org/seamonkey/source/layout/base/nsDocumentViewer.cpp#1024)

    ReplyDelete
  21. Robert O'Callahan17 May 2005 03:23

    I know, I'm just saying that that shows how to get CSS background image loads to block onload.

    ReplyDelete
  22. I would like to see this used to improve search. A rendered page gives a lot of context and information than reading plain html. I can see a program which identifies large blocks of color and high contrast areas, and then OCR-ing them to get a better understanding of a website's contents, as well as relative importances.
    Or how about audio browsers for the visually impaired using the generated image and knowledge of the underlying HTML to let users navigate the page "visually"? That should work even on websites that were not designed for accessibility.

    ReplyDelete
  23. Roc, do you know if this is in DeerPark Alpha 1?
    I was thinking about taking your code and do something interesting with it...
    Thanks

    ReplyDelete
  24. Robert O'Callahan3 June 2005 07:37

    It is in Deer Park Alpha 1.

    ReplyDelete
  25. How long until spammers combine this technology with OCR, to extract email addresses that are shown as image(s) or generated by javascript? :-(

    ReplyDelete
  26. I'm with Chris - I'd love to see a Firefox sidebar extension that allowed Omniweb style tabs

    ReplyDelete
  27. FYI, this was broken in Deer Park Alpha 1. It should be fixed in the next nightly build. (Thanks biesi!)
    the relevant bug

    ReplyDelete
  28. Robert O'Callahan4 June 2005 07:44

    ChrisH: if you can call drawWindow, then you could just crawl the window's DOM and get the email addresses directly without OCR. There is no new issue here. (Especially because currently Web content can't use this API.)

    ReplyDelete
  29. Robert O'Callahan4 June 2005 07:45

    Ted: thanks for that! I don't know why it worked in my build. I think it also worked in ben's build...

    ReplyDelete
  30. I see that you've added a drawWindow. I think it would be pretty useful to be able to draw a DOM Element as well. That would make the canvas a lot more flexible.
    Is there a reason why this only works from chrome? Wouldn't a same origin check be sufficient?

    ReplyDelete
  31. Robert O'Callahan7 June 2005 11:25

    Drawing a DOM element is a bit hard because DOM elements are not necessarily rectangular. Suppose you have a window in a paginated context or with a column, and an inline element that breaks across a page or column boundary, then the different pieces of the element could be anywhere. What do you draw in that case? Yes, it would be useful but it's hard to specify and hard to implement.
    In many cases you could just clone the content into its own hidden IFRAME and render that.

    ReplyDelete
  32. Jeremy Dunck15 June 2005 23:31

    ROC:
    "
    The plan is to have a .toDataURL() method that returns a data: URL containing a base64 encoded PNG.
    "
    In Moz/FF, is there a practical limit to the length of data: URL? I know spec says something like 2KB, but I've seen larger, and it's pertinent for Greasemonkey.
    Thanks!

    ReplyDelete
  33. Robert O'Callahan16 June 2005 00:37

    I've made data: URLs that were hundreds of kilobytes long and they worked.

    ReplyDelete
  34. Is there any way to implement this as a php feature in a script?
    Thanks

    ReplyDelete
  35. I'm new to this so forgive me if my question sounds incredibly dumb. Where can I find the drawWindow API/plugins and how do I install it? Can anyone point me to some general guide to do this sort of things?
    I have a project that can really use this feature. Thank you in advance!

    ReplyDelete
  36. Robert O'Callahan12 July 2005 20:48

    Cirius: it runs on the client only.
    luping: You just need one of the newest "Deer Park" Firefox builds.

    ReplyDelete
  37. since FireFox 1.1 shows a thumbnail of an image in the tab is it possible for me to specify an image to be used in the tab on my page? i'm not talking about the favicon image, i mean making firefox use an image that displays in the page as the tab icon. so each page will have its own icon.

    ReplyDelete
  38. Robert O'Callahan21 July 2005 21:22

    You can use a tag in your page to specify a per-page icon.

    ReplyDelete