For a long time now, people have been asking for ways to use Gecko to render a Web page to an image. Creating thumbnails of a Web page is one common desire, but there are lots of potential uses, especially if the feature is available to scripts. I have implemented a new DOM API in 1.8/FF 1.1 that makes this possible. It builds on the canvas element that has recently been implemented in Gecko 1.8 and will be enabled by default soon. (My patch hasn't been checked in yet either, so you can't try this at home just yet.)
To demo this API I've implemented a very simple extension that displays a "thumbnail view" of the currently loaded page in your sidebar. Here's a screenshot. Below is the core source code for the extension. The extension itself is no big deal, and I'm hoping the wonderfully imaginative extension developer community will take this and run with it.
thumbviewSidebar.xul
<?xml version="1.0"?>
<?xml-stylesheet href="chrome://global/skin" type="text/css"?>
<window
windowtype="global:thumbviewSidebar"
onclick="update()"
id="win"
xmlns:html="http://www.w3.org/1999/xhtml"
xmlns="http://www.mozilla.org/keymaster/gatekeeper/there.is.only.xul">
<script type="application/x-javascript" src="chrome://thumbview/content/thumbview.js"/>
<vbox flex="1" id="before"/>
<html:canvas id="canvas"/>
<vbox flex="1" id="after"/>
</window>
thumbview.js
function update() {
var w = content.innerWidth + content.scrollMaxX;
var h = content.innerHeight + content.scrollMaxY;
if (w > 10000) w = 10000;
if (h > 10000) h = 10000;
var container = document.getElementById("win");
var canvasW = container.boxObject.width;
var scale = canvasW/w;
var canvasH = Math.round(h*scale);
var canvas = document.getElementById("canvas");
canvas.style.width = canvasW+"px";
canvas.style.height = canvasH+"px";
canvas.width = canvasW;
canvas.height = canvasH;
var ctx = canvas.getContext("2d");
ctx.clearRect(0, 0, canvasW, canvasH);
ctx.save();
ctx.scale(canvasW/w, canvasH/h);
ctx.drawWindow(content, 0, 0, w, h, "rgb(0,0,0)");
ctx.restore();
}
var NavLoadObserver = {
observe: function(aWindow)
{
update();
}
};
function start() {
var obs = Components.classes["@mozilla.org/observer-service;1"].
getService(Components.interfaces["nsIObserverService"]);
obs.addObserver(NavLoadObserver, "EndDocumentLoad", false);
}
window.addEventListener("load", start, false);
Currently the drawWindow function can only be used by "chrome privileged" content, because untrusted Web content could abuse it in various ways. So extension authors and XUL application developers can use it, but normal Web pages cannot.
Update! I overhauled this entry significantly since we may not be adding a method to 'window' after all. The drawWindow method will be there though.
A feature that has been missing for over 4+ years. Great work Roc!
ReplyDeleteGreat to see this implemented! I'm sure we'll see loads of interesting extension ideas around this...
ReplyDeleteOn a remotely related issue: Any chance you can fix https://bugzilla.mozilla.org/show_bug.cgi?id=204278 (XUL elements cannot be stacked on top of browser element) for Gecko 1.9, too? IMO, that's another bug that, when fixed, might have great impact on extensions... For example, Mouse Gestures might be able to stack a transparent canvas element on top of the browser, and display actual drawn mouse trails instead of inserting hundreds of tiny HTML elements into the web page (which is the only way to do it right now).
Awsome.
ReplyDelete* Showing thumbnails of pages on the back & forward dropdown list
* Generating thumbnails of prefetched pages on google and showing them inline (greasmonkey style extension)
* A bugzilla helper extension, that would let you take a "screenshot" of the place in the page you feel is being mis-rendered.
* Bookmark gallery. When hitting ctrl+b the sidebar would not only show the bookmarks, but images of the pages too.
Another thing that I'd like to see is a binary application or an easy to use library that would generate images for you.
This could be a nice addition to gaim for example, which could show a thumbnail of a persons site in the "more information" dialog. Also any website could generate screenshots for their sites (see: http://dev.upian.com/hotlinks/)
Great, now I would like to see a thumbnail in the history, instead of the pagetitle only.
ReplyDeleteThat would be a great feature ... it is easier to remember a visited page by seeing the thumbnail than only the title ... extension idea (I would make it myself if i could) ?
This is a _superb_ work, roc. In a former professional life, I was the CTO of a web agency and I can tell you web agencies are going to _love_ you if they can generate and save (can you save the bitmap right now?) such a thumbview from within the browser, w/o external tool.
ReplyDeleteOnce the core is checked in, I strongly recommend making your extension an official extension to Firefox released by the Mozilla Foundation. This is awesome.
The most immediate thought for an extension that comes to mind: replacing the tab bar with a tab-thumbnail side bar.
ReplyDeletejens.b: that should get fixed in 1.9, but not 1.8.
ReplyDeletemvir, glazman: we should be able to do what you want once we have the ability to save the contents of a canvas. The plan is to have a .toDataURL() method that returns a data: URL containing a base64 encoded PNG. We should be able to get that into 1.8.
One thing that's not obvious here is that windows can be transparent, and drawWindow respects/captures the transparency. (Use "rgba(0,0,0,0)" as the background color.)
Another use of drawWindow is that it extends the power of the canvas element by giving you a way to conveniently draw certain kinds of content into a canvas --- e.g., text, or even SVG!
But ? That means that future versions of Minimo could have ... full zoomable pages as Opera ?
ReplyDeleteAooouuuuuuuuhhhhhh !
no. interactive zooming requires different technology. That's where our cairo work comes in.
ReplyDeleteThis is what I meant:
ReplyDeletehttp://apple.slashdot.org/apple/04/03/31/0513245.shtml?tid=126&tid=185&tid=95
http://www.acm.uiuc.edu/macwarriors/projects/trailblazer/
Excellent! This paves the way to automatic site-map generators (to make posters to stick on one's wall).
ReplyDeleteStick a spider and thumbnailer onto one of the graph layout packages and much fun can be had
I had a rough one going with KHTML and graphviz once, but gave up because at the time KHTML didn't render well enough. But Gecko doing it would rock!
- Colin
Wouldn't it be cool to have the whole page in miniature in the sidebar with a rectangular border marking the visible portion of the page (the part that's in the main window). Then you could scroll around the page by moving that rectangular border around the thumbnail.
ReplyDeleteI think i want to use this for reporter in the future.
ReplyDeleteThis looks like it has so much potential for tracking layout regressions.
Awesome work roc!
Any idea when this will be present in Nightlies?
ReplyDeleteWhat I want, on a related note, is some way I can use Gecko to replace html2ps!
ReplyDeletewill you add a way to let extension authors know when a page is completely done loading, i.e. when the background images are there as well?
ReplyDeleteWill this work on a Linux box without an X server? Or do you need an X server to do the pixel rendering? Sorry if this is a naive question, but it's a huge problem when trying to create thumbnails with khtml.
ReplyDeletejed: within a few days, perhaps
ReplyDeletemichael: someone could write a remote-controllable XUL app for this
biesi: me? no. But I'm sure you could do this yourself :-). (BTW the GECKO_FORCE_PAINT_ONLOAD environment variable forces onload to be blocked by CSS background image loads.)
Nelson: It needs X. At some point we'll land a version of tetron's null-widget code so that you can run apps without X.
Wow very nice indeed, if only we could have Omniweb like Tabs for FireFox, that would be really awesome.
ReplyDeleteSomebody needs to get onto this and fast!
>(BTW the GECKO_FORCE_PAINT_ONLOAD environment
ReplyDelete>variable forces onload to be blocked by CSS
>background image loads.)
last I checked, that environment variable did two things: not only did it it make images block onload, it also dumped each page to a file when done loading... (http://lxr.mozilla.org/seamonkey/source/layout/base/nsDocumentViewer.cpp#1024)
I know, I'm just saying that that shows how to get CSS background image loads to block onload.
ReplyDeleteI would like to see this used to improve search. A rendered page gives a lot of context and information than reading plain html. I can see a program which identifies large blocks of color and high contrast areas, and then OCR-ing them to get a better understanding of a website's contents, as well as relative importances.
ReplyDeleteOr how about audio browsers for the visually impaired using the generated image and knowledge of the underlying HTML to let users navigate the page "visually"? That should work even on websites that were not designed for accessibility.
Roc, do you know if this is in DeerPark Alpha 1?
ReplyDeleteI was thinking about taking your code and do something interesting with it...
Thanks
It is in Deer Park Alpha 1.
ReplyDeleteHow long until spammers combine this technology with OCR, to extract email addresses that are shown as image(s) or generated by javascript? :-(
ReplyDeleteI'm with Chris - I'd love to see a Firefox sidebar extension that allowed Omniweb style tabs
ReplyDeleteFYI, this was broken in Deer Park Alpha 1. It should be fixed in the next nightly build. (Thanks biesi!)
ReplyDeletethe relevant bug
ChrisH: if you can call drawWindow, then you could just crawl the window's DOM and get the email addresses directly without OCR. There is no new issue here. (Especially because currently Web content can't use this API.)
ReplyDeleteTed: thanks for that! I don't know why it worked in my build. I think it also worked in ben's build...
ReplyDeleteI see that you've added a drawWindow. I think it would be pretty useful to be able to draw a DOM Element as well. That would make the canvas a lot more flexible.
ReplyDeleteIs there a reason why this only works from chrome? Wouldn't a same origin check be sufficient?
Drawing a DOM element is a bit hard because DOM elements are not necessarily rectangular. Suppose you have a window in a paginated context or with a column, and an inline element that breaks across a page or column boundary, then the different pieces of the element could be anywhere. What do you draw in that case? Yes, it would be useful but it's hard to specify and hard to implement.
ReplyDeleteIn many cases you could just clone the content into its own hidden IFRAME and render that.
ROC:
ReplyDelete"
The plan is to have a .toDataURL() method that returns a data: URL containing a base64 encoded PNG.
"
In Moz/FF, is there a practical limit to the length of data: URL? I know spec says something like 2KB, but I've seen larger, and it's pertinent for Greasemonkey.
Thanks!
I've made data: URLs that were hundreds of kilobytes long and they worked.
ReplyDeleteIs there any way to implement this as a php feature in a script?
ReplyDeleteThanks
I'm new to this so forgive me if my question sounds incredibly dumb. Where can I find the drawWindow API/plugins and how do I install it? Can anyone point me to some general guide to do this sort of things?
ReplyDeleteI have a project that can really use this feature. Thank you in advance!
Cirius: it runs on the client only.
ReplyDeleteluping: You just need one of the newest "Deer Park" Firefox builds.
since FireFox 1.1 shows a thumbnail of an image in the tab is it possible for me to specify an image to be used in the tab on my page? i'm not talking about the favicon image, i mean making firefox use an image that displays in the page as the tab icon. so each page will have its own icon.
ReplyDeleteYou can use a tag in your page to specify a per-page icon.
ReplyDelete