Eyes Above The Waves

Robert O'Callahan. Christian. Repatriate Kiwi. Hacker.

Tuesday 8 July 2008

Using Arbitrary Elements As Paint Servers

The latest feature in my bling branch is the ability to use any element as a paint-server CSS background for another element.

There are a few motivations for this feature. Probably the biggest usage of the canvas "drawWindow" API extension in Mozilla is to create thumbnails of Web content. The problem is, drawWindow necessarily creates "snapshot" thumbnails. Wouldn't it be cooler if there was an easy way to create live thumbnails --- essentially, extra live viewports into Web content? Now there is. It should be pretty easy to add this to S5 to get slide thumbnails, for example.

Another feature that's popular these days is reflections. With element-as-background, plus a small dose of transforms and masks, reflections are easy. A while ago Hyatt introduced a feature in Webkit to use a <canvas> as a CSS background; that falls out as a special case of element-as-background.

So here's an example:

An HTML element with a rotating canvas background

And here's the markup:

p { background:url(#d); }
<body style="background:yellow;">
<p style="width:60%; border:1px solid black; margin-left:100px;">
"Lorem ipsum dolor sit amet, consectetur adipisicing elit, sed do eiusmod tempor
incididunt ut labore et dolore magna aliqua. Ut enim ad minim veniam, quis nostrud
exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat. Duis aute
irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla
pariatur. Excepteur sint occaecat cupidatat non proident, sunt in culpa qui officia
deserunt mollit anim id est laborum."
<canvas id="d" width="50" height="50"></canvas>
var d = document.getElementById("d");
var iteration = 0;
function iterate() {
var ctx = d.getContext("2d");
ctx.clearRect(0, 0, 50, 50);
ctx.fillStyle = "lime";
ctx.fillRect(-10, -10, 20, 20);
setTimeout(iterate, 10);

Unlike SVG paint servers, elements-as-backgrounds have an intrinsic size. Staying consistent with my earlier work on SVG effects for HTML, I define the intrinsic size as the bounding box of the border-boxes for the element.

As with SVG paint servers, an element-as-background is subject to all CSS background effects, including 'background-repeat' as in the example above.

Of course, the first thing any self-respecting computer scientist will think of when they read about this feature is, "can I exploit infinite recursion to create a hall-of-mirrors effect or sell exploits to gangsters?" No. We refuse to render an element-as-background if we're already in the middle of rendering the same element-as-background.

The builds I linked to in my previous post contain this feature. I've uploaded the above example and the reflection demo.

The next thing I have to do is to write up a spec proposal for all this work and get it discussed by the CSS and SVG working groups. Based on that feedback we'll figure out the best way to deliver this functionality in Gecko. Unfortunately, the approach of "make existing syntax applicable in more situations" is not amenable to using vendor prefixes to isolate experimental features.


Dave Hyatt
Yeah I like this idea, and it does work as a more generalized version of the canvas extension I did. I think you need to be able to hide the content... you almost want something like a defs element to wrap presentational markup like this (e.g., arbitrary elements that will serve as masks etc.).
Maybe display:none could just allowed on the element and not prevent it from rendering in other contexts.
Dave Hyatt
Maybe we should create a new display type in CSS for these types of graphical definitions... e.g.,
display: paint-server
or something like that. :)
Robert O'Callahan
It's easy enough to wrap the canvas in an overflow:hidden, height:0 element. Arguably, that's a hack.
display:none would suck for us because we won't create a frame tree (i.e. renderobject tree) for such elements. In this implementation they won't work as background sources.
When we do external references, you'd be able to reference an element in an external document, which would be quite clean. We'd want an API to access the DOM of such documents, but that's something we're going to want anyway.
Robert O'Callahan
Another thing that's currently lacking is the ability to refer to an element of a subdocument (e.g. an IFRAME). It would be easy to implement, the problem is devising syntax. Maybe something like this:
Where #a refers to element with ID 'a' in the styled document, which in this case is an element with a subdocument, and #b refers to element with ID 'b' inside that subdocument. This extension would also cover the issue that there is currently no way for a CSS stylesheet to reference an element in the styled document. However, this syntax would not let you reference an element in a subdocument of an external document ... sigh.
Michael K.
Isn't ideally the place to put elements that are not part of the document? I'm sure it doesn't work that way in the real world right now, but ideally.
Anyway, I really like the flexibility you gain with this feature. And unlike the SVG stuff, this would work in regular HTML right now.
Robert O'Callahan
The problem with is that those elements are display:none, which means that they don't have frame tree/render tree. More abstractly, display:none elements don't have a size.
Would background:element(url(ext.html#a#b)) work?
Very cool stuff. I can't believe some of this might get into 3.1.
Michael K.
My and roc's comments were about <head> if you were wondering. The parser stripped it out.
Wouldn't it be cooler if there was an easy way to create live thumbnails --- essentially, extra live viewports into Web content?
Yes, especially if there was support for background-size.
Jonas Sicking
This is rockin'!
I do like hyatts idea of a display: paint-server frame type that doesn't affect the rest of the frame tree, but that still allows the element to be used as a paint server. This could be useful to do a 'proper' implementation of svg:defs too.
Such a frame would probably behave like a block except maybe shrink-to-fit rather than taking up maximum width. And its size would be unaffected by the size of the page.
An alternative to explicitly having "display: paint-server", would be to say that the nearest "display:none" parent of a paint server is automatically given a paint-server frame. This avoids having to add explicit CSS syntax, and also covers the case where you have a paint server somewhere in the DOM which as part of the UI for the page is occasionally set to display:none.
None of this would be needed for the initial release of course.
Dave Hyatt
display: none would suck for us in WebKit too for the exact same reason (no frame tree).
Myk Melez
If it were possible to reference other documents, that would make the Personas extension easier to implement and much more powerful.
Dave Hyatt
I don't think this feature necessarily obsoletes the idea of CSS canvas though. For example being able to point to an image element in the document by ID doesn't obsolete the idea of just referencing that raw image directly.
The same is true for direct access to the canvas buffer (and you avoid the need for an additional element in your DOM).
You can also reuse the actual canvas image buffer when tiling, although I suppose you could optimize the paint server version to use canvas or image element image buffers as long as no scale is in effect and no border/padding/masks are present.
Robert O'Callahan
To me, having the element in the DOM (possibly the XBL shadow DOM, in an ideal world) feels better than having it in a special new namespace like the one that getCSSCanvas provides.
In practice I suspect slightly more direct access isn't going to matter much. The overhead is small. Doing the optimization you suggest would not be completely trivial since with element-as-background, styles applying to the canvas element are applied when rendering it as a background, so before we took direct access to the canvas buffer we'd have to check there were no borders, no background, no opacity, etc. Doable, but I wouldn't bother without a testcase showing it's a big win.
Dave Hyatt
On OS X it's a huge win to repeatedly draw CGImageRefs (and CGLayerRefs), since you end up using the video card. Similarly CGPatterns are much faster if they are simply tiling CGImageRefs (and not using custom tiling callback functions).
Robert O'Callahan
We don't use custom tiling callback functions here, they're only used for gradients in cairo. The only overhead here to be avoided by direct use of the canvas buffer is the creation of a temporary surface and a copy into that surface, and that only in the tile case.
An alternative optimization would be to cache a surface containing the rendering of the element subtree. That would make multiple references to a complex subtree super-fast and would also take care of the canvas case (except use more memory). Actually in the past we've even talked about a generalization of that using heuristics to cache renderings of arbitrary subtrees to speed up multi-layer animations. Joost's Gecko extensions implement something like that, albeit with author control over the caching.
Benjamin Otte
Yeah, author control over the caching will likely not work. From my Flash experience only very few people use MovieClip.cacheAsBitmap and if they do, they use it in the wrong place. Plus, caching benefits are very different between systems (don't cache as much on mobile).
Slightly similar for this topic, I've wondered about adding support to cairo for "meshes" (for lack of a better word). It would basically be a Cairo meta surface with a recorded set of rendering commands that you can then use as a normal surface (and with it: source pattern). That would sound like a natural part to move caching to.
would suck for us in WebKit too for the exact same reason not view to frame.. :D
Filim izle
you almost want something like a defs element to wrap presentational markup like this (e.g., arbitrary elements that will serve as masks etc.).
Ex: eAccelerator doesn't work with the SAPI you are using!
CSS seems like a wrong place for this. It should be canvas API.