Wednesday 26 July 2006
The Good, The Bad, And The Three-D
Last Friday I visited Linden Lab, makers of Second Life, to spend several hours talking with Callum, author of UBrowser. UBrowser is an interesting Gecko embedding that needs to direct all rendering to an offscreen area so it can be transformed by OpenGL for display. A surprising number of people have contacted me over time requesting help with this kind of problem, and I'd like to have all these people collaborating on a solution that can be properly supported in the Mozilla tree.
We need to choose between two alternative approaches:
The decision is not clear-cut. For a long time I favoured the membuf approach because it's somewhat cleaner, portable, and could run on headless servers. But with more thought and after talking things through with Callum, I favour extending existing platform ports, for the following reasons:
The plan is to work towards getting the UBrowser code ported to Linux and it and its support patches landed in Mozilla's CVS tree. Then people can use it as a starting point for their own embedding projects and start collaborating on improvements. The code is already xulrunner-based which is great.
Before my visit I had been confused by reports that plugins actually work in UBrowser, since I knew of no way to redirect plugin rendering to an offscreen bitmap. It turns out that the patch detects the offscreen painting case on Windows and asks plugins to print themselves via the NPAPI Print command, and Flash honours this. Other plugins such as Quicktime don't. We can't detect when the Flash plugin wants to repaint its window so UBrowser just updates it at 15Hz. Input events don't work in any plugin although that can probably be faked. On Linux we can probably solve these issues using the X Composite and Damage extensions, but I'm not sure what we can do on Mac or how we can complete the job on Windows. Nevertheless it's a step forward. It means that once we land this patch, the canvas drawWindow API will paint Flash content on Windows, which will be a nice bonus for the tab-preview extensions.
While there I also learned a lot about Second Life. It's a very interesting virtual world focused on the creativity and commerce of its residents. I think there's enormous potential for invention and fun --- and also vice. I certainly hope that people find ways to use it for more good than evil. Either way, I'm sure that Second Life is going to be extraordinarily successful.
We need to choose between two alternative approaches:
- Extend existing Gecko ports, e.g. the Windows port, to support offscreen embedding
- Create a new Gecko port that does not depend on any window system --- the "membuf approach" taken by the first effort in this space, Crystalzilla
The decision is not clear-cut. For a long time I favoured the membuf approach because it's somewhat cleaner, portable, and could run on headless servers. But with more thought and after talking things through with Callum, I favour extending existing platform ports, for the following reasons:
- The changes required to Mozilla code turn out to be very minor. The membuf approach requires a significant amount of new code.
- A separate membuf port would be poorly maintained compared to the platform code that everyone else uses.
- Some important things such as plugins and native themes would be difficult or impossible with membuf.
- There are a few things that are harder to do when the native platform gets in the way, but nothing major. For example Mozillla windows must be hidden. Platforms seem to be able to do this OK.
- If support in headless servers is required, they'll just have to run Xvfb or Xvnc; not an ideal solution, but probably practical.
The plan is to work towards getting the UBrowser code ported to Linux and it and its support patches landed in Mozilla's CVS tree. Then people can use it as a starting point for their own embedding projects and start collaborating on improvements. The code is already xulrunner-based which is great.
Before my visit I had been confused by reports that plugins actually work in UBrowser, since I knew of no way to redirect plugin rendering to an offscreen bitmap. It turns out that the patch detects the offscreen painting case on Windows and asks plugins to print themselves via the NPAPI Print command, and Flash honours this. Other plugins such as Quicktime don't. We can't detect when the Flash plugin wants to repaint its window so UBrowser just updates it at 15Hz. Input events don't work in any plugin although that can probably be faked. On Linux we can probably solve these issues using the X Composite and Damage extensions, but I'm not sure what we can do on Mac or how we can complete the job on Windows. Nevertheless it's a step forward. It means that once we land this patch, the canvas drawWindow API will paint Flash content on Windows, which will be a nice bonus for the tab-preview extensions.
While there I also learned a lot about Second Life. It's a very interesting virtual world focused on the creativity and commerce of its residents. I think there's enormous potential for invention and fun --- and also vice. I certainly hope that people find ways to use it for more good than evil. Either way, I'm sure that Second Life is going to be extraordinarily successful.
Comments
http://video.google.com/videoplay?docid=-5182759758975402950
With regards to Quicktime, I would have thought it would be better to work on something like Media Player Classic instead, whereby we could (a) see the code and fix bugs and (b) install it automatically via the update service.
hmm. I need to look into this more.
Slightly off topic: if anyone reading this wants to help us solve the remaining Mozilla problems, please drop me a line (cory at secondlife dot com) because we are continuing to hire developers -- including remote developers -- like crazy.
Paul Colton
http://www.aptana.com
The headless xulrunner capability is certainly something lots of people have looked at for several years (five or more). The core capability people seem to be looking for is an easy way to use the wonderful rendering capability in the Mozilla code base for a whole variety of different and interesting projects.
There are dozens of requests in many newsgroups from people expecting it ought to be possible (and easy), and being very surprised that no straightforward capability has ever been built in.
It is certainly possible to move in an Xvfb direction to eliminate the need for a display, and we currently use that approach after trying many others. However, it adds a whole bundle of complexity outside the application especially when there may be several virtual displays going (and if the capability becomes widely used, they could be from several different applications). Detecting, starting and stopping the right Xvfb services is difficult to do reliably (at least in our limited experience). We have done the same thing under Windows, where we had to write a virtual framebuffer driver to give us some Xvfb-like capabilities.
It also means that any application wanting to use the xulrunner rendering capability needs to run X I think. This is a further disadvantage in terms of size and overhead, all just to fudge a display buffer.
There are other needs in a highly useable rendering system, including rendering items that are not on the web. Mozilla/Firefox still crashes under Linux if the equivalent of the OnBeforeNavigate event is captured for the purposes of redirection, and nasty dialogs still appear when the browser trips across a missing file link when working locally from disk (even before there is any event that can be captured reliably to prevent it).
The long and the short of it is that taking an Xvfb approach may make the Mozilla code simpler and more mainstream, but it pushes the burden of size, complexity, maintainability onto the developers. It destroys a lot of the out-of-the-box straightforward functionality that could be achieved, because there is a whole set of other knowledge and skills that becomes necessary to tackle that simple hypothetical rendering job.
Having suffered with this for several years now, and having been tantalised with various descriptions of embedding Gecko and being able to use it as an independent rendering agent, I *beg* you to reconsider the direction. And with our suffering so far, it truly is *begging*.
We would certainly love to help. Our knowledge of Mozilla internals is a lot less than our knowledge of real-world applications where browser and rendering capability is needed, perhaps part of our contribution would be on requirements and testing?
If we can help in any way (or bribe anyone to do what's hard but right) please contact me at jsinton at cavelle dot com. All the best.
I'm surprised that using Xvfb seems so hard; it shouldn't be. You say that starting and stopping the right Xvfb services is hard, but it seems to me that you could write a script that starts Xvfb, starts xulrunner, and then when xulrunner exits, kills the Xvfb it started. Why doesn't that work? If this is complicated to get right, perhaps we just need to collaborate a packaged solution that does it right.
I don't know what the cost of Xvfb is but it shouldn't be that high because it's not doing much work. You could, if you wanted to, run multiple xulrunners against one Xvfb.
The other issues you mentioned are not related to this problem. I think the way to go about solving them is to create a new embedding client that implements nsIPromptService and other UI control interfaces.
It appears that there are some significant limitations on how xulrunner can interact with the X display system (Xvfb specifically) when there is no interactive user (i.e. conventional gui I believe) logged in. For example, a daemon that needs to render can create an Xvfb display and put xulrunner on it, but xulrunner dies because there is something in the X environment it doesn't like when there is no interactive login. The identical daemon/application can be started from an interactive login (same user as the daemon) and runs fine.
While no X expert on how it may be possible to work around this limitation (and the point is one shouldn't have to be an X expert in order to use xulrunner to do rendering), this would be a good specific example of the continuing pain that would result from a solution to the render-to-buffer solution that relies on systems other than memory under the control of xulrunner. There will be continuing issues that require lots of extra development work instead of having a rendering component that 'just works' all the time and everywhere.
There has been so much good work go into Mozilla and there has been such a long wait for a good solution to the rendering need that it would be a great loss to put a band-aid on it instead of a solid, complete, portable and reliable solution.
Thanks for listening!
jsinton a t cavelle d o t com
> There will be continuing issues that require
> lots of extra development work
Worse than creating and maintaining an entirely new port? I still don't think so.
> a solid, complete, portable and reliable
> solution
As I mentioned in the original post, the membuf approach cannot be regarded as a complete solution for most potential users, because it can't support plugins.