In an earlier post I raised the question of whether a truly radical redesign of the Gecko frame system could be developed incrementally (which is almost the same as asking whether it can be developed at all). I think I figured out how that could be done.
What I would do is bolt on the new layout subsystem in parallel with the old one. So basically as well as running the frame constructor to build and update frame trees in response to DOM changes, and restyling and reflowing those trees, we could also have another DOM listener that builds and maintains some other kind of structure, and hooks to run layout in that structure. Initially all rendering and other uses of layout would continue using the frame tree. With this approach, we'd always have a working browser, and it would be possible to gradually extend the second layout engine to handle more and more element and styles, and to measure its space and time performance and compare them to the existing layout, and even to check its results for consistency with the existing layout. Then at some point we'd make it possible to render using the new layout, possibly even selectable at run time. Then we'd convert over other users of the frame tree and finally stop buildling it. This approach also has the advantage that the riskiest and most uncertain work --- how easy is it to fit the requirements of real-world Web layout into the new model, and how does it perform --- is tackled first.
I'm not saying we should do anything like this. I'm mainly leaving a note for myself in case this becomes important later.