Eyes Above The Waves

Robert O'Callahan. Christian. Repatriate Kiwi. Hacker.

Saturday 22 August 2015

Hooray For WebExtensions

Many of the comments on WebExtensions miss an important point: basing WebExtensions on the Chrome extension API does not imply limiting WebExtensions to the Chrome API. For example, Bill already made it clear that we want to support addons like Tree Style Tabs and NoScript, which Chrome does not support. So Firefox addons will continue to be able to do things you can't do in Chrome (though there will be some things you can hack into Firefox's XUL today that won't be supported by WebExtensions, for sure).

WebExtensions is something we need to do and should have done many years ago, before Chrome even existed. It's what Jetpack should have been. But better late than never!

Comments

Matt
The reason people are confused is because of statements like the following which are incompatible with what you and Bill are saying: "With this API, extension developers should be able to make the same extension available on Firefox and Chrome with a minimal number of changes to repackage for each platform." That's not true if the Web Extension API != Chrome Extension API. "We have started discussions with other browser vendors to begin an effort to standardize at least some of this API…" They key part there is "some" but I can see people thinking this will be like web standards where they move much slower than innovators/developers would like and so the APIs will be limited to what browsers can agree on and then implemented sometimes years later. This makes browser vendors the gatekeepers whereas current extensions can innovate outside the bounds of a constrained API. I'm glad to hear Bill say will there likely still be some way to make XPCOM extensions but that wasn't clear in the original post and the opposite was actually communicated with a "deprecation" notice: "We don’t have a specific timeline for deprecation, but most likely it will take place within 12 to 18 months from now." So we're sending a lot of mixed/conflicting messages and the "official" post strongly implies that WebExtensions will be the only way supported at some point. I really hope that's not the case or it would mean we're killing one of our remaining strengths.
Unknown
"That's not true if the Web Extension API != Chrome Extension API." That's not a useful comparison, the thing is: It is true for Chrome Extension API ⊂Web Extension API Which is what the goal is, to have the Chrome Extension API as a miinimum baseline and then extend upon it.
Matt
The post doesn't make clear that the Web Extension API is supposed to be a superset and the post is specifically talking about making an extension work in both browsers which would require that the same APIs are present. My point is that the add-ons blog post caused confusion and negative press because it's inaccurate.
Jesse Ruderman
pkasting is willing to make changes to Chromium to help support add-ons like Tree Style Tab: https://code.google.com/p/chromium/issues/detail?id=344870#c113
JulienW
> It's what Jetpack should have been. I admit I don't completely see the difference.
Robert
Jetpack doesn't sandbox addons, does not ensure they work well in a multiprocess browser, and instead of being a stable API implemented in the browser, it was designed to be a library that ships with addons, which means you have to have repacks.
Wladimir Palant
I'm not sure what you mean by "designed" but JPM-packaged add-ons no long have any libraries in them - there is pretty much nothing but the add-on code there. So as of Firefox 38 Jetpack is an API implemented in the browser (which has its downsides, like a SeaMonkey-related bug which went unnoticed for a while and cannot be fixed for already released SeaMonkey versions). Also, the browser decides what the add-ons have access to - tomorrow the access to require("chrome") and low-level modules could be removed, and then the add-ons won't have a way of breaking out of the sandbox (or messing up multiprocess support).
Robert
If we made those changes to Jetpack (and other necessary changes like removing XUL access), it wouldn't really be Jetpack anymore. Most Jetpack addons would be broken and need significant modifications. WebExtensions requires big changes too, but at least it gives us the extra value of making it easy to have extensions work in multiple browsers.
Wladimir Palant
I have to disagree. Jetpack implemented significant changes before, quite successfully. What we have now already isn't Jetpack as it was introduced a few years ago. Lots of APIs have been replaced by better designs and new APIs have been created. Mozilla finally committed significant resources towards developing well-designed add-on APIs, if these were added to Jetpack there would be far less incentive to break out of the sandbox - most add-ons could gradually migrate away from low-level APIs. But not all of them, because you simply cannot have an API for everything. That's why WebExtensions will have native.js which is the moral equivalent of require("chrome") - sometimes the only way is breaking out of the sandbox, even if there is a risk that some future browser version will break your extension (well, either that or losing many innovative extensions which is where Chrome is right now). I also don't buy the "value of making it easy to have extensions work in multiple browsers" talk. This kind of thing doesn't come for free, it forces you into bug-for-bug compatibility with a competitor and limits your innovation. Many Chrome's APIs have non-obvious bugs and limitations or are simply badly designed. Will Mozilla fix them even though that would mean losing compatibility with existing Chrome extensions?
Robert
Compatibility isn't all-or-nothing. There is some value in making it easy to port an extension between browsers, with some testing and tweaks, without going all the way to full drop-in compatibility.
Peter Kasting
As Jesse alludes to, I think your statement about Chrome intentionally not supporting things like NoScript and Tree Style Tabs is false. "Doesn't currently support" isn't the same as "intentional WontFix". Chrome is gaining a sidebar extensions API: https://code.google.com/p/chromium/issues/detail?id=477424 -- this is the primary blocker for something like Tree Style Tabs. NoScript is definitely an extension we've reached out to the author of in the past to try and support what it needs, and AFAIK we're still interested in doing so. So I suggest rephrasing your post to not say or imply things about Chrome's extension plans :)
Robert
I was parroting what I thought I heard at BlinkOn. I removed "intentionally" from the post.
J. McNair
As with almost every Mozilla announcement bearing bad news with the good, there appears to be an attitude of "do it now, announce it next, answer questions later". I shall freely admit my lack of knowledge here, but why build on top of a hard fork of the Chrome extension APIs? Is it impossible to add needed features to JetPack and deprecate require("chrome") ? Was it just EASIER and FASTER to use Chrome's existing and documented set of APIs? Does Mozilla Corp. leadership and engineering believe that attracting Chrome developers is the single best way to build a vibrant add-on ecosystem in the future? I am being unfair to you, as one engineer can't possibly speak for a corporation, but I would like to know where to find answers to these questions. Thank you.
Anonymous
Just because you're able to support doesn't translate to extension developers wanting to revamp their extensions to suit this new model, or others to do it in their stead. That's a cold reality and in the meantime, people will switch to Chrome.
voracity
Regardless of the technical details, extensions (particularly as equals to core code) had always seemed like the key defining trait of the Firefox browser. (Mozilla's defining trait is openness.) It gave us game-changers like Firebug, AdBlock, TabMix and Greasemonkey, along with thousands of other useful smaller extensions. Whatever other problems people may have had with Firefox, its extensibility was simply untouchable. Now the extension system is being dropped in favour of an experimental cross-browser system, mostly already implemented by Firefox's major competitor. Maybe technically this is a good idea, but publicly this will play out (has played out) *really* badly and could very well kill Firefox altogether. The only *unique* reason people have to choose Firefox now is based on the openness of the company that happens to make it --- and most people (sadly!) don't see any value in that at all. I'm hoping Servo is coming along well, because it will have an awful lot of responsibility on its shoulders.
Anonymous
Robert, I would like to know why WebExtensions are "what Jetpack should have been." From my point of view, Chrome's APIs have significant issues and are generally inferior to Jetpack (I've used both). Could you maybe explain the advantages you see, either here or on https://palant.de/2015/08/23/missing-a-rationale-for-webextensions?
Unknown
Rob I love you man, and I know you mean well, but I don't think you did much research on Jetpack for this post. Jetpack was created over 5 years ago, it's purpose for existing was two fold. First to make extension development easier and more powerful, and second to provide a bridge for extensions developers to use in order to support the e10s and servo changes that all of Mozilla has known were coming for more than 5 years now. > Jetpack doesn't sandbox addons This is wrong, all jetpack modules are loaded in sandboxes, including the jetpack loader itself. > does not ensure they work well in a multiprocess browser, and instead of being a stable API implemented in the browser, it was designed to be a library that ships with addons, which means you have to have repacks. This is all wrong, if you watch Aza's videos and read his blog posts from 2009-2010, and look at the Jetpack wiki history, we had been planning to support e10s and servo, and ship the stable apis in the browser from the beginning over 5yrs ago. We did the latter about two years ago as Wladimir mentions, and having the sdk support e10s is less than 2 weeks worth of work for one person by my estimate back in June, it's quite an easy thing to do. The last great repacks was in 2013, and there will never be a need for another one. In other words this problem was fixed two years ago, so this is an invalid point. > If we made those changes to Jetpack (and other necessary changes like removing XUL access), it wouldn't really be Jetpack anymore. Most Jetpack addons would be broken and need significant modifications. What changes are you referring to here? supporting e10s? I mention above that this is not hard to do, and was part of the plan 5yrs ago. Obviously we could not support vaporware at the time, so that is why it doesn't currently support e10s, but we always planned for doing this, and that is why it is very little work to. sandboxing? I mention above that this is done already. removing xul? I mention above the jetpack team had been planning for that for over 5yrs and this is one of the reasons why it was created, because 5yrs ago it was obvious that if XUL was going to die when Servo was used then Mozilla would have to offer an extension api that will work in both gecko and servo, which jetpack is designed to do. > Most Jetpack addons would be broken and need significant modifications. This just is not true, modules for jetpack can be shared on npm, so if a module needs to be updated to support e10s, servo, or whatever then the module on npm merely needs to be updated then every add-on in the world that uses that module will be updated automatically when the add-on developer uses `npm install`. So updating any broken add-on should be as easy as updating a dependency version in the package.json. The module developer is the one that has to do work, but they only do it once and everyone benefits. Anyhow, I don't think this is true based on my recollection of the data on amo, please post data if you have it. > WebExtensions is something we need to do and should have done many years ago, before Chrome even existed. This is cute, we should have copied the chrome.* api before it existed? > It's what Jetpack should have been. But better late than never! Um no. Jetpack is a much greater product than webextensions imo. Jetpacks allow innovation, freedom, control, module reusability, tdd, and as I proved with chrome-tailor back in april (https://github.com/jetpack-labs/chrome-tailor/) Jetpacks can be used to convert google chrome apis in to firefox extensions. Web extensions allow none of these features afaict. Of course once `require('chrome')` is removed from Jetpack the freedom and innovation parts will no longer be accurate, and that is very bad decision imo. Jetpacks are now what Jetpacks should have always been, and it's painful that Mozilla does not see that.
Robert
| I don't think you did much research on Jetpack for this post. True. I combined what I athought I knew with info from other people I trust. | This is wrong, all jetpack modules are loaded in sandboxes, including the jetpack loader itself. That is not the same thing as being properly sandboxed. Every Jetpack addon that uses "require(chrome)" is not properly sandboxed. Furthermore, a key part of the sandboxing of a WebExtensions addon is an explicit manifest specifying a set of permissions, with the guarantee that whatever the addon's JS does, it can't wield permissions not granted in its manifest. As far as I know Jetpack doesn't have this; correct me if I'm wrong! | This is all wrong, if you watch Aza's videos and read his blog posts from 2009-2010, and look at the Jetpack wiki history, we had been planning to support e10s and servo, and ship the stable apis in the browser from the beginning over 5yrs ago. Yes, I recall that being the plan at some point in the early days, but then we went and took the repacks path for three years (per your chronology), so we went far off the rails. (I'm not blaming you for that, or anything else.) I apologize again for not remembering that we've fixed the repacks problem since. | having the sdk support e10s is less than 2 weeks worth of work for one person by my estimate back in June, it's quite an easy thing to do. OK, but does "sdk support" include fixing all Jetpack API modules to support e10s? Can that even be done in a non-addon-breaking way? If the answers are "yes" and "yes", we should probably still do it! I've been told that's not the case. | What changes are you referring to here? Whatever changes are needed to make all Jetpack addons e10s-friendly, plus requiring Jetpack addons to ship with a permissions manifest which is enforced, plus ensuring that XUL is disabled in any context Jetpack addons can access (unless perhaps it gets its own permission). | if a module needs to be updated to support e10s, servo, or whatever then the module on npm merely needs to be updated then every add-on in the world that uses that module will be updated automatically when the add-on developer uses `npm install`. That only works if the API is e10s-friendly. The question then is how many APIs aren't e10s-friendly (by which I mean both support running the addon in a separate process to the browser chrome, and running the addon in a separate process to the content). I assumed from what I've been told that the answer is "many", but maybe I misunderstood. | This is cute, we should have copied the chrome.* api before it existed? No, we should have assumed a similar set of requirements and designed a similar sort of API and security model and shipped them in the browser from the beginning. | Of course once `require('chrome')` is removed from Jetpack the freedom and innovation parts will no longer be accurate, and that is very bad decision imo. It's tough. I think we should allow some extensions to do absolutely anything, but we need to very carefully separate those extensions from other extensions that are constrained so we can properly sandbox with a sane permissions system. The former extensions need to be carefully reviewed and will need to be regularly updated to account for browser changes. The latter can be supported via stable APIs indefinitely and need less careful review. We need to make sure there's an incentive for developers to create the latter kind of extension if their needs fit into those constraints (or work with us to expand the APIs to their needs do fit). Arguably as long as we support the very-old-fashioned classic addons, that satisfies the needs of the do-anything extensions, and we don't need to also support that class of extensions with Jetpack and WebExtensions. Anyway, sorry where I've got things wrong, and FWIW I haven't been involved in any decision-making on these issues. And thanks for speaking up.
Wladimir Palant
| Furthermore, a key part of the sandboxing of a WebExtensions addon is an explicit manifest specifying a set of permissions, with the guarantee that whatever the addon's JS does, it can't wield permissions not granted in its manifest. Jetpack originally had this as well - only modules explicitly specified in the package manifest could be required. At some point the packaging changed and this restriction was removed. I don't know the reasoning but I guess that Mozilla's priorities shifted from enforcing sandboxing towards more flexibility. Now that the priorities are shifting back adding this functionality back would be pretty trivial. | That only works if the API is e10s-friendly. The question then is how many APIs aren't e10s-friendly (by which I mean both support running the addon in a separate process to the browser chrome, and running the addon in a separate process to the content). I assumed from what I've been told that the answer is "many", but maybe I misunderstood. I think you are. Jetpack APIs have been designed with e10s and out of process execution in mind, hence the extreme reliance on content scripts. That doesn't mean that there are no issues - since they never actually executed out of process before I would expect some issues that the API designers didn't foresee. But the issues definitely aren't massive. | No, we should have assumed a similar set of requirements and designed a similar sort of API and security model and shipped them in the browser from the beginning. IMHO that's exactly what Jetpack did. | The latter can be supported via stable APIs indefinitely and need less careful review. Sadly, that's wishful thinking in my experience. Providing stable APIs has been attempted a number of times before (remember FUEL?). The problem with those: if the platform changes in a radical way as is happening right now then these APIs won't survive either - unless the API designers knew that these changes were coming and prepared in advance. Jetpack and WebExtensions are prepared for e10s and Servo but who knows what will come up after that. Also, less careful reviews are already being performed for Jetpack-based extensions (the system warns if low-level modules are used and these parts can be reviewed more thoroughly then). The problem is, understanding extension logic still isn't optional for those reviews. The biggest problem with extensions isn't the ones that take over your computer but rather the ones that will spy on you - e.g. send your entire browsing history to some third party or even inject a remote script into every web page you are visiting. Most of the time the extension developers aren't actually malicious but simply use third-party services mindlessly (and don't get me started on jQuery-related security issues). I don't think that any kind of privilege system will make these any less problematic than they are right now. Chrome of course chose a different solution: they simply ignore the problem, spying on the user isn't prohibited behavior in Chrome Web Store.