WebExtensions (there is no XUL) took over with a thud seven months ago, which was felt as a great disturbance in the Force by most of us who wrote Firefox add-ons that, you know, actually did stuff. Many promises were made for APIs to allow us to do the stuff we did before. Some of these promises were kept and these APIs have actually been implemented, and credit where credit is due. But there are many that have not (that metabug is not exhaustive). More to the point, there are many for which people have offered to write code and are motivated to write code, but we have no parameters for what would be acceptable, possibly because any spec would end up stuck in a "boil the ocean" problem, possibly because it's low priority, or possibly because someone gave other someones the impression such an API would be acceptable and hasn't actually told them it isn't. The best way to get contribution is to allow people to scratch their own itches, but the urgency to overcome the (largely unintentional) institutional roadblocks has faded now that there is somewhat less outrage, and we are still left with a disordered collection of APIs that extends Firefox relatively little and a very slow road to do otherwise.
Or perhaps we don't have to actually rely on what's in Firefox to scratch our itch, at least in many cases. In a potentially strategically unwise decision, WebExtensions allows native code execution in the form of "native messaging" -- that is, you can write a native component, tell Firefox about it and who can talk to it, and then have that native component do what Firefox don't. At that point, the problem then becomes more one of packaging. If the functionality you require isn't primarily limited by the browser UI, then this might be a way around the La Brea triage tarpit.
Does this sound suspiciously familiar to anyone like some other historical browser-manipulated blobs of native code? Hang on, it's coming back to me. I remember something like this. I remember now. I remember them!
If you've been under a rock until Firefox 52, let me remind you that plugins were globs of native code that gave the browser wonderful additional capabilities such as playing different types of video, Flash and Shockwave games and DRM management, as well as other incredibly useful features such as potential exploitation, instability and sometimes outright crashes. Here in TenFourFox, for a variety of reasons I officially removed support for plugins in version 6 and completely removed the code with TenFourFox 19. Plugins served a historic purpose which are now better met by any number of appropriate browser and HTML5 APIs, and their disadvantages now in general outweigh their advantages.
Mozilla agrees with this, sort of. Starting with 52 many, though not all, plugins won't run. The remaining lucky few are Flash, Widevine and OpenH264. If you type about:plugins into Firefox, you'll still see them unless you're like me and you're running it on a POWER9 Talos II or some other hyper-free OS. There's a little bit of hypocrisy here in the name of utilitarianism, but I think it's pretty clear there would be a pretty high bar to adding a new plugin to this whitelist. Pithily, Mozilla has concluded regardless of any residual utility that plugins kill kittens.
It wasn't just plugins, either. Mozilla really doesn't like native code at all when it can be avoided; for a number of years in the Mozilla developer community I remember a lot of interest in making JavaScript so fast it could be used for all the media decoding and manipulation that native plugins then did. It's certainly gotten a lot faster since then, but now I guess the interest is having WASM do that instead (I'll wait). A certain subset of old-school extensions used to have binary components too, though largely to do funky system-level things that regular XPCOM in JavaScript and/or js-ctypes couldn't handle. This practice was strongly discouraged by Mozilla but many antiviruses implemented browser scanners in that fashion. This may have been a useful means to an end but binary components were a constant source of bugs when Mozilla altered something that the add-on authors didn't expect to change, and thus they kill kittens as well.
Nevertheless, we're back in a situation where we actually need certain APIs to be implemented to make certain types of functionality -- functionality which was perfectly acceptable in the XUL addons era, I might add -- actually still possible. These APIs are not a priority to Mozilla right now, but a certain subset of them can be provided by a (formerly discouraged) native component.
It's time to kill some kittens.
The particular kitten I need to kill is TCP sockets. I need to be able to talk over TCP to Gopher servers directly on a selection of port numbers. This was not easy to implement with XPCOM, but you could implement your own nsIChannel as a first-class citizen that looked to the browser like any other channel in pure JavaScript back in the day, and I did. That was OverbiteFF. I'm not the only one who asked for this feature, and I was willing to write it, but I'm not going to spend a lot of time writing code Mozilla won't accept and that means Mozilla needs to come up with an acceptable spec. This has gradually drowned under the ocean they're trying to boil by coming up with everyone's use cases whether they have anything in common or not and then somehow addressing the security concerns of all these disparate connection models and eventually forging the one ring to bind them all sometime around the natural heatdeath of the observable universe. I'm tired of waiting.
So here is OverbiteNX. OverbiteNX comes in two parts: the actual WebExtensions addon ("OverbiteNX" proper), and Onyx, a native component that the browser add-on asks to connect to a Gopher server and then receives data from it. Onyx is supported on Microsoft Windows, Linux and macOS (I've included binaries for Win32 and macOS 10.12+), and works probably anywhere else you can compile and run Firefox and Onyx as long as it supports Berkeley sockets. Onyx has no dependencies, is written in cross-platform C with a couple Windows-specific bits, and is currently presented in a single source file in an uncomplicated manner to not only serve the purpose of the extension but also to serve as an educational example of how to kill your own particular kitten. All of the pieces needed to hook it up (the Windows NSI and example JSON) are included and you can see how it connects with the front-end and the life cycle of a request in the source code. I discuss the architecture in more detail and how you can play with it. (If you are a current user of OverbiteWX, please remove it first.)
To be sure, I'm not the first to have this idea, and it's hardly an optimal solution. Besides the fact I have to coax someone to install a binary on their system to use the add-on, if I change the communication protocol (which is highly likely given that I will need to add future support for multithreading and possibly session IDs when multiple tabs are in use) then I need to get them to upgrade both the add-on and the native component at the same time. Users on weird platforms, and as a user of a weird platform I certainly empathize, have to do more work to get it to run on their system by compiling and installing it manually. The dat extension I linked to at the beginning of this paragraph gets around the binary code limitation by running the "native" component as JavaScript under Node.js, but that requires you to run Node, which is an extra step that seems unrealistic for many end-users and actually adds more work to people on weird platforms.
On the other hand, this is the only way right now it's going to work at all. I don't think Mozilla is going to move on these pain points unless they get into the situation where half the add-ons on AMO depend on external components they don't manage. So let the kitten killing begin. If you've got an idea and the current APIs don't let you implement it, see if a native component will scratch your itch and increase the pressure on Mountain View. After all, if Mozilla doesn't add useful WebExtensions APIs, there's no alternative but feline mass murder.
Provocative hyperbole aside, bug reports and pull requests appreciated for OverbiteNX. At some point in the near future the add-on piece will be submitted to AMO, which I expect to pass the automatic scanner since it doesn't do any currently proscribed operations, and then it's just a matter of keeping the native component in sync for future releases. Watch the Github project for more. And enjoy Gopherspace. Just don't tell the EU about it.
(Note for the humourless: The cat pictured is my cat. She is purring on her ottoman as this was written. No cat was harmed during the making of this blog post, though she does get an occasional talking-to. The picture was taken when she was penned up in the laundry while I was out and she was not threatened with any projectile weapon.)
I agree that this is a clever interim fix, but I'm not sold on it being a solid long-term idea.
ReplyDeleteAccording to Bugzilla, they already want to implement official TCP/UDP Socket APIs, and recent clues suggest that they want to do it soon: https://bugzilla.mozilla.org/show_bug.cgi?id=1467145#c3
Given that, I can't see why they wouldn't be willing to accept patches to help get it done sooner. That seems like a much less risky long-term prospect than having to maintain your own security-sensitive code on your own... though in fairness, maybe you'd still have to do that anyway; I don't know how much code specific to older OSX versions would be needed.
What I do know is that this isn't really a question of willingness on Mozilla's part. There are lots of approved APIs waiting for design and implementation work, but not enough people to do actually do that work quickly enough for everyone's tastes. Mere willingness isn't enough. Relatively few people who sound willing are actually taking that extra step and helping to get things done.
Maybe that's just because those folks don't have the time, or because they aren't really able to sling the lower-level code required, but either way the number of people working on the code isn't increasing. Simply adding more pressure isn't going to cut it.
I'm trying to be polite here, but did you actually RTFA?
DeleteI actually *am* willing to write the implementation, but Mozilla is going to have to say what they're willing to accept. They haven't, and they haven't made really any progress on writing the spec, which *they* have to do. I'm not working my ass off to write code that has no chance of ever being used. I've done that before and I'm not doing it again.
It's not supposed to be a long-term idea, either. I think I made that clear. But it's going to be what gets the project off the ground. And pressure is definitely needed to get Mozilla(tm) collectively thinking about what they're willing to tolerate so that, yes, people can actually write it. That hasn't happened.
First things first: know that I truly respect you and your work (otherwise I wouldn't bother writing this in the first place). I just hate the thought that you'd waste your valuable time on something self-defeating. After all, this work proves that Mozilla don't have to prioritize a socket API over other things, because OverbiteNX now exists, all using WebExtensions as they're meant to be used.
DeleteNow maybe this was just intended as a precursor to a true API proposal, at which point I apologize (that didn't come across to me in your post). But if it's not, then you're not showing a willingness to do the currently-necessary work, which is what matters now. Fair enough, but then just applying more pressure on the few people who *are* willing to do it, but are already overburdened, won't help. They're humans, not steam engines.
They most certainly will have to prioritize expanding the API surface when half the add-ons on AMO eventually depend on components that aren't distributed there.
DeleteMoreover, I don't determine what's acceptable to Mozilla, and they know what their pain points and limits are. I'm not going to boil the ocean trying to collect everyone else's use case if the idea will never get off the ground administratively. Mozilla needs to say what they're going to allow and not allow, and only they can say that.
You still aren't making any real sense to me.
ReplyDeleteWhy you can't just propose OverbiteNX's "API" as a starter proposal to Mozilla? I mean if it's good enough that you feel others can safely rely on it instead of waiting for Mozilla's rubber-stamp, then why not try that much?
You don't have to do anything beyond that if you'd rather not, but it seems like a waste not to try that much at this stage?
It honestly just makes it seem like you are unwilling to do what's *truly* necessary right now to get the APIs *you* want into Firefox proper, despite having possibly done almost all of the work already, and despite having real no commitment to follow through.
I can't say that's a sympathetic viewpoint when I know that Mozilla are also willing to do the eventual implementation *plus* everything else to get the feature specced-out, once higher-priority APIs are done.
Why we should be pressuring Mozilla more and not you? Neither of you are obligated to do the work, but only one seems willing to even try to do it right.