OverbiteNX is now available from Mozilla Add-Ons for beta testing

OverbiteNX, a successor to OverbiteFF which allows Firefox to continue to access legacy resources in Gopher in the brave courageous new world of WebExtensions, is now in public beta. Unlike the alpha test, which required you to download the repo and install the extension using add-on debugging, OverbiteNX is now hosted on Mozilla Add-Ons.

Because WebExtensions still doesn't have a TCP sockets API, nor a spec, OverbiteNX uses its bespoke Onyx native component to do network operations. Onyx is written in open-source portable C with no dependencies and is available in pre-built binaries for macOS 10.12+ and Windows (or get the repo and build it yourself on almost any POSIX system).

To try OverbiteNX, install Onyx from the links above, and then install the extension from Mozilla Add-ons. If you use(d) OverbiteWX, which is the proxy-based strict WebExtensions add-on, please disable it as it may conflict. Copious debugging output is emitted to the browser console for this test version. If you file an issue (or better still a pull request) on Github, please include the output so that we can see the execution trace.

Pro tip: sleep's good for your Power Mac

Now that I'm alternating between two daily drivers (my Quad G5 and my Talos II), the Quad G5 sleeps fairly reliably with the Sonnet USB-FireWire combo card out once I move the KVM focus off it. If I don't do that, the KVM detects the G5 has slept and moves the focus automatically away from it, which the G5 detects as USB activity, and exits sleep. The solution is a little AppleScript that waits a few seconds for me to switch the KVM and then tells the Finder to snooze. The Talos II doesn't sleep yet but I'll be interested to see if later firmware updates support that.

But sleeping the G5 has unquestionably been a good thing. Not only does it prolong its life by reducing heat (another plus in summer) as well as saving a substantial amount of energy (around 20W sleeping versus 200-250W running), but sleeping also can speed up TenFourFox. If you have lots of tabs open and those tabs are refreshing their data or otherwise running active content, then this contributes to a greater need for garbage collection and this will slow down your user experience as this overhead accumulates. (This is why running TenFourFox from a "fresh" start is much faster than when it's been chugging away for awhile.) It's possible to "pause" TenFourFox to a certain extent but the browser really isn't tested this way and may not behave properly when this is done. Sleeping the Power Mac pauses everything, so the cruft in memory that garbage collection has to clean out doesn't pile up while you're not using the machine, and everything comes back up in sync.

A whole lot of stuff has landed for TenFourFox FPR9. More about that when the beta is out, which I'm hoping to do by the middle-end of July.

Another one bites the Rust

And another one gone, and another one gone (capitalization sic):

As Herwig Bauernfeind from Bitwise Works made clear in his presentation he gave at Warpstock 2018 Toronto, Firefox for OS/2 is on its way out for OS/2 after version 52 ESR. The primary reason is because Firefox is switching to RUST. Rust is a general purpose programming language sponsored by Mozilla Research. It is unlikely that RUST will ever be ported to OS/2.

Rust was the primary reason we dropped source parity for TenFourFox also (though there were plenty of other reasons such as changes to the graphics stack, the hard requirement for Skia, Electrolysis and changes to ICU; all of this could have been worked around, but with substantial difficulty, and likely with severe compromises). Now that Firefox 52ESR, the last ESR to not require Rust support, is on its last legs, this marks the final end of "Warpzilla" and Firefox on OS/2. SPARC (and apparently Solaris in general) doesn't have rustc or cargo support either, so this is likely the end of Firefox on any version of Solaris as well. Yes, I use Firefox on my Sun Ultra-3 laptop with Solaris 10. There are probably other minor platforms just hanging on that will wink out and disappear which I haven't yet discovered.

Every platform that dies is a loss to the technical diversity of the Mozilla community, no matter how you choose to put a happy face on it.

If you were trying to get a web browser up on a new platform these days, much as it makes me sick to say it, you'd probably be better off with WebKit rather than wrestle with Rust. Or NetSurf, despite its significant limitations, though I love the fact the project even exists. At least there's Rust for the various forms of PowerPC on Linux, including 64-bit and little-endian, so the Talos II can still run Firefox.

With FPR9 TenFourFox will switch to backporting security updates from Firefox 60ESR, though any last-minute chemspills to 52ESR will of course be reviewed promptly.

UPDATE 7/5: Someone in the discussion on Hacker News found that at least $12,650 was raised by the OS/2 community, and they're going to port a Qt based browser, which means ... WebKit. I told you so.

Ad-blocker-blockers hit a new low. What's the solution?

It may be the wrong day to slam the local newspapers, but this was what greeted me trying to click through to a linked newspaper article this morning on Firefox Android. The link I was sent was from the Riverside Press-Enterprise, but this appears to be throughout the entire network of the P-E's owners, the Southern California News Group (which includes the Orange County Register, San Bernardino Sun and Los Angeles Daily News):

That's obnoxious. SCNG is particularly notorious for not being very selective about ads and they tend to be colossally heavy and sometimes invasive; there's no way on this periodically green earth that I'm turning the adblocker off. I click "no thanks." The popover disappears, but what it was covering was this:

That's not me greeking the article so you can't see what article I was reading. The ad-blocker-blocker did it so that a clever user or add-on can't just set the ad-blocker-blocker's popover to display:none or something. The article is now incomprehensible text.

My first reaction is that any possibility I had of actually paying $1 for the 4 week subscription to any SCNG paper just went up in the flames of my great furious wrath (after all, this is a blog s**tpost). The funny part is that TenFourFox's basic adblock actually isn't defeated by this, probably because we're selective about what actually gets blocked and so the ad-blocker-blocker thinks ads are getting through. But our old systems are precisely those that need adblockers because of all the JavaScript (particularly) that modern ad systems lard their impressions up with. Anyway, to read the article I actually ended up looking at it on the G5. There was no way I was going to pay them for engaging in this kind of behaviour.

The second thought I had was, how do you handle this? I'm certainly sympathetic to the view that we need stronger local papers for better local governance, but print ads are a much different beast than the dreck that online ads are. (Yes, this blog has ads. I don't care if you block them or not.) Sure, I could have subscriptions to all the regional papers, or at least the ones that haven't p*ssed me off yet, but then I have to juggle all the memberships and multiple charges and that won't help me read papers not normally in my catchment area. I just want to click and read the news, just like I can anonymously pick up a paper and read it at the bar.

One way to solve this might be to have revenue sharing arrangements between ISPs and papers. It could be a mom-and-pop ISP and the local paper, if any of those or those still exist, or it could be a large ISP and a major national media group. Users on that ISP get free access (as a benefit of membership even), the paper gets a piece. Everyone else can subscribe if they want. This kind of thing already exists on Apple TV devices, after all: if I buy the Spectrum cable plan, I get those channels free on Apple TV over my Spectrum Internet access, or I pay if I don't. Why couldn't newspapers work this way?

Does net neutrality prohibit this?

TenFourFox FPR8 available

TenFourFox Feature Parity Release 8 final is now available (downloads, hashes, release notes). There are no changes from the beta except for outstanding security patches. As usual, it will go live Monday night, assuming no changes.

TenFourFox FPR8b1 available

TenFourFox Feature Parity Release 8 beta 1 is now available (downloads, release notes, hashes). There is much less in this release than I wanted because of a family member in the hospital and several technical roadblocks. Of note, I've officially abandoned CSS grid again after an extensive testing period due to the fact that we would need substantial work to get a functional implementation, and a partially functional implementation is worse than none at all (in the latter case, we simply gracefully degrade into block-level <div>s). I also was not able to finish the HTML <input> date picker implementation, though I've managed to still get a fair amount completed of it, and I'll keep working on that for FPR9. The good news is, once the date picker is done, the time picker will use nearly exactly the same internal plumbing and can just be patterned off it in the same way. Unlike Firefox's implementation, as I've previously mentioned our version uses native OS X controls instead of XUL, which also makes it faster. That said, it is a ghastly hack on the Cocoa widget side and required some tricky programming on 10.4 which will be the subject of a later blog post.

That's not to say this is strictly a security patch release (though most of the patches for the final Firefox 52, 52.9, are in this beta). The big feature I did want to get in FPR8 did land and seems to work properly, which is same-site cookie support. Same-site cookie support helps to reduce cross-site request forgeries by advising the browser the cookie in question should only be sent if a request originates from the site that set it. If the host that triggered the request is different than the one appearing in the address bar, the request won't include any of the cookies that are tagged as same-site. For example, say you're logged into your bank, and somehow you end up opening another tab with a malicious site that knows how to manipulate your bank's transfer money function by automatically submitting a hidden POST form. Since you're logged into your bank, unless your bank has CSRF mitigations (and it had better!), the malicious site could impersonate you since the browser will faithfully send your login cookie along with the form. The credential never leaked, so the browser technically didn't malfunction, but the malicious site was still able to impersonate you and steal your money. With same-site cookies, there is a list of declared "safe" operations; POST forms and certain other functions are not on that list and are considered "unsafe." Since the unsafe action didn't originate from the site that set the cookie, the cookie isn't transmitted to your bank, authentication fails and the attack is foiled. If the mode is set to "strict" (as opposed to "lax"), even a "safe" action like clicking a link from an outside site won't send the cookie.

Same-site cookie support was implemented for Firefox 60; our implementation is based on it and should support all the same features. When you start FPR8b1, your cookies database will be transparently upgraded to the new database schema. If you are currently logged into a site that supports same-site cookies, or you are using a foxbox that preserves cookie data, you will need to log out and log back in to ensure your login cookie is upgraded (I just deleted all my cookies and started fresh, which is good to give the web trackers a heart attack anyway). Github and Bugzilla already have support, and I expect to see banks and other high-security sites follow suit. To see if a cookie on a site is same-site, make sure the Storage Inspector is enabled in Developer tools, then go to the Storage tab in the Developer tools on the site of interest and look at the Cookies database. The same-site mode (unset, lax or strict) should be shown as the final column.

FPR8 goes live on June 25th.

Let's kill kittens with native messaging (or, introducing OverbiteNX: if WebExtensions can't do it, we will)

WebExtensions (there is no XUL) took over with a thud seven months ago, which was felt as a great disturbance in the Force by most of us who wrote Firefox add-ons that, you know, actually did stuff. Many promises were made for APIs to allow us to do the stuff we did before. Some of these promises were kept and these APIs have actually been implemented, and credit where credit is due. But there are many that have not (that metabug is not exhaustive). More to the point, there are many for which people have offered to write code and are motivated to write code, but we have no parameters for what would be acceptable, possibly because any spec would end up stuck in a "boil the ocean" problem, possibly because it's low priority, or possibly because someone gave other someones the impression such an API would be acceptable and hasn't actually told them it isn't. The best way to get contribution is to allow people to scratch their own itches, but the urgency to overcome the (largely unintentional) institutional roadblocks has faded now that there is somewhat less outrage, and we are still left with a disordered collection of APIs that extends Firefox relatively little and a very slow road to do otherwise.

Or perhaps we don't have to actually rely on what's in Firefox to scratch our itch, at least in many cases. In a potentially strategically unwise decision, WebExtensions allows native code execution in the form of "native messaging" -- that is, you can write a native component, tell Firefox about it and who can talk to it, and then have that native component do what Firefox don't. At that point, the problem then becomes more one of packaging. If the functionality you require isn't primarily limited by the browser UI, then this might be a way around the La Brea triage tarpit.

Does this sound suspiciously familiar to anyone like some other historical browser-manipulated blobs of native code? Hang on, it's coming back to me. I remember something like this. I remember now. I remember them!

Plugins!

If you've been under a rock until Firefox 52, let me remind you that plugins were globs of native code that gave the browser wonderful additional capabilities such as playing different types of video, Flash and Shockwave games and DRM management, as well as other incredibly useful features such as potential exploitation, instability and sometimes outright crashes. Here in TenFourFox, for a variety of reasons I officially removed support for plugins in version 6 and completely removed the code with TenFourFox 19. Plugins served a historic purpose which are now better met by any number of appropriate browser and HTML5 APIs, and their disadvantages now in general outweigh their advantages.

Mozilla agrees with this, sort of. Starting with 52 many, though not all, plugins won't run. The remaining lucky few are Flash, Widevine and OpenH264. If you type about:plugins into Firefox, you'll still see them unless you're like me and you're running it on a POWER9 Talos II or some other hyper-free OS. There's a little bit of hypocrisy here in the name of utilitarianism, but I think it's pretty clear there would be a pretty high bar to adding a new plugin to this whitelist. Pithily, Mozilla has concluded regardless of any residual utility that plugins kill kittens.

It wasn't just plugins, either. Mozilla really doesn't like native code at all when it can be avoided; for a number of years in the Mozilla developer community I remember a lot of interest in making JavaScript so fast it could be used for all the media decoding and manipulation that native plugins then did. It's certainly gotten a lot faster since then, but now I guess the interest is having WASM do that instead (I'll wait). A certain subset of old-school extensions used to have binary components too, though largely to do funky system-level things that regular XPCOM in JavaScript and/or js-ctypes couldn't handle. This practice was strongly discouraged by Mozilla but many antiviruses implemented browser scanners in that fashion. This may have been a useful means to an end but binary components were a constant source of bugs when Mozilla altered something that the add-on authors didn't expect to change, and thus they kill kittens as well.

Nevertheless, we're back in a situation where we actually need certain APIs to be implemented to make certain types of functionality -- functionality which was perfectly acceptable in the XUL addons era, I might add -- actually still possible. These APIs are not a priority to Mozilla right now, but a certain subset of them can be provided by a (formerly discouraged) native component.

It's time to kill some kittens.

The particular kitten I need to kill is TCP sockets. I need to be able to talk over TCP to Gopher servers directly on a selection of port numbers. This was not easy to implement with XPCOM, but you could implement your own nsIChannel as a first-class citizen that looked to the browser like any other channel in pure JavaScript back in the day, and I did. That was OverbiteFF. I'm not the only one who asked for this feature, and I was willing to write it, but I'm not going to spend a lot of time writing code Mozilla won't accept and that means Mozilla needs to come up with an acceptable spec. This has gradually drowned under the ocean they're trying to boil by coming up with everyone's use cases whether they have anything in common or not and then somehow addressing the security concerns of all these disparate connection models and eventually forging the one ring to bind them all sometime around the natural heatdeath of the observable universe. I'm tired of waiting.

So here is OverbiteNX. OverbiteNX comes in two parts: the actual WebExtensions addon ("OverbiteNX" proper), and Onyx, a native component that the browser add-on asks to connect to a Gopher server and then receives data from it. Onyx is supported on Microsoft Windows, Linux and macOS (I've included binaries for Win32 and macOS 10.12+), and works probably anywhere else you can compile and run Firefox and Onyx as long as it supports Berkeley sockets. Onyx has no dependencies, is written in cross-platform C with a couple Windows-specific bits, and is currently presented in a single source file in an uncomplicated manner to not only serve the purpose of the extension but also to serve as an educational example of how to kill your own particular kitten. All of the pieces needed to hook it up (the Windows NSI and example JSON) are included and you can see how it connects with the front-end and the life cycle of a request in the source code. I discuss the architecture in more detail and how you can play with it. (If you are a current user of OverbiteWX, please remove it first.)

To be sure, I'm not the first to have this idea, and it's hardly an optimal solution. Besides the fact I have to coax someone to install a binary on their system to use the add-on, if I change the communication protocol (which is highly likely given that I will need to add future support for multithreading and possibly session IDs when multiple tabs are in use) then I need to get them to upgrade both the add-on and the native component at the same time. Users on weird platforms, and as a user of a weird platform I certainly empathize, have to do more work to get it to run on their system by compiling and installing it manually. The dat extension I linked to at the beginning of this paragraph gets around the binary code limitation by running the "native" component as JavaScript under Node.js, but that requires you to run Node, which is an extra step that seems unrealistic for many end-users and actually adds more work to people on weird platforms.

On the other hand, this is the only way right now it's going to work at all. I don't think Mozilla is going to move on these pain points unless they get into the situation where half the add-ons on AMO depend on external components they don't manage. So let the kitten killing begin. If you've got an idea and the current APIs don't let you implement it, see if a native component will scratch your itch and increase the pressure on Mountain View. After all, if Mozilla doesn't add useful WebExtensions APIs, there's no alternative but feline mass murder.

Provocative hyperbole aside, bug reports and pull requests appreciated for OverbiteNX. At some point in the near future the add-on piece will be submitted to AMO, which I expect to pass the automatic scanner since it doesn't do any currently proscribed operations, and then it's just a matter of keeping the native component in sync for future releases. Watch the Github project for more. And enjoy Gopherspace. Just don't tell the EU about it.

(Note for the humourless: The cat pictured is my cat. She is purring on her ottoman as this was written. No cat was harmed during the making of this blog post, though she does get an occasional talking-to. The picture was taken when she was penned up in the laundry while I was out and she was not threatened with any projectile weapon.)