24.2.0 is live

And so is Maverickspoof.

24.2.0, at last, released, and Mozilla tries to get rid of extra hair

Thanks to everyone who commented on our last post about targets for the next "major release" of TenFourFox. The overwhelming consensus is that, although there is some trepidation (and well-deserved, apparently) over Australis, it's the future of Firefox like it or not, and if it's technically possible people would rather continue to evolve and try to maintain source parity rather than fork.

So we'll try. 26 is still in pieces and there will probably not be an unstable release, just a set of changesets for the perverse interested. However, I am pretty certain 26 will work. After that, my thinking is to jump to either 28 or 29, since if Australis doesn't gel, there's no point in expending additional effort. I'll have some time over the holidays to deal with this since I'll be between terms in my Master's degree program.

Meanwhile, 24.2.0 is out. This includes the performance tweaks I discussed (issues 253 and 254); release notes and downloads available at the usual places. This will be the first official release of 24 to the general public, so let's make it a good one. Assuming no obvious issues, 24.2.0 becomes live Monday night Pacific time as usual and we will wave good-bye to TenFourFox 17. You'll also get to see our new Mavericks-inspired Apple swipe, as is my custom. :)

The newest technical concern is Mozilla's renewed push towards Electrolysis, or multi-process Firefox. Most of you are familiar with Chrome's use of a process per (sometimes several) tabs, although Internet Explorer 8 actually pioneered this approach some months before. The issue Electrolysis specifically addresses is unique to Firefox: most of the browser is written in JavaScript, but because all JavaScript runs on the main thread which also services user requests (including from web pages, because accesses to the document are not thread-safe), when a page's script stalls, it also stalls any browser code that is waiting to run, and the entire application must wait for the script to run or be cancelled. Using a JIT to compile JavaScript only reduces this latency; it does not eliminate it. Compared to this problem, improving browser stability and security is actually a remarkably secondary issue, though separate processes also achieve that too.

Running webpage scripts in a separate execution context is not a new idea; Mozilla played with an idea called "supersnappy" which would have done this in single-process Firefox, but the plumbing required was very complex and delicate. Electrolysis itself has been around in various forms as far back as 2009, and modern Firefox Android and Firefox OS use it, but the desktop browser does not due to lots of things that don't work. Even now, the browser is only up to the point where basic browsing works; many add-ons won't work at all. (Don't try to enable this on TenFourFox, by the way. If it starts at all, it will likely crash very soon afterwards. I'll explain why in a moment.)

The first iteration will just have a single content process within which all the tabs run, which diminishes the memory overhead while still getting most of the stability benefits and some of the performance benefits. However, a lot of work must be done on add-on compatibility. Most Jetpack-based addons (like the QTE and the MTE) will work fine with Electrolysis Firefox, but older or more low-level addons (like OverbiteFF) may need modification. There are also lots of potential race conditions because you now have two queues of JavaScript which may deal with items in non-deterministic order. Since the browser interface is JavaScript too, they really need to get this right.

The amount of time required to get this plumbing in and fix all the outstanding bugs indicates that this will become mandatory at least after Firefox 31. That's good, because as written, it won't work with 10.4; we don't have the functions needed to spawn processes the way IPC Chromium expects (which handles multiprocess Firefox), and they are stubbed out with debugging messages in the current version of TenFourFox since it doesn't need them. The underlying work to fix this is issue 66. Even then we may have IPC bugs in 10.4 getting the processes to talk to each other, although I believe Electrolysis is achievable on Tiger overall and will be beneficial to us too ultimately. Fortunately, we don't have to worry about it yet.

Back to work on TenFourFox 26, after I finish this paper. *sigh*

I'm sensing you may or may not want to continue

I got my Williams Star Trek: The Next Generation pinball in, complete with colour dot matrix display, to complement my beloved Stern Sopranos pin. Naturally, I had to thoroughly test it during the 90 day warranty period, because, you know, it's a rough job to play a lot of pinball but someone's gotta do it. The TNG game, for those of you unfamiliar with it, is a pinball classic that is constantly at or near the top in table rankings; it's one of Williams' most inspired offerings from the great Steve Ritchie.

Set for free play, once you've drained your three balls (and this game will gobble them), Counselor Troi will ask, "I'm sensing you want to continue" (to "buy" a ball). And really that's the dilemma we're at, so you tell me.

First some good news for a change. The issue I had with Firefox 26 also cropped up in 24 and Mozilla determined it was a bug and fixed it, which enabled me to get Fx26 compiling further (it still doesn't get through JavaScript, but this is due to code I just have to sit down and write, not a fundamental compiler issue). I've also completed my audit of the Australis widget changes and while some of them are going to be a headache, I'm actually a bit more confident it can be ported to 10.4. I'm still not fully convinced, but some of the 10.5-only code I was worried about was replaced with more portable implementations that also will simply work better.

Still, let's review how far we've come. We are still on a supported branch of Firefox, albeit an ESR branch (ESR24), over three years after Firefox 4 beta 7 ended PowerPC support; we have successfully ported through twenty revisions of Firefox since then. This branch of Firefox will officially receive security updates until at least early 2015, at which point we can still maintain it with backports. We implement in Fx24 most of HTML5 and, arguably, more of CSS3 than anyone else. We also already implement most of ECMAScript 6, the next standard for JavaScript (TC39/"Harmony"), and again, more completely than most other browsers. Maintenance of these three prongs (HTML5, CSS3 and TC39) is necessary to keep us current with web standards, but we are in an excellent position with regard to compliance right now.

Given all that, is it in our best interest to continue to try to get to Firefox 31, especially with all that's coming down the pike (Australis being the most notorious)? Well, that's the part I'm not sure about. I don't want to prematurely abandon keeping current with somewhat-bleeding-edge Firefox, but my experience with doing backpatches on Firefox 22 at the same time I was trying to fix Firefox 24 and maintain Firefox 17 was not a lot of fun, and I don't want us marooned on a version of Firefox stuck between ESRs with no updates at all. But I lay out the pros and cons:

Pros of staying on 24

• No Australis (also a con).
• Officially supported security and stability updates, at least until early 2015.
• More time to focus on improving the port instead of desperately attempting to keep up with the Mozilla version treadmill because merging changes will be a lot less work.
• The IonMonkey JavaScript compiler port will be a lot easier to do. (Remember, we're still on PPCBC, which has much lower latency but generates much less efficient code.) In fact, much of it is written already, so it's just a matter of getting it working.
• We can start customizing it more. One perennial request is a proper AppleScript implementation. Another is a built-in user-agent switch. Built-in adblock might also be a thought. And lots of people still want H.264 support.
• We can still implement many of those missing TC39/CSS3/HTML5 features. These features are written in high-level code in later versions of Firefox that are not far removed from ESR24, so they can be backported relatively easily.

Pros of trying to get to 31

• Australis (also a con). We can't get to 31 without it.
• If we get to 31, then we get security patches well into 2016, and could even look to continuing the port past that point.
• Although IonMonkey will be harder work to get running, it is likely to be quite a bit faster due to JavaScript improvements along the way which 24 does not have.
• Better add-on and theme compatibility.
• Bugs in the original TC39/CSS3/HTML5 features we are trying to add to 24 will already be fixed without us trying to track them down.

Cons of staying on 24

• No Australis (also a pro, depending on your perspective).
• When security support ends from Mozilla, we must provide our own.
• Themes and add-ons from versions of Firefox after 25 or 26 or so may have incompatibilities.
• We need to find and fix ourselves bugs in the original TC39/CSS3/HTML5 features we are trying to add to 24.
• It is highly unlikely further-out-future standards like CSS4 will be easily portable to 24.

Cons of trying to get to 31

• Australis (also a pro, depending on your perspective). We can't get to 31 without it.
• If we don't get to 31, we're stuck rolling our own security patches on a totally unsupported branch (i.e., whichever was the last version that worked).
• Several new technologies are landing, including off-main-thread compositing (even for software rendering, which is our standard operating mode) and generational garbage collection. These might not be portable or even compileable, but they will almost certainly be mandatory.
• A large libvpx change is scheduled for Fx28 which might break our AltiVec WebM acceleration.
• Graphics and widget changes are very likely to accumulate more interface glitches, some fixable, some less so.
• When 10.6 support goes, all supported Macs will be 64-bit in addition to requiring all the interface changes of 10.7+. If Mozilla completely deprecated 32-bit support on OS X, the only Power Macs that could support a true 64-bit build even theoretically are the G5 Macs, but there is no 64-bit Carbon (and we depend heavily on Carbon). Although 10.6 usage is still high, it is virtually certain to have been abandoned by Apple now that 10.9 is out, and Mozilla would most likely drop 10.6 support just prior to ESR31 based on their previous behaviour with 10.4 and 10.5.
• Much less available time to work on 24, which would still receive builds.

I'd like to hear your thoughts about this so I can make a decision. I might jump from 26 all the way to 29 to see if Australis will actually build (if it does not, this is a showstopper). But I'm uneasy about 10.6 support being removed as well in the (likely) very near future, and certain elements at Mozilla have been very aggressive about removing legacy code we might have been using. Either way, even if we stick with 24 as the basis for our new "unstable branch," we have to consider the consequences carefully.

Still waiting for build tags on 24.2.0 and then we'll build it and get it out to you; I'm hoping for Friday. And a big thanks to Chris T and the localizers for their hard work on our new 24-specific langpacks (see issues 42 and 61). 24 should be a good launch no matter which way we decide we want to end up.

Uproar-a australis: Australis lands, cars and cities burn, people die in streets

UPDATE: Australis is now targeted for Firefox 29.

If you've been living in a cave with D.B. Cooper counting his inflation-ravaged ill-gotten gains over the weekend, you may have missed the news that Australis landed in Firefox slated for Fx28, and appears to be sticking (I didn't make this announcement on the day of because I suspected it would be backed out for bustage, and it looks like my guess was wrong). Australis, of course, is Mozilla's new interface for Firefox, radically replacing the current appearance which has remained relatively unchanged since a minor facelift and modifications for Firefox 4.

The response has been decidedly mixed at best. One particularly dire estimate from Mozilla's own feedback system says only 20% of Firefox nightly users like the changes, though this is a small sample of apparently really p*ssed off people, and may reflect bugs or add-on incompatibility along with the differences in appearance. Still, these are the users most likely to enjoy new shiny and least likely to be offended by change purely for being change; they're on the nightly build branch, after all, so this early feedback bodes poorly. Australis' most controversial changes include larger icons and larger tabs, no more small icons mode, removal of certain customization options like hiding the navigation bar, and completely dispensing with the add-on bar altogether (marked for death in Fx4 but restored as an option after fierce complaints; it now appears that the stay of execution has expired). Oh, and tabs on the bottom? That's been marginalized for some time and now it's gone too.

Irked users are, of course, fighting back (paradoxically demonstrating Mozilla's continued high level of customizability even post-Australis); there is already an add-on to restore most of what was lost, and the most incensed Nightly users are moving over to the holly branch, which is a temporary stopgap branch of Fx28 using the previous chrome created by Mozilla in case a backout was required. (It is not likely to persist much longer now.) At least one custom Firefox rebuilder, Pale Moon, is making their intentional omission of Australis a selling point.

Australis just adds to our porting woes. Remember, we are committed to 10.4 support; there will not be a 10.5-only TenFourFox, at least not from me. Although initially I had optimism that the technical issues could be worked around, we already have at least one glitch due to the underlying widget changes (issue 247) and future changes will likely make this worse. There are also new dependencies on components of the Objective-C 2.0 runtime (10.5+) which probably can't be effectively implemented for 10.4.

Australis also has practical considerations for our older machines. Its performance is judged to be similar to old Firefox, but this is on systems with hardware acceleration; software is now a fallback, and we render entirely in software. The larger tabs and icons have a big impact on machines limited to fixed resolutions, particularly the iBooks and iMacs: this iMac G4 I'm typing on has only a 1024x768 display, and tabs this large will impair its useability and reduce the content area. If you want to get an idea of the magnitude of this effect, load up Tenfourbird, the tabs implementation of which Australis is approximately based upon. (By the way, this is not the fault of our Anonymous Builder in the Land of the Rising Sun; this is the way Thunderbird ships, and as an avid user of Tenfourbird for Usenet, we are glad to have him in the 10.4Fx ecosystem. Arigatoo gozaimashita.) Look at the swooping tabs in Tenfourbird. Those are, roughly, the size of the Australis tabs. See how much more real estate they take up? While you're at it, increase the size of the icons if you're in small icons mode, maybe throw a few more icons up there (that's where your addons will go). What does that do for your productivity?

In the early iterations I thought Australis would be a nice step forward and make the browser more attractive and pleasant to use. And, arguably, it does at least modernize the appearance, even if (once again) it's clearly a design decision driven by Google Chrome. But I am now rather repelled, personally, by the significant functionality cost and furthermore quite alarmed we will be unable to make it operate reasonably well, if at all, on a 10.4 Macintosh. Off-main thread compositing is also looming on the horizon, which may not be compatible with our threading model without significant work (and has a big chance of causing graphical glitches on software renderers), and I've only made a small amount of progress on getting JavaScript in Firefox 26 to compile without backing out the patch I don't want to back out.

It's not a good day.

MacPorts bustage and trouble with 26

Floodgap was again down today because of a nasty winter storm that knocked the power out for almost twelve hours (the UPSes are, best case, good for maybe three). There was some hideous arcing early this morning but it doesn't look like there was any hardware damage. Unfortunately, the wind is still really howling outside and I hope we don't have a repeat episode tonight.

For 24.2.0 final, scheduled for 10 December, there will be a total of three tweaks on tap: increasing the garbage collection timeslice (as mentioned before), which should also help reduce memory usage by cleaning up more fully; again locking cores to one; and removing the blurred boxshadow from the video stand-alone document so that playback doesn't chug when the blur gets invalidated and must be redrawn. No other critical bugs have cropped up.

That's the good news. After this, it's all bad news. The port for 26 is running aground; JavaScript does not compile with either gcc 4.6.3, 4.6.4 or 4.8.2 due to a security patch of arguable utility. This patch does not compile when applied to 24 either, but we can back it out safely on 24 because no functionality actually relies on it, and there is some question over whether the problem it fixes is even exploitable. This isn't true for 26 because it may be an issue for generational garbage collection, which is imminent in Firefox, and code likely does depend on it. I am waiting for Mozilla's opinion since I'm not sure how to safely rewrite the code so that it compiles and is forward-proofed.

Oh, but before I could do that, trying to install gcc 4.8.2 caused MacPorts to try to rebuild Perl 5.12, which failed (even though I already had it on the system) due to a linker problem with Xcode 2.5. Forcing the install caused it to fail building cloog due to bad isl headers. Upgrading isl manually enabled it to complete and then I could build the other prerequisites and told it to ignore the Perl one, and finally I successfully built gcc 4.8.2 ... which then broke gcc 4.6.3 because of the new compiler libraries. One rebuild, a lot of cold sweat and about 24 hours later, the toolchain is working again and I'm crosschecking the libraries to make sure. As a result, we're on gcc 4.6.4 but 4.6.3 will still be supported, since it was known to work. All this to say that you should be very careful about MacPorts updates right now -- it is doubtful anyone is routinely testing port changes, let alone new ports, against 10.4 or 10.5 and several ports are already demonstrably busted. I need to backup /opt and then I'll try to submit fixes to the portfiles for 10.4 as I have copious spare time. It may be worth our while to generate a binary set of the toolchain components that can be installed separately to guard against this, since it will almost certainly happen again in the future.

I have a low threshold for not advancing to 26 even if it works, mostly because I don't want to be dropping source parity on a non-ESR release so close to an existing ESR branch, and because some of the stuff for Australis that is landing now is alarming and may not even be workable for 10.5, let alone 10.4, which may force our hand anyway. (But hey, I predicted our demise for Firefox 5, 10 and 19, and we made it through all of those releases; still, we've existed much longer than we've had a right to.) I'll make that decision based on what changes I have to make to get 26 to work. If these changes are unacceptable or impossible (or I can't figure out what to do in time), or the resulting product is iffy, then we stay on 24 and we go to feature parity. I'll decide this after 24.2.0 comes out.

The MacTubes Enabler. Because we loved all those damn System Enablers back in the day.

H.264 is still several cycles away -- IonMonkey and Australis have highest priority for new development. But that doesn't mean we shouldn't look for shortcuts, like the QuickTime Enabler, which has gradually evolved over time into a general purpose tool for pushing video to QuickTime for external decoding (where possible, of course, and it isn't always).

In that vein many of you have explored and used MacTubes, which is a delightful tool for direct playback of YouTube videos. It's kept up to date with Google's frequent screwings around on the backend, it has a variety of playback methods including Flash, WebKit and QuickTime, and it's probably the highest performance way to play YouTube videos on a Power Mac (even HD is possible with sufficient grunt). I downloaded the source to play with and discovered it can be fed a URL through Launch Services, so I tried that, and it worked -- it figured out the video location and automatically started playing it.

Thus, may I present the MacTubes Enabler for your consideration. The MTE reuses a bit of the Carbon-CoreFoundation-JavaScript glue from the QTE, but is far simpler: it just pushes the YouTube page you're on to MacTubes, and MacTubes does the heavy lifting and starts playing it. Piece of cake.

The MTE also has a trick that the QTE doesn't yet: you can direct it to start playing in MacTubes and close the tab in one step. One big irritation involving the QTE, and with YouTube in particular, is that YouTube will start streaming the WebM video while the QTE is trying to load the H.264 version at the same time (this can be bad enough on a slow link to make QuickTime think there's no data and give up). There used to be browser addons that would defeat YouTube's autoplay feature, but none of them work anymore and Google seems to be trying to bust these techniques also, thus this feature (that will always work): now the full bandwidth is devoted to streaming the video through MacTubes. If people like this feature, I'll add it to the QTE. (Another idea might be to back up to the previous page, or if there is none, then close it. What do you think?)

The MTE does not replace the QTE (and, for that matter, the MTE is completely independent of the QTE; you can have both of them installed simultaneously, even): you'll still need the QTE to play non-YouTube HTML5 video, which MacTubes doesn't handle, and if MacTubes becomes unsupported one day we'll still need QuickTime. (However, I'm able to build MacTubes from source on my G5 independently, so that's a solveable problem if its author someday decides to hang up his hat.) I'm also not planning to merge the MTE and the QTE for the foreseeable future because it's nice to have a playback alternative, and saving video and getting to HD formats is a little easier in QuickTime Player.

Consider the MTE a beta and give it a spin. Download it from its wiki page. The MTE requires TenFourFox 24. Test it out on Jean-Claude Van Damme injuring himself, or maybe some of MTV's finest cartoons when they actually played music.

17.0.11 is live. I'm going to make one tiny change to garbage collection scheduling to help intermittent stalls prior to release of 24.2.0, which should be the first public release. It looks like we will have a mostly full set of langpacks, too, so a "great work, guys" to Chris and all the localizers.

17.0.11 rises from the grave like an undead zombie

Mozilla is issuing a proactive fix for a security issue with secured connections that affects all versions of Firefox going back to ESR17. While nothing is currently exploiting it, the issue was inadvertently disclosed through a publicly visible commit, so the decision was made for an urgent point release and we're going to go ahead and match that for 17. I don't know when this will hit the release channel officially but I will probably push the button for us Friday night or Saturday. There is only that one change in this release and the 17.0.9 changesets still apply. Downloads, release notes (for the last time, hopefully).

24.1.0 is affected by this also but it's going to be replaced by 24.2.0 in less than two weeks, so I'm going to simply monitor the situation for right now because the fix in question is merely proactive (and an exploit must be PPC OS X-specific). I have not had any more reports of menu bar freezes, so I am going to assume issue 248 is fixed by our hack. If you are a localizer, time is running out before we need to nail down the langpacks for launch. Tune in to issue 42 and/or issue 61 to coordinate with Chris.

I'm nearly done merging the new patches into what will hopefully become TenFourFox 26 and then we will be back on target for the new unstable branch, assuming it works. I'm still hoping for IonMonkey by Fx27 or Fx28 and I should have some time to get to this during the holidays. Still no word on the ETA for Australis.