Well, let's see how less bad 31 beta 2 is for you

OK, beta 2. This is the same Mozilla code drop as beta 1 with the following changes: IME keyboard composition of characters is fixed (accented characters, Japanese, Chinese, Korean, etc.), navigating to an MP3 audio file doesn't crash the browser now, off-main-thread compositing is turned off, generational garbage collection is turned off, and I've tuned the GC timeslice a little more based on some heavy testing this afternoon while hacking up phlegmy yuck from my inflamed respiratory passages.

Of the remaining problems, performance seems a bit better overall. The G5 doesn't care about OMTC, but does benefit from having GGC off. The Sawtooth, iBook and iMac G4 systems did much better with both of them off, so I turned everything off. The stall with Adblock is still there, but is literally about half as long (i.e., I measured it with a stopwatch), so this is an improvement, and it's still one-time-only on all of my test systems.

This is, at least right now in my virus-addled state, as much as I am currently able to do to crank up the browser until I can get better profiling support again. It gets it back to a state approximating Fx29 (minus the bugs). I am going to ship 31, and probably when it reaches release -- the work is done, and staying on 24 is not a viable option. Download it and get used to it. But while you use it, you have a small question and a big question for your homework assignment:

1. It looks like Safari bookmarks import is broken again. It is possible, nay, probable, that Mozilla doesn't test against Safari 4 anymore and possibly not even Safari 5.0. There is no Mozilla bug for this but that might simply be because Mozilla doesn't care about older versions of Safari. I don't know how bookmarks change between versions but it's probably time to consider just ripping this code out since we advise people to use HTML export from Safari anyway. Opinions?

2. Is 31 where we should drop source parity, i.e., fork and add things to 31 rather than trying to keep up? It's far easier to do this at the beginning of an ESR cycle because we can just start adding things we want (HTTP/2, SPDY updates, NSS updates, root certificate updates, ECMA6/HTML5/CSS3 features) as they start rolling out while still having security support from Mozilla, rather than try to catch up on a huge backlog when support runs out (Classilla). I had long thought what would doom us is some 10.6+ specific feature that's critical for the browser, but so far we've been able to hack around all that. Instead, what's more likely to doom us now is that our machines are just getting too slow to handle what Firefox expects to throw at them. The average age of a Power Mac is at least a decade; even the quad G5 is celebrating its eighth birthday. We are already having to cut marquee features to keep browser performance acceptable, and soon we will not be able to cut them: while we might get away with no generational GC for awhile (especially since Mozilla doesn't use it on FxOS devices yet), we already know OMTC will be mandatory soon, and those are just the land mines we know about right now. If we fork and go to feature parity, at least we can keep the browser core reasonably up to date but not have to contend with these issues.

The situation I worry about is that we will struggle up to 38 and have a browser that is crushingly slow on all but the highest-spec G5 systems with no solution for laptops or other G4 computers (let alone the G3), and it will be much harder to backport the things we want by that time. 31 is already a substantial compromise between performance and future compatibility. Where should that dividing line be? And don't forget that 10.6 support will be fading away too; when 10.6 goes, Mozilla can make even more assumptions about the hardware that won't be true for us. (Right now 10.6 is still holding on to about 20% of the Mac user base, which is an incredibly stable figure, but it's not growing and it's likely to slip as hardware ages, computers are upgraded and Apple stops supporting building against the 10.6 SDK.)

While you think about that, downloads, and updated release notes. There will be one more scheduled beta a couple weeks prior to final release just to put any lingering issues to bed, and in July we'll also do our annual update on the state of the Power Mac userbase. Now I have to go cough up some more nasty mucus, so excuse me.

Well, let's see how bad 31 is for you

I'm going to release what I have so far and see what you think. I'm typing this blog entry in 31.0b1, so I guess you can consider that an improvement over last time. Localizers, strings are frozen, so you can start your engines (see the announcement in issue 42).

I still don't have profiling working in any meaningful sense, even though I can compile and run a gprof build successfully; it looks like gprof has a bug on OS X preventing it from generating timing information, so it might be worth a dive into gcrt0.o some weekend when Scarlet Johansson is not visiting. It does generate function call counts, though, so it does run. I still need to do some more experimentation on how to get some sort of runtime instrumentation operating again.

While doing some comparisons to see what else had changed between 29 and 31, based on your glowing reviews of 29, it appears there is a "bug" in 29 that I "fixed" in 31. In Fx26, I tried using our custom CoreGraphics backend for doing web page rendering, found it too slow and occasionally crashy, and set it to use Cairo as before. In 29, Mozilla changed the backend selection code; now content couldn't be anything but CoreGraphics. In 31, I discovered this, and "fixed" it to use Cairo. Not only does setting it to CoreGraphics work, though, it's now become incredibly faster by comparison. In fact, if you're on a very fast G5 like a quad, try this trick: go to a standard-definition YouTube WebM video, start it playing (wait for the comments to load), make sure your processor is set to Highest performance, and click Full screen. This quad G5 effortlessly scales a 16:9 SD WebM video to fill an entire 1920x1080 display. Only hardware can do that. That's some hot stuff. I like.

So that was most of the problem; the rest was tuning the GC a bit more, and something that cropped up on the G4 iBook/1.33 which may have something to do with this WebM regression people are reporting on G4s. For 29, Mozilla introduced off-main-thread composition, which as we discussed uses a background thread to receive screen updates. These updates are coalesced; OMTC won't bother displaying half-frames, even though it will show full frames as fast as it receives them. On the quad (and probably any dual CPU Power Mac), this is no problem because the thread runs on another CPU and the G5 generates plenty of frames plenty fast. On G3s and G4s with no power management, this is also no problem, because they don't power throttle and run at their max clock speed, and frankly any hardware assistance at all will be much smoother than the old method. The 1GHz iMac G4 and the 450MHz Sawtooth, for example, really like 31.

On laptops or uniprocessor desktops set for power management, however, the processor may be throttled or forced to clock slew, and OMTC will be starved for frames. We shouldn't be angry with Mozilla for doing this; it's unfortunate but logical given that virtually every Intel consumer chip in the last few years has been a multicore device.

One solution is to crank CPU performance up to Highest or at least Automatic, but this isn't much of a solution for a laptop running on battery. The other solution is to turn OMTC off (set layers.offmainthreadcomposition.enabled to false and restart the browser). Now, I don't like this solution much because it's totally unsustainable: Mozilla has made no secret that main-thread compositing will be removed as soon as the last platform supports it (Linux), and cool things like asynchronous pan-and-zoom require OMTC. Disabling OMTC will be supported on 31 for as long as the ESR lasts, but I predict main-thread compositing will be removed somewhere around Fx33 or Fx34 and then you'll have to deal with the additional CPU requirements; OMTC is such a fundamental change that I cannot realistically maintain the old rendering system.

If this makes a substantial difference to WebM or other responsiveness on the systems in this group, since it doesn't really affect the outlier systems, I'm willing to ship 31 with OMTC disabled (I'll need lots of good evidence that it helps, though -- something reproducible, not anecdotal reports). However, you can be sure that it will disappear in the very near future and this will be the last stable branch we ship this way unless Mozilla finds a critical problem with it.

The only other remaining issue is shortly after startup the app beachballs for around 30 seconds. I can't make this happen on a clean profile, so I suspect an add-on is causing it, but I can't figure out which one between the affected systems; it's possible a number of them behave this way. The browser only does it the one time, so it's more of a nuisance, but it would be nice to fix. If it does it for you as well, I'd appreciate you trying to isolate what seems to set it off (make sure that it doesn't occur with a clean profile first, though).

Remember to make sure the MTE and QTE are up to date, and to undo any custom GC settings changes before you upgrade (download, release notes). I have not decided if we will release on 31.0 or 31.1, but we'll see what you think. I have the flu, so I need to go to bed. See you in the morning.

You may have noticed there are no 31 beta builds yet

That's because you don't want to use them. :(

During the aurora phase I run the build in a separate sandbox profile so it doesn't taint my main one, and mild-moderate performance issues are expected at that level because it's still relatively early in development. Plus, generally during Aurora I'm running a debugging build anyway, so performance isn't a primary concern. But by beta, it should be good. And I'm really upset: 31 is not. After just a few minutes of minor usage, scrolling on even very simple Netscape 3.0-compatible sites stutters on the quad G5 and the browser's responsiveness is unpredictable. OMTC doesn't make a difference. I can't even imagine how bad it would be on the iMac G4. About the only thing good I can say about it is that it uses almost 33% less memory than 24 does, but as far as speed is concerned it makes 24 look like a cheetah on Dexedrine.

My rule of thumb is whether I'd use the browser in the current state. And the answer is, I'm typing this blog entry in 24.6.0. :(

The next step, normally, would be to throw it into Shark and do some profiling to see where it's spending all its time. Except ... Shark doesn't understand the DWARF-2 debugging symbols we've used since Fx19 with gcc 4.6. In fact, not only does it not understand them, it causes pilotfish to lock up in an endless loop trying to process them. (Perhaps Shark with Xcode 3 does understand them correctly(?), but even if that were true it doesn't do me a lot of good on 10.4.) Shark does work with the stripped optimized builds, but then the profiling data just becomes a clot of PowerPC assembly language and hexadecimal addresses with no correlation to source code.

Instead of releasing builds, then, I'm trying to get TenFourFox to work with gprof, since gcc 4.6 will properly emit profiling information with -pg. However, this won't be anywhere near as good as Shark, since Shark would have been able to get profiling data right at the time it started going off the rails. Furthermore, it may well be that the problem is in something essential I can't revert. If I can't figure it out, the next step is to disable generational GC; we would then revert to the exact-rooting GC in 29, which would still get some memory savings, but that's a one-trick pony -- we would not make Fx38 without GGC. If that doesn't work either, then we will have to drop source parity, because I can't see myself using the browser in its current state and I'm certainly not going to subject anyone else to it.

This also means that IonMonkey is going to get backburnered again. I can't get it meaningfully working anyhow; it will run simple scripts, but anything that triggers a bailout eventually crashes, which is the same state it's been in for the last few months. I need a second set of eyes to look at the code because I'm spinning my wheels, and I'm trying to keep the browser afloat never mind enhancing it. Perhaps you're looking for a project?

I'm really tired.

24.6.0 available

24.6.0 is available (release notes, files). Other than the usual ESR fixes, there is nothing special in this release, just a regular maintenance update. It will be slotted in Monday evening Pacific as usual.

I have not made much headway on getting past this critical problem with IonMonkey, but I'm still planning for the formal 31.0 beta to be released later next week regardless.

Yosemite is supported

We are proud to say that Yosemite is fully supported by TenFourFox.

Further work on 31

I am pleased to announce that the bugs mentioned in the last blog entry are all stamped out of 31, including editor key combinations now working again, webcam video+audio working again, the endian bug in the JavaScript VM found and, of course, a new compatible QTE. This blog entry is being typed in it. As far as I am concerned, absent a new major bug discovered during your testing, 31 is now in a shippable state.

There are two regressions I am tracking and both are big question marks. Chris mentioned a WebM performance regression; I can't reproduce this on the quad G5, but the G5 never had a problem with WebM even before the AltiVec patches went in, and my usual iMac G4 tester never ran WebM without a problem. I've turned the loop and bilinear filters down as low as possible, with fortunately minimal visual change on most YouTube videos, and made sure that video frames are being passed directly to the compositor. I hope that helps; I don't really have anything else to offer there. libvpx changed dramatically in 29 and it might be that it's just too heavy now for the G4s that used to run it acceptably.

I'm pretty sure, by the way, that OMTC (off-main thread compositing) is not to blame for this, which was also enabled in 29. OMTC puts the compositor on a separate asynchronous thread so that compositing graphics layers together doesn't block the main business of the browser. Multithreading is not great on Power Macs running OS X and is worse in 10.4, so OMTC makes some sites scroll better and some sites scroll worse, but OMTC is a prerequisite for future features like Async Pan-Zoom which finally lets you move around on the page while the browser is working on it and Mozilla is openly warning they won't support synchronous on-main thread compositing for much longer. OMTC is the future and fortunately it works on Tiger, so we're moving forward with it.

The second regression is JavaScript. While generational garbage collection (GGC) improves browser memory usage even more, it comes at a price; the changes for tracking these new short-lived objects add further overhead to the JIT and even with some additional fiddling we still take a hit of about 10% relative to 24. The G5 takes it on the chin even more, about 15%. This is still substantially better than the naked interpreter, but the interpreter is now so deoptimized that this isn't saying much anymore.

The biggest decline was due to us failing to run a particular type of analysis which is built into Ion, but we don't run Ion, so this didn't run and PPCBC generated terrible code as a result. I hacked this back together and it now works mostly as it did before, but this makes the definitive solution clear: we need IonMonkey working on PowerPC or we'll end up incessantly running into these kinds of edge cases in future versions. Plus, more to the point, Baseline Compiler was never intended as a definitive JIT solution despite the fact we're using it like one. My previous efforts with Ion ran aground in the bailout code and I think the main problem is related to Ben's sophisticated but complex branch optimization we used in JaegerMonkey (the previous JIT), so that needs to be decoupled from the low-level assembler which I'm working on over the weekend and then Ion dragged back up to a testable state. If I can make progress on that, I am considering delaying 31 until Ion passes tests (no asm.js; I continue to believe that it's irrelevant on big-endian because of all the little-endian code compiled for it) like we did with 24. If I can't, we'll just do the best we can.

For now the current plan is a 31 beta shortly after 24.6.0 which will still use PPCBC unless I have a major breakthrough; after that we'll see. 24.6.0 is still on schedule for an on-time release the second week of June.

31: close to beta (plus: new QTE and Mac|Life goes Low End)

Good news: there's going to be another stable release; 31 builds, links, gets off the ground and generally works. Yay! 24 will go into maintenance mode until 31 is ready for prime time, but the chance is better than even money that we'll have an on-time launch.

Between 29 and 31, Mozilla reworked the basic layers canvas code and this fixed the display problem with Google Maps and a lot of other sites. Plus, there don't appear to be any new regressions with the graphics code switching to Moz2D, and our "blue thumbnails" patch sticks fine in 31, so that's a lot of potential problems avoided and repaired.

The four major bugs we have left are all holdovers from 29, except this one: Google Maps does display correctly now, but it crashes with a null pointer assertion within the JavaScript virtual machine even with the JIT off. This looks like an endian problem introduced with bug 912456 or a related patch; simply wallpapering the assertion to make the browser continue doesn't cause any crashes, but Google Maps won't download new map content, so that's not very helpful. I consider this a showstopper. However, I also have a good idea where the problem is.

The second major bug is that the current version of the QuickTime Enabler does indeed hang up in TenFourFox 29+, as reported by a couple folks earlier. This is a problem with the old Add-on SDK jetpack version in v116, confirmed in that the MacTubes Enabler, which has a later jetpack, doesn't do this. Both of them will need a jetpack update for 31, but I updated the QTE jetpack and added a couple minor bug fixes and released that as QTE v120. You can get it from the QuickTime Enabler wiki page. It will work with 24 and 31 both, and then the QTE and MTE will be refreshed again after 31 is released. So that should be solved.

The two remaining issues are also 29 holdovers and while I consider them important to fix, I don't consider them showstoppers. It does look like editing keys have regressed per your reports, but not all keys have, so I'm not sure what's going on (menu keys and productivity shortcuts, for example, all function). This looks like our bug, but while I intend to fix it I don't consider this a shipping blocker for 31.0. Also, we still have the problem where webcam audio does not work (and neither does video-with-audio). However, our patch to force webcam video and video-without-audio does work, so the webcam can still be used for snapshots as its primary use case, and this is also shippable even if it's suboptimal.

31 also uses Mozilla's new generational garbage collector (GGC), which is even more aggressive about memory than the exact rooting GC in 29. The settings 29 use that we experimentally determined earlier in this blog are tuned for the older conservative garbage collector and may be counterproductive with GGC. I will likely change the time slice and some other parameters for 31 beta, so if you are using these custom GC settings please revert them before upgrading or the browser may not perform well.

Other than that, the browser appears to work fine and I think we're in good shape for another successful stable branch. Once I have the Google Maps problem fixed, I'll grab the beta source when 31 migrates from aurora and we should have a full-spectrum 31 beta release available a few days after the 24.6.0 update. Speaking of, 24.6.0 will be a security update only; there are no bugs on our side that will land, and I am not likely to do further feature work on 24 as the switchover to 31ESR approaches.

By the way, if you've got a newsstand nearby, you might want to pick up the current (June 2014) issue of Mac|Life and their article on rejuvenating your old Mac:

Even though it's a relatively basic article and their tips won't be big news to most of this blog's denizens, it's nice to see classic Macs still getting coverage in major Mac publications, and they even have a little section on Linux and alternative operating systems. Thanks to @thedoctor on App.net for spotting their shouts-out to TenFourFox and Classilla -- that's how you know you've arrived.