It's time to stop using Flash on Power Macs. Period.

There are already a large number of reasons not to use Java on Power Macs, not the least of which being Flashback, which exploited a hole in the Java VM sandbox to run a native binary and thus gain control of the Mac. If the binary had been compiled universal, then the exploit would have worked on Power Macs, too.

Well, Flash just got that sort of exploit, but it's worse, because this attack will work successfully against Power Macs too. "Rosetta Flash" (how appropriate!) can steal credentials and cookies by generating valid, malicious SWFs through JSONP that abuse Flash's ability to bypass the same-origin policy that would normally protect against such an attack.

There are ways that a service can protect itself from Rosetta Flash, but that's where it gets even worse for Power Mac users: the recommended server-side change to retard the attack won't work on Flash 10.1. Yes, you heard right. Even if the server implements the recommended server-side protection, Flash 10.1 will still download and execute the malicious SWF. And that's critical, because there is no Flash 10.2 or above for Power Macs, just various hacks that change the version number.

This exploit is fully weaponized, as they say in the biz. It's ready for any attacker to use. So Flash is dead: it is no longer safe for use on Power Macs, TenFourFox won't run it anyway, and if you are still using TenFourFox 17 to run Flash applets, I warned you this day would come.

So far all indications are that beta 3 is substantially faster than beta 2. I will make one last tweak to the garbage collector timeslice before release for a little better long term throughput. The only remaining bug is issue 280, which is a crash with the G5 using 7450 branching; I'm just going to revert to G5 branching as was originally in 24/26/29. I'm now starting to think that the performance delta might be a benchmark artifact anyway, and it's not worth the trouble to debug prior before release. Meanwhile, 31 will come out on schedule simultaneously with 24.7.0 to conclude ESR24. Another year of support ahead!

Well, let's just wave our hands around a bit with 31.0b3 for you

Today in the USA, we celebrate July 4th where America freed itself from the little-endian tyranny of King Tim by throwing Mac Pros into the harbour. Or something like that, I was always daydreaming about Alyssa Milano in civics class. Anyway, Beta 3 is now available (downloads, release notes) corresponding to the Firefox 31 beta 6 code drop. I am not able to distinguish much performance difference from 29 now, and the initial beachball with Adblock is gone on the quad G5 and lasts only a few seconds on the iBook and iMac G4 systems. On battery in Reduced mode, I consider my iBook G4/1.33's performance acceptable -- I can get into Gmail and my office mail much more quickly, and scrolling is significantly less sluggish. This also fixes the problem with spurious extra Window menu entries. Make sure you update your Adblock Edge and Plus; there were some recent updates for ABE, at least.

There were two problems with JavaScript, and I appreciate Chris Trusch for giving me reliable figures that I could replicate (performance issues cannot be fixed without them). For this purpose I compared performance in debug builds, which is not something I usually do for obvious reasons and is what led to me finding the second problem. Indeed SunSpider and microbenchmarks were performing worse in 31 and it turned out to be another side effect of the added Ion optimization analysis I had to work around once already. I added another check to not run it except in the situations that 29 runs it, and while the analysis is more heavyweight than 29's, the added improvements to inline caches in 31 generally even the impact. I think I can port this change forward; it's not very complex. Interestingly, V8 was at most fractionally perturbed by this, at least on the quad, so we're going to have to watch both benchmarks again in the future.

However, when I built the optimized version, it was still the same speed as beta 2 despite the debug build now being substantially faster. That didn't make any sense at all. Why would a debug build now beat the optimized build by almost a factor of two? It's not only bigger, it's deliberately more inefficient!

After about a week comparing source code, it became obvious that indeed there was nothing in Mozilla's debug code that was doing an operation with side effects or something getting cached that wasn't in the opt build. To prove that, I turned Mozilla's -DDEBUG on for the G5 opt build and the numbers indeed sank further. Incredibly, Shark would run the JS shell at least (even if it won't run the browser, so now I'm reexamining my theory about pilotfish -- perhaps it's just objecting to the size of XUL, which would not be surprising), and comparing the performance traces in Shark showed that the G5 opt build was slow across the board even in things that had no debug code at all, like C++ object constructors. WTF?

The other difference in a debug build is the compiler settings. So, over the second week, I did like we used to do to find the bad extension and ran and re-ran builds with different permutations of compiler and build settings to find the setting in the debugging configuration file that made the difference. And the winner was, incredibly, --enable-debug-symbols=-gdwarf-2 -- adding that to all the optimization configs greatly improved performance. For JavaScript, and now the entire browser, the opt build is once again substantially faster than the debug build just as you would expect. (One other note is that the G5-specific branch stanzas we relied on for 11-29 look like they are not working well in PPCBC, so the G5 build now uses the same shortened branch stanzas Ben innovated for JaegerMonkey on G3 and G4. This further improves V8 by about 8%.)

I don't have a good explanation for why DWARF-2 symbols weren't needed in 29 but are needed now; all I can think of is that somehow we crossed some internal maximum and we need the more efficient modern representation, even though the opt build does strip out most of the symbol table. This may have a consequence for debugging and crash reports, but I don't think we can go back to the old STABS symbols for 31. We'll just have to deal with the problems as they come up.

Your remaining homework:

• I'm toying with bumping the GC timeslice back to 100ms as it was in 29 (it remains at 30ms right now, same as 24). You can try this by setting javascript.options.mem.gc_incremental_slice_ms to 100 and restarting the browser. This should not make much immediate difference; this is for later on when there are more compartments for the garbage collector to search. I just want to make sure that it doesn't slow the browser initially.

• Now that we have substantially changed the browser foundation, we should reexamine some earlier performance conclusions. If you are on a multi-core Power Mac, such as a dual G4/G5 or the quad, try turning OMTC back on and restarting the browser. Does it help? Does it help on a uniprocessor Mac? It's hard for me to judge on my test systems. To try this, toggle layers.offmainthreadcomposition.enabled to true and restart the browser. The G5 seems a bit better, but it's a wash on the iMac, and the iBook is still questionably worse.

• The most important question: release 31 on time on July 22, or wait for 31.1? It's important to us to prove to the Mozilla community we're still a viable and well-maintained port, and I'd like to release on time to make that clear. However, if you have a good reason to wait, I'm willing to consider it; either way, though, we will release no later than 31.1. We need to stay current and ESR24 is going away.

I have not come to a firm decision about dropping source parity, though some code changes in 33 are making me unhappy (they're not portkillers, but they're a lot of work and will invalidate substantial portions of our changesets). I might jump straight to 33 from 31 to deal with those and the irregexp migration at the same time. If that goes awry, that would make my decision for me.

One outstanding bug is a printing crash on Tiger Server only. I have never supported the Server versions of 10.4/10.5, mostly because I have no way to test them and I don't have a need to run OS X Server personally. However, if you're using it, please look at issue 279. While I will not hold the release to fix this and I don't have the ability right now to fix it myself, if you do, I will accept your patch as long as it doesn't regress the regular OS X client versions.

In July we will have our annual state of the Power Mac userbase, and I also have some fun future blog posts in the hopper on my expensive but incredibly fun Amiga 4000T (with Picasso IV, 68060 CPU card and Hydra NIC), and my old Wallstreet G3 now dualbooting Rhapsody 5.6 (Mac OS X Server 1.2) and OS 9.2.2. Plus, I picked up a Cube G4 when I was in Berkeley last week that's waiting for a power supply. The shell is in excellent shape with no major nicks or scuffs and I think it'll be a great test system for playing around with MorphOS in my copious spare time. If you're in the Bay Area and looking to pick up some more Power Macs, they've got plenty at M.A.C. on University and Shattuck and they're willing to deal. Check it out.

Progress report on 31b3

After almost two weeks of digging I was able to find two separate issues, one specifically in JavaScript and one affecting most of the browser, which seem to improve the benchmarks back to 29-levels. However, the proof is the browser's overall performance, of course, and when I fired up the G5 test build this morning before work, even with the processors set to Reduced mode I was delighted to note there's no more beachball with Adblock. This is a very good sign, but I'm going to withhold judgment until I've done some more analysis on the iBook G4; the iBook, especially if the CPU is throttled, has had the most issues with 31. The G5 is building a 7450 test build right this moment while I'm typing this over a coffee Mr. Pibb break. If the iBook's performance on battery is acceptable, then we will proceed with beta 3, hopefully over the weekend.

OMTC and generational GC will remain disabled for 31 just to give us every possible advantage.

Well, let's see how less bad 31 beta 2 is for you

OK, beta 2. This is the same Mozilla code drop as beta 1 with the following changes: IME keyboard composition of characters is fixed (accented characters, Japanese, Chinese, Korean, etc.), navigating to an MP3 audio file doesn't crash the browser now, off-main-thread compositing is turned off, generational garbage collection is turned off, and I've tuned the GC timeslice a little more based on some heavy testing this afternoon while hacking up phlegmy yuck from my inflamed respiratory passages.

Of the remaining problems, performance seems a bit better overall. The G5 doesn't care about OMTC, but does benefit from having GGC off. The Sawtooth, iBook and iMac G4 systems did much better with both of them off, so I turned everything off. The stall with Adblock is still there, but is literally about half as long (i.e., I measured it with a stopwatch), so this is an improvement, and it's still one-time-only on all of my test systems.

This is, at least right now in my virus-addled state, as much as I am currently able to do to crank up the browser until I can get better profiling support again. It gets it back to a state approximating Fx29 (minus the bugs). I am going to ship 31, and probably when it reaches release -- the work is done, and staying on 24 is not a viable option. Download it and get used to it. But while you use it, you have a small question and a big question for your homework assignment:

1. It looks like Safari bookmarks import is broken again. It is possible, nay, probable, that Mozilla doesn't test against Safari 4 anymore and possibly not even Safari 5.0. There is no Mozilla bug for this but that might simply be because Mozilla doesn't care about older versions of Safari. I don't know how bookmarks change between versions but it's probably time to consider just ripping this code out since we advise people to use HTML export from Safari anyway. Opinions?

2. Is 31 where we should drop source parity, i.e., fork and add things to 31 rather than trying to keep up? It's far easier to do this at the beginning of an ESR cycle because we can just start adding things we want (HTTP/2, SPDY updates, NSS updates, root certificate updates, ECMA6/HTML5/CSS3 features) as they start rolling out while still having security support from Mozilla, rather than try to catch up on a huge backlog when support runs out (Classilla). I had long thought what would doom us is some 10.6+ specific feature that's critical for the browser, but so far we've been able to hack around all that. Instead, what's more likely to doom us now is that our machines are just getting too slow to handle what Firefox expects to throw at them. The average age of a Power Mac is at least a decade; even the quad G5 is celebrating its eighth birthday. We are already having to cut marquee features to keep browser performance acceptable, and soon we will not be able to cut them: while we might get away with no generational GC for awhile (especially since Mozilla doesn't use it on FxOS devices yet), we already know OMTC will be mandatory soon, and those are just the land mines we know about right now. If we fork and go to feature parity, at least we can keep the browser core reasonably up to date but not have to contend with these issues.

The situation I worry about is that we will struggle up to 38 and have a browser that is crushingly slow on all but the highest-spec G5 systems with no solution for laptops or other G4 computers (let alone the G3), and it will be much harder to backport the things we want by that time. 31 is already a substantial compromise between performance and future compatibility. Where should that dividing line be? And don't forget that 10.6 support will be fading away too; when 10.6 goes, Mozilla can make even more assumptions about the hardware that won't be true for us. (Right now 10.6 is still holding on to about 20% of the Mac user base, which is an incredibly stable figure, but it's not growing and it's likely to slip as hardware ages, computers are upgraded and Apple stops supporting building against the 10.6 SDK.)

While you think about that, downloads, and updated release notes. There will be one more scheduled beta a couple weeks prior to final release just to put any lingering issues to bed, and in July we'll also do our annual update on the state of the Power Mac userbase. Now I have to go cough up some more nasty mucus, so excuse me.

Well, let's see how bad 31 is for you

I'm going to release what I have so far and see what you think. I'm typing this blog entry in 31.0b1, so I guess you can consider that an improvement over last time. Localizers, strings are frozen, so you can start your engines (see the announcement in issue 42).

I still don't have profiling working in any meaningful sense, even though I can compile and run a gprof build successfully; it looks like gprof has a bug on OS X preventing it from generating timing information, so it might be worth a dive into gcrt0.o some weekend when Scarlet Johansson is not visiting. It does generate function call counts, though, so it does run. I still need to do some more experimentation on how to get some sort of runtime instrumentation operating again.

While doing some comparisons to see what else had changed between 29 and 31, based on your glowing reviews of 29, it appears there is a "bug" in 29 that I "fixed" in 31. In Fx26, I tried using our custom CoreGraphics backend for doing web page rendering, found it too slow and occasionally crashy, and set it to use Cairo as before. In 29, Mozilla changed the backend selection code; now content couldn't be anything but CoreGraphics. In 31, I discovered this, and "fixed" it to use Cairo. Not only does setting it to CoreGraphics work, though, it's now become incredibly faster by comparison. In fact, if you're on a very fast G5 like a quad, try this trick: go to a standard-definition YouTube WebM video, start it playing (wait for the comments to load), make sure your processor is set to Highest performance, and click Full screen. This quad G5 effortlessly scales a 16:9 SD WebM video to fill an entire 1920x1080 display. Only hardware can do that. That's some hot stuff. I like.

So that was most of the problem; the rest was tuning the GC a bit more, and something that cropped up on the G4 iBook/1.33 which may have something to do with this WebM regression people are reporting on G4s. For 29, Mozilla introduced off-main-thread composition, which as we discussed uses a background thread to receive screen updates. These updates are coalesced; OMTC won't bother displaying half-frames, even though it will show full frames as fast as it receives them. On the quad (and probably any dual CPU Power Mac), this is no problem because the thread runs on another CPU and the G5 generates plenty of frames plenty fast. On G3s and G4s with no power management, this is also no problem, because they don't power throttle and run at their max clock speed, and frankly any hardware assistance at all will be much smoother than the old method. The 1GHz iMac G4 and the 450MHz Sawtooth, for example, really like 31.

On laptops or uniprocessor desktops set for power management, however, the processor may be throttled or forced to clock slew, and OMTC will be starved for frames. We shouldn't be angry with Mozilla for doing this; it's unfortunate but logical given that virtually every Intel consumer chip in the last few years has been a multicore device.

One solution is to crank CPU performance up to Highest or at least Automatic, but this isn't much of a solution for a laptop running on battery. The other solution is to turn OMTC off (set layers.offmainthreadcomposition.enabled to false and restart the browser). Now, I don't like this solution much because it's totally unsustainable: Mozilla has made no secret that main-thread compositing will be removed as soon as the last platform supports it (Linux), and cool things like asynchronous pan-and-zoom require OMTC. Disabling OMTC will be supported on 31 for as long as the ESR lasts, but I predict main-thread compositing will be removed somewhere around Fx33 or Fx34 and then you'll have to deal with the additional CPU requirements; OMTC is such a fundamental change that I cannot realistically maintain the old rendering system.

If this makes a substantial difference to WebM or other responsiveness on the systems in this group, since it doesn't really affect the outlier systems, I'm willing to ship 31 with OMTC disabled (I'll need lots of good evidence that it helps, though -- something reproducible, not anecdotal reports). However, you can be sure that it will disappear in the very near future and this will be the last stable branch we ship this way unless Mozilla finds a critical problem with it.

The only other remaining issue is shortly after startup the app beachballs for around 30 seconds. I can't make this happen on a clean profile, so I suspect an add-on is causing it, but I can't figure out which one between the affected systems; it's possible a number of them behave this way. The browser only does it the one time, so it's more of a nuisance, but it would be nice to fix. If it does it for you as well, I'd appreciate you trying to isolate what seems to set it off (make sure that it doesn't occur with a clean profile first, though).

Remember to make sure the MTE and QTE are up to date, and to undo any custom GC settings changes before you upgrade (download, release notes). I have not decided if we will release on 31.0 or 31.1, but we'll see what you think. I have the flu, so I need to go to bed. See you in the morning.

You may have noticed there are no 31 beta builds yet

That's because you don't want to use them. :(

During the aurora phase I run the build in a separate sandbox profile so it doesn't taint my main one, and mild-moderate performance issues are expected at that level because it's still relatively early in development. Plus, generally during Aurora I'm running a debugging build anyway, so performance isn't a primary concern. But by beta, it should be good. And I'm really upset: 31 is not. After just a few minutes of minor usage, scrolling on even very simple Netscape 3.0-compatible sites stutters on the quad G5 and the browser's responsiveness is unpredictable. OMTC doesn't make a difference. I can't even imagine how bad it would be on the iMac G4. About the only thing good I can say about it is that it uses almost 33% less memory than 24 does, but as far as speed is concerned it makes 24 look like a cheetah on Dexedrine.

My rule of thumb is whether I'd use the browser in the current state. And the answer is, I'm typing this blog entry in 24.6.0. :(

The next step, normally, would be to throw it into Shark and do some profiling to see where it's spending all its time. Except ... Shark doesn't understand the DWARF-2 debugging symbols we've used since Fx19 with gcc 4.6. In fact, not only does it not understand them, it causes pilotfish to lock up in an endless loop trying to process them. (Perhaps Shark with Xcode 3 does understand them correctly(?), but even if that were true it doesn't do me a lot of good on 10.4.) Shark does work with the stripped optimized builds, but then the profiling data just becomes a clot of PowerPC assembly language and hexadecimal addresses with no correlation to source code.

Instead of releasing builds, then, I'm trying to get TenFourFox to work with gprof, since gcc 4.6 will properly emit profiling information with -pg. However, this won't be anywhere near as good as Shark, since Shark would have been able to get profiling data right at the time it started going off the rails. Furthermore, it may well be that the problem is in something essential I can't revert. If I can't figure it out, the next step is to disable generational GC; we would then revert to the exact-rooting GC in 29, which would still get some memory savings, but that's a one-trick pony -- we would not make Fx38 without GGC. If that doesn't work either, then we will have to drop source parity, because I can't see myself using the browser in its current state and I'm certainly not going to subject anyone else to it.

This also means that IonMonkey is going to get backburnered again. I can't get it meaningfully working anyhow; it will run simple scripts, but anything that triggers a bailout eventually crashes, which is the same state it's been in for the last few months. I need a second set of eyes to look at the code because I'm spinning my wheels, and I'm trying to keep the browser afloat never mind enhancing it. Perhaps you're looking for a project?

I'm really tired.

24.6.0 available

24.6.0 is available (release notes, files). Other than the usual ESR fixes, there is nothing special in this release, just a regular maintenance update. It will be slotted in Monday evening Pacific as usual.

I have not made much headway on getting past this critical problem with IonMonkey, but I'm still planning for the formal 31.0 beta to be released later next week regardless.