Tobias gets a beer. This is the G5 build of the Firefox 5 beta, although for some reason it says Aurora. This was built with his 10.4-linked 64-bit ld, and appears to have all of our standard features. And it works!
Sunday, May 29, 2011
Tobias gets a beer: Aurora on PowerPC
Tobias gets a beer. This is the G5 build of the Firefox 5 beta, although for some reason it says Aurora. This was built with his 10.4-linked 64-bit ld, and appears to have all of our standard features. And it works!
Saturday, May 28, 2011
4.0.2 final available
While we sort out what we're going to do with Firefox 5, 4.0.2 final is now available. I have decided to abandon the base-maintenance distinction for patches while I wait to see how Mozilla will handle release driving for mozilla-2.0, so it will be the same bucket o'patches for right now.
Thursday, May 26, 2011
Four-ever, yours
Let's just come out and say it: after a number of sleepless days struggling with it, it does not look like there will be a "TenFourFox 5." Before you panic, that does not mean that TenFourFox will be unsupported or will not get new features! It just means we won't be building from the post-Mozilla 2.0 codebase anymore, and there is a bright silver lining to it.
Before we do that, let's post-mortem Firefox 5 on 10.4.11 PPC first. This is nerdy, so if you just want to skip on to the new revised roadmap, go for it.
As explained in previous entries, my main concern was that Mozilla as of Mozilla 5.0 requires Chromium IPC to be built (IPC is now required), and IPC requires building a giant library called libxul. In Mozilla 2.0/Firefox 4.0, this was optional, and we built without them because Chromium IPC requires lots of 10.5-only OS functions and libxul was too big to link with the Tiger 32-bit linker.
It is possible to get Chromium IPC to compile "just enough" to make Mozilla's IPC happy, because Mozilla's IPC code does not use very much of Chromium at all. Chromium includes lots of system abstraction code, which Mozilla doesn't need because it uses NSPR, and as a result almost all of the stuff that didn't build can be safely stubbed out with empty functions. The exception is the launch code, which requires posix_spawn and friends (which 10.4 doesn't have either), but even that right now is only used by out-of-process plugins which were marked for death in "10.4Fx 5" anyway. It would be a problem when Mozilla goes to full one-process-per-tab, but that is not imminent, and we can work around that too. So that much is doable, and does compile.
Linking, however, turned out to be hugely troublesome. Tiger has two linkers, the regular Darwin /usr/bin/ld which is full-featured but only 32-bit, and the 64-bit /usr/bin/ld64 which can deal with larger files but lacks many important features. In particular, a number of the indirect linking features Mozilla requires for dylibs are not supported in ld64, though they do give just warnings instead of errors. The reason this is important is that the linker can only deal with libraries that fit within its addressing space, and even with optimization and stripping libxul is too big to fit in 32-bit space.
The first problem was Apple's libtool: it doesn't let you choose your linker. To get around this, I wrote a shim Perl script to act as a switchable ld, calling ld32 (the renamed original /usr/bin/ld) ordinarily except if LD64 is set to one in the environment. I then hacked the Python build argument generators to call another Perl shim that looks to see if libxul is being linked or linked against, and turn on 64-bit linking if so. That much works.
The next problem, however, was that debug libxul can't be linked even by the 64-bit Tiger linker: it crashes with a malloc() error, strongly suggesting this is an operating system limitation. The size of this library, by the way, approaches 750 megabytes and I'm not even exaggerating. Since I doubt Mozilla would distribute an app that large :), something is already wrong.
But that wasn't all. For G4 and G5 optimized builds, the WebM accelerator assembly code requires special linker settings and ld64 doesn't support them either, so they couldn't be built as is. (ld32 couldn't even start, by the way.) Losing this code would be a serious blow to us, but would be doable if we could get the basic browser at least to stand up.
Unfortunately, we couldn't. The G3 version did, finally, appear to link in ld64 with a crapload of warnings, but when I started it up in the debugger it crashed immediately in darwin_gcc3_preregister_frame_info. This is a very low-level symbol in the C runtime (crt1.o) and suggests that dyld could not resolve the links. This could be for any number of reasons, including the app has to be in 64-bit mode for dyld to resolve references in a library of that size in 10.4 (a show-stopper), dyld can't handle a library of that size at all in 10.4 (also a show-stopper), the linker actually did fail and didn't realize it (also a show-stopper), or I munged the code while making it acceptable to the compiler to such an extent that it can't enter its main loop (and since we can't build the debug libxul, we have no good way of debugging it). It could even be all of the above.
Mind you, I'm not totally admitting defeat, but I am pretty confident now that we cannot build Firefox 5 as written on 10.4.11 PPC, at least not with our current tools. It should be possible to build it on 10.5.8 PPC, and there are some early builds on El Furbe of 4.2 (which was 5's original version number), but I don't know how well these work and his buildbot seems to have stopped again. That doesn't help us, though, because we support G3 Macs and 10.4, and we always will. More to the point, I personally only run 10.4 and can't support a "TenFiveFox." If someone out there wants to apply the TenFourFox enhancements and run off 10.5-only builds, I will gladly direct people their way, but here we stay on 10.4.
I know Tobias and some others (David Fang?) had built some custom tools themselves. I am doubtful that the tools are the actual problem (I suspect the real issue is with 10.4's dyld), though I can't test that myself because I don't want to corrupt my working build system with unofficial tools, but here's your chance to put your money where your mouth is. If you can get it to link with your custom linkers, you will have done us all a great favour and win our userbase's ever-lasting regard, plus proven that they work. Remember, it's got to work in 10.4 to qualify, but as long as the browser can start, there is hope. Here's how:
When Firefox 5 comes out, it will move to mozilla-release, and then in 6 weeks be obliterated -- no more numerical branches. At that point, assuming no one has cracked the problem we discussed above, we drop source parity and continue our development purely from Mozilla 2.0 at feature parity state.
First, let's discuss what this means practically. Camino is very popular with PowerPC users, and 2.0 is based on Firefox 3.0 (Mozilla 1.9), which is three full generations behind TenFourFox. Despite this, it still manages to get the job done for a great many things; I myself am a former Camino user. Even Camino 2.1 will "only" be based on Firefox 3.6 (1.9.2). So even if we just phoned it in and did only security updates, functionally speaking TenFourFox would still be generationally ahead for several years. Remember how long WaMCom usefully persisted on OS 9 with nobody at all updating it?
But, of course, we will do feature updates (since we will be at feature parity), and because we are very close to Firefox 5 we can adopt quite a bit of the source code with only minimal changes and risk. We do need to pick features and support them with fixes, but the process should be much cleaner and less iffy than the case with Classilla where perilous backport glue code is sometimes necessary (and not always successful). We cannot take code that depends on IPC, of course, but layout, content and media should work, and some careful graphics updates. However, we should not mess with the UI except if it's buggy, because this may require saddling our slower users with more baggage than their computer can handle and the current browser at least is a known quantity. We will also wait on JavaScript updates in case they break the nanojit, and IonMonkey is in the future, so we should see what turns out with that.
And, since we aren't moving to TenFourFox 5, updates will still function properly in the way they are, and the plugin code won't change. So you get to keep plugins after all. I hope you're happy. ;) My arguments over security still stand, however, and I still strongly recommend something like Flashblock.
This is the new feature parity roadmap; my intention is to have a release every six to eight weeks, with four to six weeks of dev time and two weeks of "pre" testing:
4.0.2 still comes out probably this weekend. It looks very successful; no major problems were reported with the second beta, and the JavaScript speed and stability increase was universal.
4.0.3 is the next feature release. This will turn on the nanojit for the browser chrome and hopefully add load-store to the nanojit (these were features intended for TenFourFox 5 originally), along with configurable WebM filtering settings. It will also include security fixes and Firefox 5 bug fixes that we can safely backport. It may or may not include some new safe-looking features that do not significantly change the browser, such as the Firefox 5 changes and enhancements to HTTP and XMLHttpRequest. I might defer these to a '4.0.4'.
4.1 is where the magic will start. Among the features I think we need to have include Firefox 5's improvements to canvas, CSS animation, WebSockets (when they finish screwing around with the IETF spec, likely by Firefox 6) and SSL False Start. These are all features likely to be leveraged by future sites. Similarly, we should try to adopt some of their SSE/NEON vectorized improvements, especially for text scanning, along with libjpeg-turbo. These features are developing and we need to constantly watch for new bugs, so we should keep this set of new supported web features small and essential. If 4.1 is delayed, there will be a 4.0.4.
Because 4.1 changes the browser core in significant ways, we should not report ourselves as Firefox 4 anymore. We might adopt a built-in user agent changer like Classilla. More on that later.
So, the road moves on, just in a different direction. I'm going to start writing up new roadmaps and worklists in the meantime while 4.0.2 builds on the G5. And for the hacker builders, you've got eight weeks or so of Firefox 5 to see if you can get it running and succeed where I failed. In the meantime, TenFourFox remains "four-ever, yours."
Before we do that, let's post-mortem Firefox 5 on 10.4.11 PPC first. This is nerdy, so if you just want to skip on to the new revised roadmap, go for it.
As explained in previous entries, my main concern was that Mozilla as of Mozilla 5.0 requires Chromium IPC to be built (IPC is now required), and IPC requires building a giant library called libxul. In Mozilla 2.0/Firefox 4.0, this was optional, and we built without them because Chromium IPC requires lots of 10.5-only OS functions and libxul was too big to link with the Tiger 32-bit linker.
It is possible to get Chromium IPC to compile "just enough" to make Mozilla's IPC happy, because Mozilla's IPC code does not use very much of Chromium at all. Chromium includes lots of system abstraction code, which Mozilla doesn't need because it uses NSPR, and as a result almost all of the stuff that didn't build can be safely stubbed out with empty functions. The exception is the launch code, which requires posix_spawn and friends (which 10.4 doesn't have either), but even that right now is only used by out-of-process plugins which were marked for death in "10.4Fx 5" anyway. It would be a problem when Mozilla goes to full one-process-per-tab, but that is not imminent, and we can work around that too. So that much is doable, and does compile.
Linking, however, turned out to be hugely troublesome. Tiger has two linkers, the regular Darwin /usr/bin/ld which is full-featured but only 32-bit, and the 64-bit /usr/bin/ld64 which can deal with larger files but lacks many important features. In particular, a number of the indirect linking features Mozilla requires for dylibs are not supported in ld64, though they do give just warnings instead of errors. The reason this is important is that the linker can only deal with libraries that fit within its addressing space, and even with optimization and stripping libxul is too big to fit in 32-bit space.
The first problem was Apple's libtool: it doesn't let you choose your linker. To get around this, I wrote a shim Perl script to act as a switchable ld, calling ld32 (the renamed original /usr/bin/ld) ordinarily except if LD64 is set to one in the environment. I then hacked the Python build argument generators to call another Perl shim that looks to see if libxul is being linked or linked against, and turn on 64-bit linking if so. That much works.
The next problem, however, was that debug libxul can't be linked even by the 64-bit Tiger linker: it crashes with a malloc() error, strongly suggesting this is an operating system limitation. The size of this library, by the way, approaches 750 megabytes and I'm not even exaggerating. Since I doubt Mozilla would distribute an app that large :), something is already wrong.
But that wasn't all. For G4 and G5 optimized builds, the WebM accelerator assembly code requires special linker settings and ld64 doesn't support them either, so they couldn't be built as is. (ld32 couldn't even start, by the way.) Losing this code would be a serious blow to us, but would be doable if we could get the basic browser at least to stand up.
Unfortunately, we couldn't. The G3 version did, finally, appear to link in ld64 with a crapload of warnings, but when I started it up in the debugger it crashed immediately in darwin_gcc3_preregister_frame_info. This is a very low-level symbol in the C runtime (crt1.o) and suggests that dyld could not resolve the links. This could be for any number of reasons, including the app has to be in 64-bit mode for dyld to resolve references in a library of that size in 10.4 (a show-stopper), dyld can't handle a library of that size at all in 10.4 (also a show-stopper), the linker actually did fail and didn't realize it (also a show-stopper), or I munged the code while making it acceptable to the compiler to such an extent that it can't enter its main loop (and since we can't build the debug libxul, we have no good way of debugging it). It could even be all of the above.
Mind you, I'm not totally admitting defeat, but I am pretty confident now that we cannot build Firefox 5 as written on 10.4.11 PPC, at least not with our current tools. It should be possible to build it on 10.5.8 PPC, and there are some early builds on El Furbe of 4.2 (which was 5's original version number), but I don't know how well these work and his buildbot seems to have stopped again. That doesn't help us, though, because we support G3 Macs and 10.4, and we always will. More to the point, I personally only run 10.4 and can't support a "TenFiveFox." If someone out there wants to apply the TenFourFox enhancements and run off 10.5-only builds, I will gladly direct people their way, but here we stay on 10.4.
I know Tobias and some others (David Fang?) had built some custom tools themselves. I am doubtful that the tools are the actual problem (I suspect the real issue is with 10.4's dyld), though I can't test that myself because I don't want to corrupt my working build system with unofficial tools, but here's your chance to put your money where your mouth is. If you can get it to link with your custom linkers, you will have done us all a great favour and win our userbase's ever-lasting regard, plus proven that they work. Remember, it's got to work in 10.4 to qualify, but as long as the browser can start, there is hope. Here's how:
- Make sure your system is set up to build the current TenFourFox as a prerequisite (see the wiki) and that it can, in fact, build TenFourFox 4.0.
- Clone the mozilla-beta repo.
- Download the current set of Firefox 5 patches and serially, in numerical order, hg import them into the repo.
- Move your /usr/bin/ld to /usr/bin/ld32, and move SHIM.ld in the repo root to /usr/bin/ld (you need to do this as root, obviously). You don't need to undo this at the end unless you want to as it is designed to be transparent. Replace this with your custom linkers as indicated, of course.
- Build as instructed in the wiki. Good luck. Post your experiences.
When Firefox 5 comes out, it will move to mozilla-release, and then in 6 weeks be obliterated -- no more numerical branches. At that point, assuming no one has cracked the problem we discussed above, we drop source parity and continue our development purely from Mozilla 2.0 at feature parity state.
First, let's discuss what this means practically. Camino is very popular with PowerPC users, and 2.0 is based on Firefox 3.0 (Mozilla 1.9), which is three full generations behind TenFourFox. Despite this, it still manages to get the job done for a great many things; I myself am a former Camino user. Even Camino 2.1 will "only" be based on Firefox 3.6 (1.9.2). So even if we just phoned it in and did only security updates, functionally speaking TenFourFox would still be generationally ahead for several years. Remember how long WaMCom usefully persisted on OS 9 with nobody at all updating it?
But, of course, we will do feature updates (since we will be at feature parity), and because we are very close to Firefox 5 we can adopt quite a bit of the source code with only minimal changes and risk. We do need to pick features and support them with fixes, but the process should be much cleaner and less iffy than the case with Classilla where perilous backport glue code is sometimes necessary (and not always successful). We cannot take code that depends on IPC, of course, but layout, content and media should work, and some careful graphics updates. However, we should not mess with the UI except if it's buggy, because this may require saddling our slower users with more baggage than their computer can handle and the current browser at least is a known quantity. We will also wait on JavaScript updates in case they break the nanojit, and IonMonkey is in the future, so we should see what turns out with that.
And, since we aren't moving to TenFourFox 5, updates will still function properly in the way they are, and the plugin code won't change. So you get to keep plugins after all. I hope you're happy. ;) My arguments over security still stand, however, and I still strongly recommend something like Flashblock.
This is the new feature parity roadmap; my intention is to have a release every six to eight weeks, with four to six weeks of dev time and two weeks of "pre" testing:
4.0.2 still comes out probably this weekend. It looks very successful; no major problems were reported with the second beta, and the JavaScript speed and stability increase was universal.
4.0.3 is the next feature release. This will turn on the nanojit for the browser chrome and hopefully add load-store to the nanojit (these were features intended for TenFourFox 5 originally), along with configurable WebM filtering settings. It will also include security fixes and Firefox 5 bug fixes that we can safely backport. It may or may not include some new safe-looking features that do not significantly change the browser, such as the Firefox 5 changes and enhancements to HTTP and XMLHttpRequest. I might defer these to a '4.0.4'.
4.1 is where the magic will start. Among the features I think we need to have include Firefox 5's improvements to canvas, CSS animation, WebSockets (when they finish screwing around with the IETF spec, likely by Firefox 6) and SSL False Start. These are all features likely to be leveraged by future sites. Similarly, we should try to adopt some of their SSE/NEON vectorized improvements, especially for text scanning, along with libjpeg-turbo. These features are developing and we need to constantly watch for new bugs, so we should keep this set of new supported web features small and essential. If 4.1 is delayed, there will be a 4.0.4.
Because 4.1 changes the browser core in significant ways, we should not report ourselves as Firefox 4 anymore. We might adopt a built-in user agent changer like Classilla. More on that later.
So, the road moves on, just in a different direction. I'm going to start writing up new roadmaps and worklists in the meantime while 4.0.2 builds on the G5. And for the hacker builders, you've got eight weeks or so of Firefox 5 to see if you can get it running and succeed where I failed. In the meantime, TenFourFox remains "four-ever, yours."
Wednesday, May 18, 2011
4.0.2pre beta 2, crushing benchmarks, changing lives, etc.
I bet chtrusch is pretty proud of himself for ruining beta 1 with his bug report, and I hope he's happy. Actually, I know he already is, and you'll be happy too, because with the test case he reported we have finally stamped out issue 37 which was a long-running JavaScript crash bug we had wallpapered over for 4.0 and 4.0.1.
And, because that wallpapering was done by blacklisting certain traces in the JavaScript accelerator, we no longer need the wallpaper since the underlying problem (running out of stack space) has now been fixed by giving the app more stack. SunSpider was a mixed bag; the G5 barely moved, but the 7400 had an unpredictable improvement of around 5-15%. However, in V8, the quad G5 in Highest went from a score of 656 to a reliable 822 (higher is better), and Dromaeo -- which includes V8 in its composite score -- also duly increased, from 93 runs/sec to 110 runs/sec!
While you're testing things, please also look at the good work Narcotix and chtrusch are doing with localizations in issue 42 and issue 61, including a locale installer so that you don't have to manually sling files around.
Before we get to the goodies, some business items. I'm trying to work in with Mozilla about security content and that means being a good neighbour about currently extant security issues. Mozilla still does not plan to issue a formal Firefox 4.0.2, but they are amenable to putting fixes into the mozilla-2.0 tree for downstream consumers such as ourselves. They do not, however, want such fixes distributed outside of the Mozilla repos until Firefox 5 is released, so we probably cannot release the maintenance changeset pack until then as this contains the actual fixes involved. Sorry, builders. The current release date for Fx5 is still June 21st, although it may slip due to the rapid release process not yet having been tested for full rollout. However, I am asking explicitly to make sure we can still get binaries out to your good selves on time June 1st.
There are a couple other potential security issues I am reviewing for inclusion, though I may defer these to 4.0.3 pending Firefox 5.
Speaking of, Mozilla is no longer planning to issue numbered release branches with Fx5 - i. o. w., there is going to be only one mozilla-release repo and it will only contain the current version; there will be no archival branches like they used to have in the CVS days and in hg until Mozilla 2.0. This is a problem for our builders because when the next version comes down the pike, there will be nothing to pull from and apply changesets to. What we might have to do is clone and hg export a tarball so that the builders still have a stable "source source" while we retain source parity. Hopefully, however, Mozilla will keep stashing their own tarballs somewhere and we can build on top of that, perhaps. I'm still not sure what to do with this yet and we'll worry about it if we get Fx5 running.
In the meantime, change your life today with beta 2 (and crush benchmarks, etc.); read the release notes first:
And, because that wallpapering was done by blacklisting certain traces in the JavaScript accelerator, we no longer need the wallpaper since the underlying problem (running out of stack space) has now been fixed by giving the app more stack. SunSpider was a mixed bag; the G5 barely moved, but the 7400 had an unpredictable improvement of around 5-15%. However, in V8, the quad G5 in Highest went from a score of 656 to a reliable 822 (higher is better), and Dromaeo -- which includes V8 in its composite score -- also duly increased, from 93 runs/sec to 110 runs/sec!
While you're testing things, please also look at the good work Narcotix and chtrusch are doing with localizations in issue 42 and issue 61, including a locale installer so that you don't have to manually sling files around.
Before we get to the goodies, some business items. I'm trying to work in with Mozilla about security content and that means being a good neighbour about currently extant security issues. Mozilla still does not plan to issue a formal Firefox 4.0.2, but they are amenable to putting fixes into the mozilla-2.0 tree for downstream consumers such as ourselves. They do not, however, want such fixes distributed outside of the Mozilla repos until Firefox 5 is released, so we probably cannot release the maintenance changeset pack until then as this contains the actual fixes involved. Sorry, builders. The current release date for Fx5 is still June 21st, although it may slip due to the rapid release process not yet having been tested for full rollout. However, I am asking explicitly to make sure we can still get binaries out to your good selves on time June 1st.
There are a couple other potential security issues I am reviewing for inclusion, though I may defer these to 4.0.3 pending Firefox 5.
Speaking of, Mozilla is no longer planning to issue numbered release branches with Fx5 - i. o. w., there is going to be only one mozilla-release repo and it will only contain the current version; there will be no archival branches like they used to have in the CVS days and in hg until Mozilla 2.0. This is a problem for our builders because when the next version comes down the pike, there will be nothing to pull from and apply changesets to. What we might have to do is clone and hg export a tarball so that the builders still have a stable "source source" while we retain source parity. Hopefully, however, Mozilla will keep stashing their own tarballs somewhere and we can build on top of that, perhaps. I'm still not sure what to do with this yet and we'll worry about it if we get Fx5 running.
In the meantime, change your life today with beta 2 (and crush benchmarks, etc.); read the release notes first:
Sunday, May 15, 2011
4.0.2pre now available
4.0.2pre "beta" is now available for testing. The only TenFourFox specific change is issue 58, suggested by chtrusch for those systems that don't react well to plugins forcing smooth scrolling; you can now set tenfourfox.layout.smooth_scroll_plugins to false if you don't want that to happen. Speaking of plugin prefs, there are now separate prefs for full-page and embedded plugins, and actually now would be a good time to open about:config and type tenfourfox in the filter box to see all the custom prefs that are available presently. There will be more in the future.
The compiler changes we were hoping to take advantage of unfortunately made the browser dreadfully unstable -- it couldn't even open a window. We will not mess around with the compiler again until we must use gcc 4.2.
As mentioned previously, there will be no official Firefox 4.0.2 -- TenFourFox 4.0.2 is the same as Firefox 4.0.1, with security patches I have reviewed and applied while we wait for Firefox 5 to enter the beta channel and I can investigate if I can port it. This is part of our commitment to maintain a secure browser regardless of what actual version of Mozilla/Gecko we are using. There should be no compatibility issues. You can see the security issues applied (please note that some will require security privileges to view) by reading the rough draft of the release notes. On final release, these will be in the maintenance changesets, minus test cases so that we don't leak proof-of-concepts for any bugs that are still sec-locked. The patches in question were already approved for Firefox 5 and we are simply backporting them to Firefox 4.
Firefox 5 goes to beta channel tomorrow and I plan to pull source and do some preliminary work on it in between moving into my new house. For now,please try 4.0.2pre. UPDATE 5/17: See comments; a crash bug has been isolated and repaired. Stand by for a rebuild tonight or tomorrow. If there are no issues, it will be converted into final release on June 1st.
The compiler changes we were hoping to take advantage of unfortunately made the browser dreadfully unstable -- it couldn't even open a window. We will not mess around with the compiler again until we must use gcc 4.2.
As mentioned previously, there will be no official Firefox 4.0.2 -- TenFourFox 4.0.2 is the same as Firefox 4.0.1, with security patches I have reviewed and applied while we wait for Firefox 5 to enter the beta channel and I can investigate if I can port it. This is part of our commitment to maintain a secure browser regardless of what actual version of Mozilla/Gecko we are using. There should be no compatibility issues. You can see the security issues applied (please note that some will require security privileges to view) by reading the rough draft of the release notes. On final release, these will be in the maintenance changesets, minus test cases so that we don't leak proof-of-concepts for any bugs that are still sec-locked. The patches in question were already approved for Firefox 5 and we are simply backporting them to Firefox 4.
Firefox 5 goes to beta channel tomorrow and I plan to pull source and do some preliminary work on it in between moving into my new house. For now,
G3- G4/
7400 - G4/
7450 G5
Wednesday, May 11, 2011
Watching the floor drop out under our feet
Well, this was unexpected. Multiple places across Bugzilla with 4.0-ready patches are now being denied to land on mozilla-2.0, and the reason appears to be that there will not be a Firefox 4.0.2. Instead, Mozilla is dropping Firefox 4 after 4.0.1 and going straight to Firefox 5. In fact, Firefox 3.6 will be supported longer than 4.0 will!
(I should clarify that while a dummy 4.0.2 release is being [lightly] considered as a migration strategy for the end of Firefox 3.5, it would have no new content and would explicitly be just a version bump. We aren't going to do that.)
That was the bad news, now for the good news. My request to receive Mozilla security group privileges was accepted, and I am now able to view security issues as they are submitted and as they develop. Using this new access, I've been able to identify some security and stability fixes that apply to 4.0.1 (and some to 3.6, but there will be a 3.6.18 for those of you who are still unsure about jumping to TenFourFox) and that can be accepted by us safely, to keep us at security parity while I ponder what to do with Firefox 5 as it enters the beta channel.
Because this essentially forces us into unsupported mode early, we're going to adopt a new strategy to reduce branch-to-branch-jump work and yet crosspatch future patches to our codebase in a reasonably straightforward manner. Once we lose source parity, this will change, but as long as we maintain source parity and are built from a Mozilla release we will maintain this new strategy (builders take note):
- Each branch will have a base set of patches specific to that branch. In the case of Firefox 4.0.1, it is the changeset that is already available and that you can download now. These sets are the "base" and are the unique TenFourFox portions the browser is built upon. They should be unique to that branch and should not carry future work unless absolutely necessary, i.e., no changesets that would be part of a future Firefox branch. The reason for this is that the base will be carried forward to the next release of Firefox and modified as needed, so we don't want changesets that would already be in that release (most importantly stability and security fixes we backported from the future branch). The base may be added to if we find TenFourFox-specific bugs that we need to repair in point releases.
- On top of that will come specific future changesets that we are backporting for stability or security. There are ten of these I am reviewing for probable inclusion in TenFourFox 4.0.2, and about six others of low severity that we may or may not take. None of these are highly critical, but we should fix them. These changests are already in Firefox 5; we do not want to carry them forward in the base. These changesets will be maintained separately and will not be reapplied to "mozilla-5.0" or whatever they're going to call it because they don't need to be. The version and milestone should be maintained here too because we definitely don't want to have to merge that in base.
So, for 4.0.2, new base changes will bethe compiler string modifications and UPDATE: the compiler changes make the app really unstable, so we're not shipping them for 4.0.2 the pref for enabling or disabling smooth scroll with plugins. We will then establish a maintenance changeset with the new version and milestone numbers and the security/stability patches, which I'll document in the release notes. Builders need to apply both the base and the maintenance changesets to generate a complete version. When Firefox 5 comes out of beta, then we'll immediately clone that repo and apply the Firefox 4 base to it, making any additional changes that we will need and thus creating a new base for that release branch (assuming it works, of course -- see the previous entry for why this is probable but not guaranteed). Assuming a TenFourFox 5, while we wait for Firefox 6, we will watch for security updates and backport to 5.0.1 as necessary.
I have not yet decided if there will be a 4.0.3. If there are no major issues by the time TenFourFox 5 dawns and it seems to work, then we will simply release. If there are some issues identified of sufficient severity that do still affect us, I will issue a 4.0.3 for those who can't give up plugins just yet so that it can buy them some time.
Like every version of TenFourFox, you will never be upgraded automatically to a new version unless you download and install it yourself. I don't like this personally and I don't like forcing it on people, and we don't have the infrastructure for it anyway. If we liked forced upgrades, we wouldn't be using Power Macs. ;)
The current plan is to select and land patches this week and chug out builds over the weekend for you, the loyal beta audience. 4.0.2pre should be available on Monday, and if there are no major issues 4.0.2 final will come out on June 1st. As always, comments and questions solicited.
(I should clarify that while a dummy 4.0.2 release is being [lightly] considered as a migration strategy for the end of Firefox 3.5, it would have no new content and would explicitly be just a version bump. We aren't going to do that.)
That was the bad news, now for the good news. My request to receive Mozilla security group privileges was accepted, and I am now able to view security issues as they are submitted and as they develop. Using this new access, I've been able to identify some security and stability fixes that apply to 4.0.1 (and some to 3.6, but there will be a 3.6.18 for those of you who are still unsure about jumping to TenFourFox) and that can be accepted by us safely, to keep us at security parity while I ponder what to do with Firefox 5 as it enters the beta channel.
Because this essentially forces us into unsupported mode early, we're going to adopt a new strategy to reduce branch-to-branch-jump work and yet crosspatch future patches to our codebase in a reasonably straightforward manner. Once we lose source parity, this will change, but as long as we maintain source parity and are built from a Mozilla release we will maintain this new strategy (builders take note):
- Each branch will have a base set of patches specific to that branch. In the case of Firefox 4.0.1, it is the changeset that is already available and that you can download now. These sets are the "base" and are the unique TenFourFox portions the browser is built upon. They should be unique to that branch and should not carry future work unless absolutely necessary, i.e., no changesets that would be part of a future Firefox branch. The reason for this is that the base will be carried forward to the next release of Firefox and modified as needed, so we don't want changesets that would already be in that release (most importantly stability and security fixes we backported from the future branch). The base may be added to if we find TenFourFox-specific bugs that we need to repair in point releases.
- On top of that will come specific future changesets that we are backporting for stability or security. There are ten of these I am reviewing for probable inclusion in TenFourFox 4.0.2, and about six others of low severity that we may or may not take. None of these are highly critical, but we should fix them. These changests are already in Firefox 5; we do not want to carry them forward in the base. These changesets will be maintained separately and will not be reapplied to "mozilla-5.0" or whatever they're going to call it because they don't need to be. The version and milestone should be maintained here too because we definitely don't want to have to merge that in base.
So, for 4.0.2, new base changes will be
I have not yet decided if there will be a 4.0.3. If there are no major issues by the time TenFourFox 5 dawns and it seems to work, then we will simply release. If there are some issues identified of sufficient severity that do still affect us, I will issue a 4.0.3 for those who can't give up plugins just yet so that it can buy them some time.
Like every version of TenFourFox, you will never be upgraded automatically to a new version unless you download and install it yourself. I don't like this personally and I don't like forcing it on people, and we don't have the infrastructure for it anyway. If we liked forced upgrades, we wouldn't be using Power Macs. ;)
The current plan is to select and land patches this week and chug out builds over the weekend for you, the loyal beta audience. 4.0.2pre should be available on Monday, and if there are no major issues 4.0.2 final will come out on June 1st. As always, comments and questions solicited.
Monday, April 25, 2011
Towards TenFourFox 5 and 4.0.2pre
I'm putting this post out a little early to talk about what we're going to do with Firefox 5 because my hand's been forced a little bit; I've been seeing some well-intentioned blog posters saying, among other things, that TenFourFox is nice and holds the fort but isn't going to get any updates. Besides the fact we just had one, this is unequivocally, unbelievably, unimaginably false. I personally use TenFourFox; I wouldn't shoot myself in the foot by failing even to maintain security updates, let alone additional features. When (not if) we reach a state where we cannot hack "Firefox.next" to build on 10.4 PPC, then we simply carry on maintenance on the branch we know is still working. If nothing else, the browser will remain safe to use even if the technology reaches a state of arrested development. Don't forget that Camino currently uses a three-year-old renderer, but they maintain and land security fixes from later branches to keep the browser safe to use. We'll be doing the exact same thing. I'll discuss that in more detail in a moment.
On that note, let's talk about Firefox 5. Firefox 5 really should be called 4.1 because it is, indeed, the minor feature update to Firefox 4 (as opposed to 4.0.1, which is the minor maintenance update). That said, it does have a number of important features we will want. Very few of these are user-facing, mind you. I've been playing with Aurora (essentially the "stable alpha") on my lonely Core 2 Duo mini, and at a UI level you're not going to see much difference. Like Snow Leopard to Mozilla's "Leopard," Firefox 5 really improves very little of the interface relative to Fx4.
Of the features that are there, however, the most important one is CSS Animation. Remember, Flash or any other plugin will not be enabled in "TenFourFox 5," and in this HTML5 world, we want to encourage animation and graphics methods that do not rely on Flash. CSS Animation is part of that transition away from proprietary, buggy and (in our case) unmaintained ways of doing graphics and animation in the browser. There are also quite a few performance improvements in Firefox 5 that will surely benefit us, because sadly our machines aren't going to get any faster by themselves.
For TenFourFox 5, in addition to getting all that working, I would also like to expand some of our marquee features to add even better performance. For example, the JavaScript nanojit is now stable and has achieved enough cross-processor parity that we can flip it on for the browser as well. In fact, I've been running my 4.0.1 in that manner for over a month (go into about:config, set javascript.options.tracejit.chrome to true), which has been nice and stable, and gives a noticible speed boost at the cost of additional memory needed for the traces. This is kind of a big change to throw at 4.x, but we definitely want it in 5.x. I'm also hoping to finally fix once and for all the problem with our generated prologue and certain kinds of native code calls, which are handled currently with a safe but slower abort to the interpreter. If these calls can be embedded in the trace, we can get even faster, but this has an obvious stability impact and I want extensive beta coverage. Finally, I want to support JavaScript typed arrays in the nanojit; these currently fall back on the interpreter, and the apps that are likely to attempt using typed arrays for increased performance are ironically going to perform worse on TenFourFox because of the fallback. This work has already been done preliminarily. I think this work list is fully achievable, especially because a lot of it is already done or in progress.
In the would-be-nice department, I want to continue our progress towards full AltiVec acceleration of the content chain. We've started (I think very successfully) by enabling AltiVec in libpixman and libvpx. There are some trivial additional code conversions that I can do in libvpx, and I'd like to try implementing them. In addition, Fx5 adds libjpeg-turbo, which is already faster in and of itself, but also gives us the chance to convert the MMX/SSE code it includes into AltiVec/VMX. This should greatly speed up our image display and processing. Finally, I want to track down other places in the code where Mozilla has already implemented SSE or SIMD equivalents for C code and write VMX versions. This work is unlikely to be fully completed for TenFourFox 5, but some parts should make it.
As I alluded to previously, Firefox 5 is not a slam dunk: we need to get Chromium IPC working on Tiger. I know it's possible to get it to compile, but linking was historically an issue because with IPC everything is clotted into a huge superlibrary called libxul (and in debug mode, this was too large for Tiger's linker to pull together), so it has never been tested. Fx4 does not require it (just strongly encourages it), but Fx5 demands it. Again, I remain cautiously confident the port is possible, but even if it is not, let me reiterate again that even if we remain stuck on Mozilla 2.0 forever, we will still get security updates landed on TenFourFox and the browser will remain functional and safe. You have my word on it: I eat my own dogfood. I hope that Classilla's longevity and update history demonstrates to people that TenFourFox is here for the long haul too.
That aside, Fx5 sounds pretty good, right? Well, don't get too raring to go just yet, because we have to consider the ramifications of the migration. Since our last discussion on this topic, more information has become available, and most of that information from our perspective is bad news. Fx4 will be the last of the "big" releases and the last to retain the old release EOL timeframe, which is usually six months of updates after release of the next version (and you can bet that under the new rapid release framework it won't be a second longer). Fx5 and later will be unsupported the second the next version comes out.
What does that mean for support? Well, barring something extraordinary, as soon as Fx6 emerges there will be no more Fx5 updates. Period. Similarly, when Fx7 comes out (!), no more Fx6. This means the safety net that would ordinarily be under us with a stable branch like Fx4 will no longer exist once we jump to Fx5. If there is a serious bug discovered in Fx5 and Fx6 has already come out, then there will be no fix for it in Fx5; Mozilla wants you to simply update. This is not ill-conceived on their part, but it sucks for people like us trying to actually build around the updates because the ground keeps shifting under our feet, and may require us to do a significant amount of backporting. I'm also not sure what this will mean for extension compatibility. Mozilla has plans in place for carrying forward addons semi-automatically, but if this process cuts off backward compatibility for old versions at the same time (as, possibly, a carrot for updating), that could really bite us.
To balance this and to spare much unnecessary work, this is how the rollout will happen, assuming the port of Fx5 is successful.
- Work on TenFourFox 5 will not commence until Firefox 5 is released. Other than chemspill releases, we know no more work will occur on the final release branch once it becomes final because the next 6-week release will clobber it. Therefore, we know nothing is going to change for Mozilla 5, and this simplifies our porting.
- Until work begins, there will be 4.x updates, and at least one more 4.x release (4.0.2pre; we'll get to that) will have feature work.
- Once work begins, we will have our own separate formal betas, just as before. I envision no more than three, most likely. 4.x updates will continue simultaneously as pure security and stability updates only.
- Once we are final, we will publish a limited set of 5.x security and stability releases while we look at Firefox 6, which should be out by then or soon. I am working on getting security access at Mozilla so that I can review and backport security fixes as they happen, though in the short term we can still get this information from publicly-available Mercurial commits, and they will land on 5.x too.
The cycle will then repeat as long as we are able to compile Firefox.next, whatever it happens to be. If we jumped to Firefox 6, then there will be 6.x betas (with 5.x releases), followed by 6.x releases while we look at Fx7, and so forth. In essence we will be emulating the old Mozilla release mechanism, except that we will be actually maintaining our own old branches instead of relying on Mozilla to do it for us.
At the point where we can no longer move forward, we set up our own Mercurial repo and maintain the last good branch "forever," backporting all security fixes that apply, any bug fixes that apply, and (very carefully) new features that we might need -- limited largely to browser core, layout, content, DOM and JS. JavaScript is pretty straightforward to keep current, fortunately, because it is largely self-contained. The rest will be examined on a case-by-case basis. This is what I've referred to as security parity and feature parity in the Wiki.
I really hope this puts people's concerns to rest; TenFourFox is here to stay. I need a browser that works on my G5. You can rely on this because I have my own skin in the game, as do, presumably, others who have contributed and who will contribute in the future.
Now, about 4.0.2pre. This is a feature release, but with only one feature, which is to try new compiler settings. Since this can lead to very subtle bugs, I really want to keep everything else constant to avoid nasty little variables I can't control for. Part of this will be to enable -ftree-vectorize for G4/G5 and attempt a proper 32-bit compile for G5 without using the hackish hybrid compile string we use now (64-bit Firefox on G5 still has issues and we don't really benefit from 64-bit anyway in the same way that x86 does). This won't affect the G3 releases, obviously. Similarly, it will include whatever other fixes for Firefox 4.0.2 that Mozilla has landed. By the way, they are holding their official release of Firefox 4.0.1 until later this week due to a last-minute issue, but this is apparently an oddiment of their update infrastructure and does not concern us. 4.0.2pre will be available when Mozilla releases it to the beta channel and I'll give you a heads up when it is about to arrive.
Well, that's enough blather; if this does not put your questions to rest, ask them in the comments.
On that note, let's talk about Firefox 5. Firefox 5 really should be called 4.1 because it is, indeed, the minor feature update to Firefox 4 (as opposed to 4.0.1, which is the minor maintenance update). That said, it does have a number of important features we will want. Very few of these are user-facing, mind you. I've been playing with Aurora (essentially the "stable alpha") on my lonely Core 2 Duo mini, and at a UI level you're not going to see much difference. Like Snow Leopard to Mozilla's "Leopard," Firefox 5 really improves very little of the interface relative to Fx4.
Of the features that are there, however, the most important one is CSS Animation. Remember, Flash or any other plugin will not be enabled in "TenFourFox 5," and in this HTML5 world, we want to encourage animation and graphics methods that do not rely on Flash. CSS Animation is part of that transition away from proprietary, buggy and (in our case) unmaintained ways of doing graphics and animation in the browser. There are also quite a few performance improvements in Firefox 5 that will surely benefit us, because sadly our machines aren't going to get any faster by themselves.
For TenFourFox 5, in addition to getting all that working, I would also like to expand some of our marquee features to add even better performance. For example, the JavaScript nanojit is now stable and has achieved enough cross-processor parity that we can flip it on for the browser as well. In fact, I've been running my 4.0.1 in that manner for over a month (go into about:config, set javascript.options.tracejit.chrome to true), which has been nice and stable, and gives a noticible speed boost at the cost of additional memory needed for the traces. This is kind of a big change to throw at 4.x, but we definitely want it in 5.x. I'm also hoping to finally fix once and for all the problem with our generated prologue and certain kinds of native code calls, which are handled currently with a safe but slower abort to the interpreter. If these calls can be embedded in the trace, we can get even faster, but this has an obvious stability impact and I want extensive beta coverage. Finally, I want to support JavaScript typed arrays in the nanojit; these currently fall back on the interpreter, and the apps that are likely to attempt using typed arrays for increased performance are ironically going to perform worse on TenFourFox because of the fallback. This work has already been done preliminarily. I think this work list is fully achievable, especially because a lot of it is already done or in progress.
In the would-be-nice department, I want to continue our progress towards full AltiVec acceleration of the content chain. We've started (I think very successfully) by enabling AltiVec in libpixman and libvpx. There are some trivial additional code conversions that I can do in libvpx, and I'd like to try implementing them. In addition, Fx5 adds libjpeg-turbo, which is already faster in and of itself, but also gives us the chance to convert the MMX/SSE code it includes into AltiVec/VMX. This should greatly speed up our image display and processing. Finally, I want to track down other places in the code where Mozilla has already implemented SSE or SIMD equivalents for C code and write VMX versions. This work is unlikely to be fully completed for TenFourFox 5, but some parts should make it.
As I alluded to previously, Firefox 5 is not a slam dunk: we need to get Chromium IPC working on Tiger. I know it's possible to get it to compile, but linking was historically an issue because with IPC everything is clotted into a huge superlibrary called libxul (and in debug mode, this was too large for Tiger's linker to pull together), so it has never been tested. Fx4 does not require it (just strongly encourages it), but Fx5 demands it. Again, I remain cautiously confident the port is possible, but even if it is not, let me reiterate again that even if we remain stuck on Mozilla 2.0 forever, we will still get security updates landed on TenFourFox and the browser will remain functional and safe. You have my word on it: I eat my own dogfood. I hope that Classilla's longevity and update history demonstrates to people that TenFourFox is here for the long haul too.
That aside, Fx5 sounds pretty good, right? Well, don't get too raring to go just yet, because we have to consider the ramifications of the migration. Since our last discussion on this topic, more information has become available, and most of that information from our perspective is bad news. Fx4 will be the last of the "big" releases and the last to retain the old release EOL timeframe, which is usually six months of updates after release of the next version (and you can bet that under the new rapid release framework it won't be a second longer). Fx5 and later will be unsupported the second the next version comes out.
What does that mean for support? Well, barring something extraordinary, as soon as Fx6 emerges there will be no more Fx5 updates. Period. Similarly, when Fx7 comes out (!), no more Fx6. This means the safety net that would ordinarily be under us with a stable branch like Fx4 will no longer exist once we jump to Fx5. If there is a serious bug discovered in Fx5 and Fx6 has already come out, then there will be no fix for it in Fx5; Mozilla wants you to simply update. This is not ill-conceived on their part, but it sucks for people like us trying to actually build around the updates because the ground keeps shifting under our feet, and may require us to do a significant amount of backporting. I'm also not sure what this will mean for extension compatibility. Mozilla has plans in place for carrying forward addons semi-automatically, but if this process cuts off backward compatibility for old versions at the same time (as, possibly, a carrot for updating), that could really bite us.
To balance this and to spare much unnecessary work, this is how the rollout will happen, assuming the port of Fx5 is successful.
- Work on TenFourFox 5 will not commence until Firefox 5 is released. Other than chemspill releases, we know no more work will occur on the final release branch once it becomes final because the next 6-week release will clobber it. Therefore, we know nothing is going to change for Mozilla 5, and this simplifies our porting.
- Until work begins, there will be 4.x updates, and at least one more 4.x release (4.0.2pre; we'll get to that) will have feature work.
- Once work begins, we will have our own separate formal betas, just as before. I envision no more than three, most likely. 4.x updates will continue simultaneously as pure security and stability updates only.
- Once we are final, we will publish a limited set of 5.x security and stability releases while we look at Firefox 6, which should be out by then or soon. I am working on getting security access at Mozilla so that I can review and backport security fixes as they happen, though in the short term we can still get this information from publicly-available Mercurial commits, and they will land on 5.x too.
The cycle will then repeat as long as we are able to compile Firefox.next, whatever it happens to be. If we jumped to Firefox 6, then there will be 6.x betas (with 5.x releases), followed by 6.x releases while we look at Fx7, and so forth. In essence we will be emulating the old Mozilla release mechanism, except that we will be actually maintaining our own old branches instead of relying on Mozilla to do it for us.
At the point where we can no longer move forward, we set up our own Mercurial repo and maintain the last good branch "forever," backporting all security fixes that apply, any bug fixes that apply, and (very carefully) new features that we might need -- limited largely to browser core, layout, content, DOM and JS. JavaScript is pretty straightforward to keep current, fortunately, because it is largely self-contained. The rest will be examined on a case-by-case basis. This is what I've referred to as security parity and feature parity in the Wiki.
I really hope this puts people's concerns to rest; TenFourFox is here to stay. I need a browser that works on my G5. You can rely on this because I have my own skin in the game, as do, presumably, others who have contributed and who will contribute in the future.
Now, about 4.0.2pre. This is a feature release, but with only one feature, which is to try new compiler settings. Since this can lead to very subtle bugs, I really want to keep everything else constant to avoid nasty little variables I can't control for. Part of this will be to enable -ftree-vectorize for G4/G5 and attempt a proper 32-bit compile for G5 without using the hackish hybrid compile string we use now (64-bit Firefox on G5 still has issues and we don't really benefit from 64-bit anyway in the same way that x86 does). This won't affect the G3 releases, obviously. Similarly, it will include whatever other fixes for Firefox 4.0.2 that Mozilla has landed. By the way, they are holding their official release of Firefox 4.0.1 until later this week due to a last-minute issue, but this is apparently an oddiment of their update infrastructure and does not concern us. 4.0.2pre will be available when Mozilla releases it to the beta channel and I'll give you a heads up when it is about to arrive.
Well, that's enough blather; if this does not put your questions to rest, ask them in the comments.
Subscribe to:
Posts (Atom)