State of the user base: Power Macs 7 years after the switch

Most of you know that your copy of TenFourFox checks in periodically with the servers here at Floodgap Orbiting Headquarters to look for new versions. Unlike the NSA, I only get your IP address, plus your CPU type, browser and OS version (all of which can be spoofed); no personal information is transmitted (so I cannot identify individual machines or users); and you can turn it off. But the aggregated information from version requests is also useful to look at the state of Power Mac users nearly seven years to the month after the last Apple model transitioned to Intel (August 2006, with the original Mac Pro replacing the Power Mac G5).

There's no way to know exactly how many Power Macs are still out there. A conservative estimate probably puts the worldwide count somewhere in the low hundred thousands for active use, maybe shy of a million in some sort of infrequent operation and innumerable more in closets and storage, but this is all just supposition. Some get only occasional use, I'm quite sure some don't use TenFourFox, and of those that do they may only run it periodically. Furthermore, some may dual boot Linux or *BSD, some might be in Mac OS 9 most of the time now, and some might not even be running Mac OS of any sort anymore. We just know that you're out there, somewhere.

The total TenFourFox userbase must be less than 18,000 machines or so, based on the average number of downloads per version, though the actual number is likely in that ballpark. Google Code has tracked our download numbers over time under the Downloads tab and the number has declined a bit from the days of TenFourFox 4, but is still a surprisingly large number of machines and relatively stable. This number, by the way, is totally organic and grown by word of mouth; we don't advertise anywhere, so it's really gratifying to see these kinds of stats for what is essentially a hobby project. Of these, a minority check in with the update server daily, approximately 4,000, more on weekdays and slightly fewer on weekends. This is an approximation, because I can't actually identify or track individual machines, but a stable one.

This population of machines that checks in daily is, serendipitously, the most interesting group to study. Since checkins only happen daily if the Mac is on and the browser is running, these are likely to be people using a Power Mac regularly for their normal tasks, so let's take a closer look at the data. What might a daily Power Mac user look like 7 years after the switch?

First, some methodology: only fully valid, "legible" checkins were used; a handful of checkins demonstrated activity like TenFourFox would, but had obscured or obviously bogus user agent strings, so they were dropped. Builds tagged as "Debugging" were also dropped if they checked in, since they are not release builds by definition. The numbers are a composite of several days averaged out, so take them as merely representative and not exact. I expect that machines may drop in and out from day to day; thus, the numbers observed may not necessarily be the same machines each day, even though the numbers are broadly similar. Okay, enough fine print. Here we go.

Users by operating system

First, here's the breakdown by operating system. The operating system is directly checked by TenFourFox and reported, unless it was overridden by a user-agent tool, so it is fairly reliable. No surprises that 10.5 is the majority, but 10.4 maintains a very strong percentage. (Note that since 10.3 and prior cannot run TenFourFox, we don't track it. There is probably a non-trivial number of machines in that category. Classilla does not have automatic checkins, so I can't say anything about OS 9 here.) Interestingly, a very small sliver of 10.6 users was detected, around two or three a day, despite the fact that Firefox (at least for the moment) still runs natively on Snow Leopard. On one notable outlier day, we picked up eight of them.

Users by CPU

Now the breakdown by CPU. While we could write code to detect the CPU at runtime, TenFourFox doesn't use it so that build automation is less complex; instead, this number comes from the build that the user chose to download, so these numbers are a bit less reliable (more below). The top two are no surprise: G4/7450, not only because of the Power Mac G4 but the large number of PowerBook G4s, and of course the G5. G4/7400 and G3 builds represent less than 10% of the daily checkins combined, though they equal around 20% of the total downloads, suggesting these machines are not usually daily drivers. A sliver of "386" (Intel) users still rock the presently stalled experimental Intel build.

Users by OS and CPU

Combining the two is where it starts getting interesting. I've harmonized the colours in iWork and exploded out the slices so that you can visually compare this image to the prior one. Most G5 users prefer 10.5 by over 2 to 1, and this is pretty much as expected given that the G5 seems to perform better in 10.5 than 10.4; it's the logical choice for G5 users if you don't need 10.4 for compatibility. (I do, though, so Tiger forever.) For G4/7450 users, however, 10.4 has a slight edge over 10.5, likely because lower specification machines don't do as well in Leopard.

Unfortunately, our numbers get hosed a bit in the minority slices because of builds running on an obviously different host architecture. The 7400 users on 10.5 are probably real, since the OpenFirmware hack to get the installer working is well-known, but most G4/7400 users are on 10.4 as expected (as are almost all the G3 owners). The 10.6 users must be using a PPC build of some sort under Rosetta, so there's not a whole lot more that can reliably be said about them. The so-called "G3 + 10.5" users, on the other hand, are bogus without a doubt; these might be Intel users, since we advertise the G3 build specifically for Intel under Rosetta in the FAQ. Most of the true Intel build users are on 10.4, which is bad news for issue 209.

Users by TenFourFox version

While Power Mac users may not be up with the latest and (if Apple marketing is to be believed) greatest new hardware, the TenFourFox userbase is pretty good at upgrading. Roughly three-quarters of the active users are on 17.0.x, the current stable major version, and over half of those are on 17.0.7, the most current release in that series. (The number plunges to around 500 for the immediately preceding version 17.0.6 and rapidly falls off from there.) Add the four percent of users on 22, and almost 80% of our user base is on a supported branch of some sort. From this, developers should confidently infer that most active Power Mac users would gladly update their software if you offered. You should offer. I'm just saying.

The biggest chunk of users off the wagon are still on 10.0.x, our previous stable major version, which has not been updated for almost nine months; more alarming is that 10.0.11, the last release in that branch, is not at all the majority. I have some theories about this, but they'll all be moot pretty soon. They represent around 10% of the user base.

I've also combined all of the pre-10 versions into a single slice rather than break them down into infinitesimal shards, since summed up they represent only a bit over five percent of users. These are all scattered handfuls with no one version having a clear predominance and the numbers are too small in general to make reliable hypotheses. However, there is a slightly significant correlation between our "10.6" users and versions prior to 8.0, which was when we started the transition to methodjit, implying that these folks are still relying on good old tracejit (on the other hand, those that aren't are uniformly on 17, as they should be). At least a couple users are still using a pre-4.0 beta (!). Y'all upgrade soon now, y'hear?

A similarly small fraction of our user base is using unstable branch builds (i.e., versions 11-15 and 19-22 inclusive), which is understandable, but distressing for maintaining a solid and statistically relevant beta test population. Some people are still using unstable builds as old as 11.0 (shame on you). Fortunately, most of you are current. The 22 figure includes those brave souls who dared to try the 22.1 experiment that failed, sigh.

***

The best news, though, is looking at the user base figures over time. Seven years after the switch, the Power Mac may be a strict minority in the Apple ecosystem, but we're still alive and well despite their best efforts to stick the shiv in. While we suffer some attrition from machines breaking down and people moving on, the user base at least from this surveyor's eye may be small but still remains constant. Heck, you can still buy used ones easily and inexpensively from resellers like PowerMax, meaning it's a great time to pick up spare gear and keep your machine humming. I think I'll probably make this an annual feature of this blog from now on just to see how we're doing in our little RISC foxhole. Here's to seven more years of getting screwed by Cupertino.

The jemalloc experiment, once again, runs aground

There's an unacceptable memory leak going on in the jemalloc build which balloons total memory usage on my G5 by about 500MB overnight while idle, and in the morning garbage collection cripples trying to browse with it. The baseline browser has a mild bump overnight, so there may be other leaks going on, but it does not suffer the same issue and browses normally. I don't know what's going on here, but this is a showstopper for jemalloc and neither 22.1 or 24 will have it enabled until this is better understood. It's a shame since it helped on some known troublesome sites, but it's once again a bridge too far and while you're welcome to continue using it the patches are going back in mothballs until I have more time to understand the phenomenon.

While I was trying to analyze the problem, I did notice that we wait a long time to do garbage collection. Mozilla themselves thought this was a problem on slow systems and FirefoxOS in fact garbage collects at a threshold an order of magnitude less than regular Firefox. Since we're closer in spec to FirefoxOS devices, now we do too. It does seem to cut back on some of the random pauses and stutters after the browser has run for awhile, and it will be in 22.1 final even though it did not help with jemalloc. That said, the experience was valuable in that it looks like issue 231 needs to be enabled for 10.5 as well.

BaselineCompiler/Firefox 24 update: most branches and calls seem to be fixed and the generated code now gets far enough to try to call into the JavaScript virtual machine. Near as I can tell from the generated output, unlike methodjit (JaegerMonkey) in 10 through 22, BC emits calls into the JS VM for individual JSOps instead of generating code for those ops (which is now what Ion does). This is slower to run but faster to compile, not unlike some old 8-bit BASIC compilers that instead of generating actual machine code just pre-interpreted the BASIC into a string of calls into the interpreter, removing the parsing overhead. Getting those calls working should speed completion significantly. It does do a little more in that arithmetic options are directly compiled, though, so we'll deal with those later. Right now I just want to compile this: js --baseline-eager --no-ti --no-ion -e 'var i=0' ... and we'll grow from there.

Incidentally, this required me doing bug fixes to Apple gdb to handle frames better. I'll make our hacked gdb-768 available a little later (it's the Xcode 3.0 debugger built for 10.4 with additional patches).

22.1: the backup strategy (hello, jemalloc)

First, a word on 24. BaselineCompiler is now to the point where it can generate code without asserting for simple scripts, which is no mean feat. The trampoline is verifiably wrong in at least two places where I can't understand what I was thinking when I wrote it, and there are several bugs in the code generator for branches that I'm working out, but we are at stage 2 (linking and building) and getting very close to stage 3 (compiling and running simple scripts). At that point, the fourth and final state is to get it to pass the test suite, and then building the rest of the browser. That is still at least four to six weeks of work. Don't expect a beta anytime soon.

For some time on this blog I've been mentioning my attempts in issue 218 to get jemalloc, the higher-performance memory allocator "normal" Firefox uses, working on 10.4. Right now, we use the OS X system allocator, which when a lot of things are trying to compete for it, is like a whole bunch of people trying to suck a milkshake through a single straw: it's messy, it involves other people's saliva and it's slow. At the lowest level there's a lot of memory allocation and deallocation going on as bits of data move from one browser subdomain to another, so things that speed that process should speed the whole browser. And, mostly (see below), it does. But changing the way that a single application handles memory management is fraught with peril when the rest of the system is using a different one, as we found out in TenFourFox 21: it was noticeably faster, but it had several unconscionable showstoppers like, uh, cut and paste didn't work. That was a bummer. We can't ship a browser like that.

Because of the delays with 24, however, we need a backup strategy in case 17 runs off support while I'm trying to finish 24. Since 22 is closer to 24, it would be much easier to backport security updates to that than 17, and much less risky, and heaven forbid 24 doesn't work we'll need to use 22 to drop source parity. (23 is not an option; there's no methodjit in it, and BaselineCompiler is too new.)

Whatever bug was in 21 that prevented cut and paste and certain other widget operations from working was fixed somehow in 22 with the widget changes afoot for Australis, and now the browser basically works. But I mentioned there's a problem with one part of the system using one way to manage memory and us using another: when the two meet, they may not be compatible. Fortunately because Mozilla reinvents the wheel with XUL and handles much of its own interface and does a lot of its work independent of the operating system, this occurs in very few places, but the one that's there is a doozy: anywhere that we hand a pointer to a block of memory to the operating system that jemalloc allocated, but that the operating system is expected to ask jemalloc to free when it's no longer in use, we trip a serious OS X bug that hands the wrong pointer back to jemalloc. Mozilla analyzed this in bug 702250, where dragging and dropping large images brought the entire browser down. The operating system bug in question seems only partially fixed in 10.5, and as a result the level of crashes Mozilla observed caused them to turn off jemalloc entirely for 10.5 users in Firefox 10. We don't have that luxury if we want the level of performance we're going to need, so we either have to fix the problem or wallpaper it.

Testing "22.1" (as I dub it) showed that it was really easy, mercifully in this case, to replicate the problem in 10.4. Bug 702250 not only affects 10.4, it's markedly worse: any image, not just big ones, that is dragged to the desktop or to other applications will crash the browser. (Interestingly, dragging and dropping URLs and text is fine.) However, because the Clipboard is already allocated when the browser starts, Firefox is able to transmit data to it without issue, so you can copy an image and then paste it into another application, or right click and save it to disk. Since those work, the simplest solution here was to disable image dragging since fully functional workarounds are easily available.

The second part is a little trickier. Certain components ask the operating system for icon images, such as the Applications pane in Preferences. When those icons are no longer needed, the OS frees them, and the browser dies, to such a point where in the original build of 22.1 simply opening that pane would crash the browser. It also manifests in the Downloads window -- as icons are scrolled off screen, the OS will poisonously free those too, and bang. In this test version, icons are simply turned off. However, I don't think that's very nice or polished, but we can reinvent the wheel ourselves: create a new icon service that uses internal copies of icon images rather than asking the OS for them. It won't cover every file type, but it will work, and because the image lifetime is internal to TenFourFox the browser will not crash. (Other icons like favicons and browser chrome are similarly unaffected.)

jemalloc is not a panacea. It specifically speeds up operations where a lot of little bitty allocations need to be done repeatedly, such as very complex layouts or pages with a lot of widgets. However, it has a strange interaction I have not yet figured out with CoreGraphics: painting is slower, and painting big areas (particularly areas where the browser has to transform in software) is really slow. For most sites, the effect of the former greatly outweighs the effect of the latter, thus a net benefit, and on some pages very much so, but on those few that have relatively simplified layout but require lots of graphics painting jemalloc is slower. The best example I can find is my rat bastard electric company, Southern California Edison, where their new site design drags every time that damn entire window-full of smiling happy people who love to pay their confiscatory electricity rates advances to the next slide. It was bad on the old version, but it's worse now. It's also sluggish on my 10.6 Core 2 Duo Mac mini running real Firefox, by the way, but it's a much faster CPU so it does not lag near as much (I suspect hardware acceleration also improves it). For reasons I can't explain, the effect also seems to be worse on the G5; my iMac G4 and iBook G4 really light on fire with this, but the quad has some weird slowdowns.

A weird thing I noticed, but have not been able to resolve or consistently reproduce, is a questionable problem with large uploads (downloads are fine). Ironically, it occurred while trying to upload this test release, but it does not always happen and a debug build showed no problems. Even when it does happen, it works fine on the second try, and doesn't do it on all sites or even all uploads. I really need a site where you are unable to upload at all to deal with this, so I hesitate to call it a "bug" until I have good STRs on my part. Please don't report intermittents because I have enough of those already.

Also, while jemalloc may improve repeated allocations in a straight line the problems with multithreading and multiprocessing persist even with jemalloc enabled, so it does not solve issue 231. That brings us to our test builds -- you'll note there are only two, one for G5 and one for G4/7450. That's because this test version has been pretty banged on with 10.4, but not at all with 10.5. I particularly need multi-CPU testers, even better if you can demonstrate the performance is significantly better with Tiger than Leopard. If that's the case, we need to extend issue 231 to disable multiple CPUs on Tiger and Leopard, not just Tiger. Don't worry, G3 and G4/7400 owners: you are still loved and you will get the next version too (G4/7400 users can try the 7450 one, but it is not optimized for your platform).

Note thee well that I haven't decided that jemalloc is the way we're going and this is not releaseable in this state to a general audience although it is to you, you crazy wild unstable branch users, you. I need to know about other crashes you can find and reliable differences between it and the original 22.0 -- other than jemalloc, issue 231 and the fix for leaking font refs, this is the same browser. Reproducible, confirmable, consistent differences will be investigated thoroughly; intermittent, unconfirmable ones won't. With some experimentation, we'll see if we want to move the stable users to this if I have to put them on 22 for a little bit, or if this exercise is even worth it at all. I think it is, but let's see what you think.

Again, G5 and G4/7450 only, with 10.5 testers as a priority -- G3 and G4/7400 will come with the next scheduled update. 10.4 users are of course welcome, but I really want to know how it performs on 10.5 and a comparison of 10.4 to 10.5 performance if anyone can do that.

Back to debugging the trampoline.

Back to Ion

The idea was to do work on Ion from my hotel room, but I left my iBook G4's power adaptor at the office and so I had to make three hours or so of battery life last for three days. But, I got to try a Wagyu ribeye steak. So there were compensations. And the Fuze+ performed sterling service aboard the airplane.

After I got back from America's Team's City, I sat down and for the past several days banged out code and managed to drag JavaScript back to fully compiling and linking with IonMonkey. Some very, very preliminary testing indicates it can emit code, but I'm having issues with how Ion labels are managed and bridging them to the machinery in our custom assembler, which we still use for code generation because of Ben's good work tuning it. I'm trying a couple of different ideas and I might just have to hack Ion to make some of this workable.

And now for something completely different: the SanDisk Sansa Fuze+

Periodically I have used this blog to talk about useful devices and software packages that still work with PowerPC Tiger, mostly to steal audience away from Dan DeVoto. My current phone is a Galaxy Nexus, which Samsung saw fit not to put a microSD card slot in, and the internal flash is not accessible like every other proper damn device as a USB Mass Storage Class device -- only Media Transfer Protocol, which has never been properly supported on any Power Mac, even with tools like xnjb. I have various workarounds for getting photos and videos off the device so I can back those up, but nothing that works reliably for getting music and videos on it, other than glacially slow Bluetooth transfers.

So I'm supposed to be on a plane tomorrow to Dallas so that I can swelter like a pig at departmental expense. I'd rather do this with some music of my choosing, but the Galaxy Nexus is not going to serve that purpose without a network connection in the air (Amazon Cloud Player probably wouldn't be happy with in-flight WiFi either, to say nothing of the rest of the passengers), so I picked up an 8GB SanDisk Sansa Fuze+ for $80 at Worst Best Buy on the way out of the office.

The Fuze+ is a 320x240 resolution ARM-based music and video player. At $80 (and this is a Best Buy price -- you can do a lot better), you should not expect much, and that's what you get: the screen is low resolution and has crummy angles, fonts are jaggy, the UI is laggy, the touch pad is wacky and the interface is crappy. But you also get a device that has a microSD card slot for expansion, supports lots of audio codecs (I prefer uncompressed music, and while it does not support AIFF, it does support uncompressed WAV and FLAC), has good sound (at least for FLAC), has decent battery life (about 18-24 hours of audio play on a charge depending on how much you have the screen on), charges over regular USB with a standard micro-USB port, and most important of all has USB Mass Storage support as well as Media Transport Protocol support. Heck, they even throw in an FM radio and a voice recorder.

Does it work with 10.4? Of course. In fact, with USB MSC support, it works on Linux too, and even works with good old Mac OS 9. Format the unit from its internal menu (do NOT format it from Disk Utility; you could brick it) to get rid of the stupid sample tracks it comes with, set it to Mass Storage mode from the internal menu, and plug it in. Create folders under Music and just copy your music over. You can create .m3u playlists as well which can reference any audio file, not just MP3s (older Sansa devices used the bizarre PLP/PLA format, but the Fuze+ does not support these anymore). The only stipulations are that the .m3u file must have DOS line endings (i.e., \r\n), should avoid referencing files outside of that folder, and must be EXTM3U format (i.e., the first line must be #EXTM3U\r\n). Don't use MacRoman; use Windows-1252. BBEdit can help you do this right. Put the playlist in the same folder with the music files it references and after you unmount and unplug it from the computer, the playlist and your new music will appear in the playlists screen along with any other playlists anywhere on the device (it will also find playlists in the Trash, so make sure you empty the Trash before you put it away). Play and enjoy. I wrote a little Perl script to turn my AIFF files into FLAC and construct the M3U playlists for me. ID3 tags are helpful, but they are not necessary.

Note that if you change the M3U playlist by hand from the Mac after you've already uploaded it to the device, the Fuze+ may not notice and might still play the songs in the old order. If that's the case, you'll have to remove that playlist and its files, unmount the device and let the Fuze+ discover they are gone, then remount the device, add them back and let the Fuze+ rediscover them. This is the only glitch I've found with this method.

Fortunately, if the built-in firmware is not sufficient for you, I am delighted to discover that not only is the Sansa Fuze+ supported by Rockbox, but that the Rockbox Utility used for installing it and managing its firmware still works fully with PowerPC 10.4. Great work, guys! Please note that as of this writing the Fuze+ port is still considered unstable and only mostly works, but it looks like it hasn't long to go before it becomes a regular version. I should also add that I have not done any testing of the Fuze+'s MPEG-4 video support, but I don't expect it to perform particularly well with that task and neither should you (it's not what I bought it for in any case).

ObTenFourFoxStuff. I discovered a memory leak in 22 which is due to our custom font code -- we do not use the CoreText/CGFont-based backend of Firefox any more for enumerating fonts, instead using good old ATSUI and converting ATS font references to CGFontRefs on the fly where needed. Well, it turns out we're leaking those synthesized references, and have been ever since 19 (17 is unaffected), but for some reason it's worse in 22. That fix will be part of 22.0.1 as well (issue 232). Again, it does not affect 17.0.x.

The death march to the new Australis interface continues inexorably in the bloodied halls of Mountain View. Instead of "ship in 25" it is currently "land in 25, ship in 26," and various intractable performance problems are making the probability of "land in 26, ship in 27" increase daily. One particularly serious performance issue was specific to 10.6, which alarms me that either Australis is not being written with 10.6 in mind or that it requires hardware assistance only available on 10.7 and up to be performant. Worst of all, the release drivers were perfectly willing to ship with that particular issue unresolved (though other improvements eventually mitigated it) which we should take as evidence of 10.6's shrinking window of support at the administrative level. The strong dependence on gradients is also going to make coding it up unfun, since 10.4 does not have the utility CGGradient class and requires lots of inconvenient CGShading workarounds. We'll deal with that as the dust starts to clear in the 25 timeframe.

The patches for 24 are all down and merged, but I haven't tried to build it yet. I'm actually scared to, frankly.

Useful tips: making threads out of Tiger can be difficult (plus: Tenfourbird is awesome)

Because it has all these claws and sharp bits and it doesn't like being made into thread I have been displeased with the performance thus far of 22 after a while using it regularly (recall that I was on vacation in the beautiful Pacific Northwest and my iBook G4 usually runs 17 so that I can make sure the stable branch is still working well), and I think I finally know why. It's a manifestation, yet again, of Tiger's comparatively poor multithreading performance compared to Leopard and later kernels, the crux of which is issue 193 (and should hopefully be solved once and for all if I can get jemalloc functioning properly, which is issue 218). Unfortunately, the performance issues appear to be because of Mozilla's well-intentioned attempts to move stuff off the main thread, and of the local test systems the multi-processor Power Macs -- the ones you'd think would benefit most -- actually do tremendously worse with this approach in 10.4.

Permit me to demonstrate. If you're on a single processor Mac, going to Butterfly Labs (the reliable test case for this problem, though I'm sure there are others) is sluggish and beachballs a bit, though it eventually loads. My iMac G4 grinds into it after around 30 seconds, which is lousy, but bearable. Now do it on a multi processor Mac. On my 10.4 quad G5, it takes minutes despite having four times the cores, four times the L2 cache and 2.5 times the clock speed, even with processor performance set to Highest, during which the browser is unresponsive for long periods of time. That, to express it gently, sucks. After some exhaustive debugging I am fully convinced that this too is almost entirely due to inefficiencies in the Tiger kernel's implementation of pthreads. The processors just trip over each other trying to stay synced and the threads get starved for resources.

I am told that issue 193 does not affect 10.5 nearly as badly if at all as it does 10.4, so I've made a speculative fix to NSPR and JavaScript tracked as issue 231. Until I can get jemalloc to work correctly and solve the whole problem, the simplest stopgap solution seems to be to make TenFourFox believe there is only one processor on 10.4 systems, regardless of how many it actually has, which causes the browser not to attempt to spin things into multiple threads. (The patch checks Gestalt() to see if it it's on 10.4; if it isn't, it runs the normal processor detection code.) I'm typing this in such a build on the G5 and the weird intermittent stops and seizes I also observed with 22 are now nearly undetectable, even with the processors throttled to Reduced mode. 10.5 users, please try the reliable test site above on single and multi-processor Power Macs if you have them available. The difference should be extreme on the multi-processor Power Mac if 10.5 is affected by this also and if it is, then we just make it detect only one processor on any Power Mac until we can work around the poor kernel performance later. (jemalloc is number two on my priority list, behind, of course, BaselineCompiler/IonMonkey.) Either way, one form or another of this fix will appear in 22.0.1, which is the interim release to replace Fx23 while I work on Fx24. It won't help sites much that were already bad in previous versions, but it will definitely fix the accumulated regressions.

Other things. Now that Google has made Google Groups slow and hard to navigate, which I used to use for reading Mozilla newsgroups (though I guess that's better than what they did to Google Reader), I finally have an excuse to try Tenfourbird. Normally I read Usenet and E-mail from a shell prompt, but I decided to use Tenfourbird and talk to news.mozilla.org directly to see how well it works, and to keep those groups separate from my regular .newsrc file. And, well, wow. It's quick and fast, and seems pretty much bug-free. I am delighted to see that our anonymous builder in the land of the rising sun has kept pace with us, and I hope he makes the jump to 24 when we do. ありがとうございました。