So long, Opportunity rover

It's time to say goodbye to another PowerPC in space, this time the Opportunity rover, also known as the Mars Exploration Rover B (or MER-1). Finally declared at end of mission today after 5,352 Mars solar days when NASA couldn't re-establish contact, it had been apparently knocked off-line by a dust storm and was unable to restart. Originally intended for a 90 Mars solar day mission, its mission became almost 60 times longer than anticipated and it traveled nearly 30 miles on the surface in total. Spirit, or MER-2, its sister unit, had previously reached end of mission in 2010.

Both Opportunity and Spirit were powered by the 20MHz BAE RAD6000, a radiation-hardened version of the original IBM POWER1 RISC Single Chip CPU and the indirect ancestor of the PowerPC 601. Many PowerPC-based spacecraft are still in operation, both with the original RAD6000 and its successor the RAD750, a radiation-hardened version of the G3.

Meanwhile, the Curiosity rover, which is running a pair of RAD750s (one main and one backup, plus two SPARC accessory CPUs), is still in operation at 2,319 Mars solar days and ticking. There is also the 2001 Mars Odyssey orbiter, which is still circling the planet with its own RAD6000 and is expected to continue operations until 2025. Curiosity's design is likely to be reused for the Mars 2020 rover, meaning possibly even more PowerPC design will be exploring the cosmos in the very near future.

TenFourFox FPR12 available

TenFourFox Feature Parity Release 12 final is now available for testing (downloads, hashes, release notes). There are no additional changes except for one outstanding security update and to refresh the certificate and TLD stores. As usual it will go live Monday evening Pacific time assuming no difficulties.

For "lucky" FPR13 I want to take a whack at solving issue 541, since my ability to work on Github from the G5 is seriously impaired at the moment (I have to resort to various workarounds or do tasks from the Talos II with regular Firefox). Since this has some substantial regression risk it will probably be the only JavaScript change I do for that release pending further feasibility tests on the whole enchilada. However, a couple people have asked again about AppleScript support and there is an old patch around that I think could be dusted off and made to work. That release is scheduled for March 19.

Speaking of the Talos II, I should be getting my second POWER9 system in soon, a 4-core Raptor Blackbird we'll be using as a media system. I've already got the mATX case picked out and some decent peripherals and it will probably run Fedora also, since I'm pretty accustomed to it by now. If these systems are starting to interest you but the sticker shock of a full T2 loadout is too much, the Blackbird can give you a taste of next-generation Power ISA without too much pain to your pocketbook.

Meanwhile, over on our sister Talospace blog, if you've been thinking about the Linux plunge (either with a POWER9 or on your own system) but your Mac habits die hard, here's a better way to get the Command key to work properly than faffing about with AutoKey and you can still run Mac OS X apps in virtualization or emulation.

TenFourFox FPR12b1 available

TenFourFox Feature Parity 12 beta 1 is now available (downloads, hashes, release notes). As before, this is a smaller-scope release with no new features, just fixes and improvements. The big changes are a fix for CVE-2018-12404, a holdover security fix from FPR11 that also helps improve JavaScript optimization, and Raphael's hand-coded assembly language AltiVec-accelerated string matching routines with special enhancements for G5 systems. These replace the C routines I wrote using AltiVec intrinsics, which will be removed from our hacked NSPR libc source code once his versions stick.

Unfortunately, we continue to accumulate difficult-to-solve JavaScript bugs. The newest one is issue 541, which affects Github most severely and is hampering my ability to use the G5 to work in the interface. This one could be temporarily repaired with some ugly hacks and I'm planning to look into that for FPR13, but I don't have this proposed fix in FPR12 since it could cause parser regressions and more testing is definitely required. However, the definitive fix is the same one needed for the frustrating issue 533, i.e., the new frontend bindings introduced with Firefox 51. I don't know if I can do that backport (both with respect to the technical issues and the sheer amount of time required) but it's increasingly looking like it's necessary for full functionality and it may be more than I can personally manage.

Meanwhile, FPR12 is scheduled for parallel release with Firefox 60.5/65 on January 29. Report new issues in the comments (as always, please verify the issue doesn't also occur in FPR11 before reporting a new regression, since sites change more than our core does).

TenFourFox FPR11 available

TenFourFox Feature Parity Release 11 final is now available for testing (downloads, hashes, release notes). Issue 525 has stuck, so that's being shipped and we'll watch for site or add-on compatibility fallout (though if you're reporting a site or add-on that doesn't work with FPR11, or for that matter any release, please verify that it still worked with prior versions: particularly for websites, it's more likely the site changed than we did). There are no other changes other than bringing security fixes up to date. Assuming no problems, it will go live tomorrow evening as usual.

FPR12 will be a smaller-scope release but there will still be some minor performance improvements and bugfixes, and with any luck we will also be shipping Raphaël's enhanced AltiVec string matcher in this release as well. Because of the holidays, family visits, etc., however, don't expect a beta until around the second week of January.

Edge gets Chrome-plated, and we're all worse off

I used to think that WebKit would eat the world, but later on I realized it was Blink. In retrospect this should have been obvious when the mobile version of Microsoft Edge was announced to use Chromium (and not Microsoft's own rendering engine EdgeHTML), but now rumour has it that Edge on its own home turf -- Windows 10 -- will be Chromium too. Microsoft engineers have already been spotted committing to the Chromium codebase, apparently for the ARM version. No word on whether this next browser, codenamed Anaheim, will still be called Edge.

In the sense that Anaheim won't (at least in name) be Google, just Chromium, there's reason to believe that it won't have the repeated privacy erosions that have characterized Google's recent moves with Chrome itself. But given how much DNA WebKit and Blink share, that means there are effectively two current major rendering engines left: Chromium and Gecko (Firefox). The little ones like NetSurf, bless its heart, don't have enough marketshare (or currently features) to rate, Trident in Internet Explorer 11 is intentionally obsolete, and the rest are too deficient to be anywhere near usable (Dillo, etc.). So this means Chromium arrogates more browsershare to itself and Firefox will continue to be the second class citizen until it, too, has too small a marketshare to be relevant. Then Google has eaten the Web. And we are worse off for it.

Bet Mozilla's reconsidering that stupid embedding decision now.

Something for the weekend: Classic MacOS Lua

First, a TenFourFox FPR11 update: the release is delayed until December 10-ish to coincide with the updated release date of Firefox 66/60.4 ESR. Unfortunately due to my absence over the holidays this leaves very little development time for FPR12 in December, so the beta is not likely to emerge until mid-January. Issue 533 ("this is undefined") is still my biggest priority because of the large number of sites still using the tainted version of Uglify-ES, but I still have no solution figured out yet, and the 15-minutes-or-longer build time to reconstruct test changes in JavaScript if I touch any headers seriously slows debugging. If you've had issues with making new shipments in United Parcel Service's on-line shipping application, or getting into your Citibank account, this is that bug.

So in the meantime, since we're all classic Mac users here, try out MacLua, a new port of the Lua programming language to classic MacOS. I'm rather fond of Lua, which is an incredibly portable scripting language, ever since I learned it to write PalmOS applications in Plua (I maintained the Mac OS X cross-compiler for it). In fact, I still use Plua for my PalmOS-powered Hue light controller.

MacLua gives you a REPL which you can type Lua into and will run your Lua scripts, but it has two interesting features: first, you can use it as an MPW tool, and second, it allows plugins that could potentially connect it to the rest of the classic Mac Toolbox. The only included component is a simple one for querying Gestalt as an educational example, but a component for TCP sockets through MacTCP or OpenTransport or being able to display dialogue boxes and other kinds of system resources would seem like a logical next step. This was something really nice about Plua that it included GUI and network primitives built-in as included modules. The author of this port clearly has a similar idea in mind.

You can still compile Lua natively on 10.4, and that would probably be more useful if you wanted to write Lua scripts on an OS X Power Mac, but if you have a 68K or beige Power Mac around this Lua port can run on systems as early as 7.1.2 (probably any 68020 System 7 Mac if you install the CFM-68K Runtime Enabler). I look forward to seeing how it evolves, and the fact that it was built with QEMU as a Mac emulator not only is good evidence of how functional QEMU's classic Mac emulation is getting but also means there may be a chance at some other ports to the classic Mac OS in the future.

And now for something completely different: SCSI2SD is not a panacea

When well-meaning but underinformed folks hear you keep old Macs alive, without fail they tell you to replace the capacitors, and install a SCSI2SD.

The first bit of "advice" drives me up the wall because while it's true for many Macs, it's now uttered in almost a knee-jerk fashion anywhere almost any vintage computer is mentioned (this kind of mindlessness infests pinball machine restoration as well). To be sure, in my experience 68030-based Macs and some of the surrounding years will almost invariably need a recap. All of my Color Classics, IIcis, IIsis and SE/30s eventually suffered capacitor failures and had to be refurbished, and if you have one of these machines that has not been recapped yet it's only a matter of time.

But this is probably not true for many other systems. Many capacitors have finite lifetimes but good quality components can function for rather long periods especially with regular periodic use; in particular, early microcomputers from the pre-surface mount era (including early Macs) only infrequently need capacitor replacement. My 1987-vintage Mac Plus and my Commodore microcomputers, for example, including my blue foil PET 2001 and a Revision A KIM-1 older than I am, function well and have never needed their capacitors to be serviced. Similarly, newer Macs, even many beige Power Macs, still have long enough residual capacitor lifespan that caps are (at least presently) very unlikely to be the problem, and trying to replace them runs you a greater risk of damaging the logic board than actually fixing anything (the capacitor plague of the early 2000s notwithstanding).

That brings us to the second invariant suggestion, a solid state disk replacement, of which at least for SCSI-based systems the SCSI2SD is probably the best known. Unlike capacitors, this suggestion has much better evidence to support it as a general recommendation as indicated by the SCSI2SD's rather long list of reported working hardware (V5 device compatibility, V6 device compatibility). However, one thing that isn't listed (much) on those pages is whether the functionality is still the same for other OSes these computers might run. That will become relevant in a moment.

My own history with SCSI2SD devices has been generally positive, but there have been issues. My very uncommon IBM ThinkPad "800" (the predecessor of the ThinkPad 850, one of the famous PowerPC IBM ThinkPads) opened its SCSI controller fuse after a SCSI2SD was installed, and so far has refused to boot from any device since then. The jury is still out on whether the ThinkPad's components were iffy to begin with or the SCSI2SD was defective, but the card went to the recycler regardless since I wasn't willing to test it on anything else. On the other hand, my beautiful bright orange plasma screen Solbourne S3000 has used a V5 SCSI2SD since the day I got it without any issues whatsoever, and I installed a 2.5" version in a Blackbird PowerBook 540c which runs OS 7.6.1 more or less stably.

You have already met thule, the long-suffering Macintosh IIci which runs the internal DNS and DHCP and speaks old-school EtherTalk, i.e., AppleTalk over DDP, to my antediluvian Mac (and one MS-DOS PC) clients for file and print services on the internal non-routable network. AFAIK NetBSD is the only maintained OS left that still supports DDP. Although current versions of the OS apparently continue to support it, I had problems even as early as 1.6 with some of my clients, so I keep it running NetBSD 1.5.2 with my custom kernel. However, none of the essential additional software packages such as netatalk are available anywhere obvious for this version, so rebuilding from scratch would be impossible; instead, a full-system network backup runs at regular intervals to archive the entire contents of the OS, which I can restore to the system from one of the other machines (see below). It has a Farallon EtherWave NuBus Ethernet card, 128MB of RAM (the maximum for the IIci) and the IIci 32K L2 cache card. It has run nearly non-stop since 1999. Here it is in the server room:

In 2014 it finally blew its caps and needed a logic board refurbishment (the L1 card was recapped at the same time). Its hard disk has long been marginal and the spare identical drive had nearly as many hours of service time. Rotating between the two drives and blacklisting the bad sectors as they appeared was obviously only a temporary solution. Though my intent was to run the drive(s) completely into the ground before replacing them, bizarre errors like corrupt symbols not being found in libc.so told me the end was near. Last week half of the house wasn't working because nothing could get a DHCP lease. I looked on the console and saw this:

The backup server had a mangled full system tar backup on it that was only half-completed and timestamped around the time the machine had kernel-panicked, implying the machine freaked out while streaming the backup to the network. This strongly suggested the hard disk had seized up at some point during the process. After I did some hardware confidence testing with Snooper and Farallon's diagnostics suite, I concluded that the RAM, cache, CPU, network card and logic board were fine and the hard disk was the only problem.

I have three NetBSD machines right now: this IIci, a Quadra 605 with a full 68040 which I use for disk cloning and system rescue (1.6.2), and a MIPS-based Cobalt RaQ 2 which acts as an NFS server. I briefly considered moving services to the RaQ, but pretty much nothing could replace this machine for offering AppleShare services and I still have many old Mac clients in the house that can only talk to thule, so I chose to reconstruct it. While it still came up in the System 7.1 booter partition (NetBSD/mac68k requires Mac OS to boot), I simply assumed that everything was corrupt and the hard disk was about to die, and that I should rebuild to a brand new device from the last successfully completed backup rather than try to get anything out of the mangled most recent one.

Some people had success with using SCSI2SDs on other NetBSD systems, so I figured this was a good time to try it myself. I had a SCSI2SD v5 and v6 I had recently ordered and a few 8GB microSD cards which would be more than ample space for the entire filesystem and lots of wear-leveling. I tried all of these steps on the v5 and most of them on the v6, both with up-to-date firmware, and at least a couple iterations with different SD media. Here goes.

First attempt: I set up the Q605 to run a block-level restore from an earlier block-level image. Assuming this worked, the next step would have been overlaying the image with changed files from the last good tar backup. After the block-level image restore was complete I put the SCSI2SD into the IIci to make sure it had worked so far. It booted the Mac OS partition, started the NetBSD booter and immediately panicked.

Second attempt: I manually partitioned the SCSI2SD device into the Mac and NetBSD root and swap partitions. That seemed to work, so next I attempted to use the Mac-side Mkfs utility to build a clean UFS filesystem on the NetBSD partition, but Mkfs didn't like the disk geometry and refused to run. After suppressing the urge to find my ball-peen hammer, I decided to reconfigure the SCSI2SD with the geometry from the Quantum Fireball drive it was replacing and it accepted that. I then connected it to the Q605 and restored the files from the last good tar backup to the NetBSD partition. It mounted, unmounted and fscked fine. Triumphantly, I then moved it over to the IIci, where it booted the Mac OS partition, started the NetBSD booter and immediately panicked.

Third attempt: theorizing maybe something about the 1.6 system was making the 1.5.2 system unstable, I decided to create a 1.5.2 miniroot from an old install CD I still had around, boot that on the IIci, and then install on the IIci (and eliminate the Q605 as a factor). I decided to do this as a separate SCSI target, so I configured the SCSI2SD to look like two 2GB drives. Other than an odd SCSI error #5, Mkfs seemed to successfully create a new UFS filesystem on the second target. I then ran the Mac-side Installer to install a clean 1.5.2. The minishell started and seemed to show a nice clean UFS volume, so I attempted to install the package sets (copied to the disk earlier). After a few files from the first install set were written, however, the entire system locked up.

Fourth attempt: maybe it didn't like the second target, and maybe I had to take a break to calm down at this point, maybe. I removed the second target and repartitioned the now single 2GB SCSI target into a Mac, miniroot, and full root and swap partitions. This time, Mkfs refused to newfs the new miniroot no matter what the geometry was set to and no matter how I formatted the partition, and I wasn't in any mood to mess around with block and fragment sizes.

Fifth attempt: screw the miniroot and screw Apple and screw the horse Jean-Louis Gassee rode in on and all his children with a rusty razorblade sideways. I might also have scared the cat around this time. I repartitioned again back to a MacOS, swap and root partition, and Mkfs on the IIci grudgingly newfsed that. I was able to mount this partition on the Q605 and restore from the last good backup. It mounted, unmounted and fscked fine. I then moved it over to the IIci, where it booted the Mac OS partition, started the NetBSD booter and immediately panicked.

At this point I had exceeded my patience and blood pressure, and there was also a non-zero probability the backup itself was bad. There was really only one way to find out. As it happens, I had bought a stack of new-old-stock Silicon Graphics-branded Quantum Atlas II drives to rebuild my SGI Indigo², so I pulled one of those out of the sealed anti-static wrap, jumpered it to spin up immediately, low-level formatted, partitioned and Mkfsed it on the IIci, connected it to the Q605 and restored from the tar backup. It mounted, unmounted and fscked fine. I then moved it over to the IIci, where it booted the Mac OS partition, started the NetBSD booter and ... the OS came up and, after I was satisfied with some cursory checks in single-user, uneventfully booted into multi-user. The backup was fine and the system was restored.

Since the Atlas II is a 7200rpm drive (the Fireball SE it replaced was 5400rpm) and runs a little hot, I took the lid off and placed it sideways to let it vent a bit better.

As a nice bonus the Atlas II is a noticeably faster drive, not only due to the faster rotation but also the larger cache. There's been no more weird errors, the full system backup overnight yesterday terminated completely and in record time, most system functions are faster overall and even some of the other weird glitchiness of the machine has disappeared. (The device on top of the case is the GSM terminal I use for sending text message commands to the main server.) I've allocated another one of the NOS Atlas drives to the IIci, and I'll find something else for rebuilding the Indigo². I think I've got some spare SCSI2SDs sitting around ...

So what's the moral of this story?

The moral of the story is that SCSI is awfully hard to get right and compatibility varies greatly on the controller, operating system and device. I've heard of NetBSD/mac68k running fine with SCSI2SD devices on other Quadras, and it appeared to work on my Quadra 605 as well (the infamous "LC" Quadra), but under no circumstances would NetBSD on my IIci want any truck with any hardware revision of it. Despite that, however, the IIci worked fine with it in MacOS except when the NetBSD installer took over (and presumably started using its own SCSI routines instead of the SCSI Manager) or tried to boot the actual operating system. It's possible later versions of NetBSD would have corrected this, but it's unlikely they would be corrected in the Mac-side Mkfs and Installer because of their standalone codebase, and the whole point of a solid-state disk emulator is to run the old OS your system has to run anyway. Of course, all of this assumes my IIci doesn't have some other hardware flaw that only manifests in this particular fashion, but that seems unlikely.

Thus, if you've got a mission-critical or otherwise irreplaceable machine with a dying hard drive, maybe your preservation strategy should simply be to find it another spinning disk. SCSI2SD is perfectly fine for hobbyist applications, and probably works well enough in typical environments, but it's not a panacea for every situation and the further you get away from frequent usage cases the less likely it's been tested in those circumstances. If it doesn't work or you don't want to risk it, you're going to need that spare drive after all.

There's also the issue of how insane it is to use almost 30-year-old hardware for essential services (the IIci was introduced in 1989), but anyone who brings that up in the comments on this particular blog gets to stand in the corner and think about what they just said.