Thursday, November 15, 2018

And now for something completely different: SCSI2SD is not a panacea

When well-meaning but underinformed folks hear you keep old Macs alive, without fail they tell you to replace the capacitors, and install a SCSI2SD.

The first bit of "advice" drives me up the wall because while it's true for many Macs, it's now uttered in almost a knee-jerk fashion anywhere almost any vintage computer is mentioned (this kind of mindlessness infests pinball machine restoration as well). To be sure, in my experience 68030-based Macs and some of the surrounding years will almost invariably need a recap. All of my Color Classics, IIcis, IIsis and SE/30s eventually suffered capacitor failures and had to be refurbished, and if you have one of these machines that has not been recapped yet it's only a matter of time.

But this is probably not true for many other systems. Many capacitors have finite lifetimes but good quality components can function for rather long periods especially with regular periodic use; in particular, early microcomputers from the pre-surface mount era (including early Macs) only infrequently need capacitor replacement. My 1987-vintage Mac Plus and my Commodore microcomputers, for example, including my blue foil PET 2001 and a Revision A KIM-1 older than I am, function well and have never needed their capacitors to be serviced. Similarly, newer Macs, even many beige Power Macs, still have long enough residual capacitor lifespan that caps are (at least presently) very unlikely to be the problem, and trying to replace them runs you a greater risk of damaging the logic board than actually fixing anything (the capacitor plague of the early 2000s notwithstanding).

That brings us to the second invariant suggestion, a solid state disk replacement, of which at least for SCSI-based systems the SCSI2SD is probably the best known. Unlike capacitors, this suggestion has much better evidence to support it as a general recommendation as indicated by the SCSI2SD's rather long list of reported working hardware (V5 device compatibility, V6 device compatibility). However, one thing that isn't listed (much) on those pages is whether the functionality is still the same for other OSes these computers might run. That will become relevant in a moment.

My own history with SCSI2SD devices has been generally positive, but there have been issues. My very uncommon IBM ThinkPad "800" (the predecessor of the ThinkPad 850, one of the famous PowerPC IBM ThinkPads) opened its SCSI controller fuse after a SCSI2SD was installed, and so far has refused to boot from any device since then. The jury is still out on whether the ThinkPad's components were iffy to begin with or the SCSI2SD was defective, but the card went to the recycler regardless since I wasn't willing to test it on anything else. On the other hand, my beautiful bright orange plasma screen Solbourne S3000 has used a V5 SCSI2SD since the day I got it without any issues whatsoever, and I installed a 2.5" version in a Blackbird PowerBook 540c which runs OS 7.6.1 more or less stably.

You have already met thule, the long-suffering Macintosh IIci which runs the internal DNS and DHCP and speaks old-school EtherTalk, i.e., AppleTalk over DDP, to my antediluvian Mac (and one MS-DOS PC) clients for file and print services on the internal non-routable network. AFAIK NetBSD is the only maintained OS left that still supports DDP. Although current versions of the OS apparently continue to support it, I had problems even as early as 1.6 with some of my clients, so I keep it running NetBSD 1.5.2 with my custom kernel. However, none of the essential additional software packages such as netatalk are available anywhere obvious for this version, so rebuilding from scratch would be impossible; instead, a full-system network backup runs at regular intervals to archive the entire contents of the OS, which I can restore to the system from one of the other machines (see below). It has a Farallon EtherWave NuBus Ethernet card, 128MB of RAM (the maximum for the IIci) and the IIci 32K L2 cache card. It has run nearly non-stop since 1999. Here it is in the server room:

In 2014 it finally blew its caps and needed a logic board refurbishment (the L1 card was recapped at the same time). Its hard disk has long been marginal and the spare identical drive had nearly as many hours of service time. Rotating between the two drives and blacklisting the bad sectors as they appeared was obviously only a temporary solution. Though my intent was to run the drive(s) completely into the ground before replacing them, bizarre errors like corrupt symbols not being found in libc.so told me the end was near. Last week half of the house wasn't working because nothing could get a DHCP lease. I looked on the console and saw this:

The backup server had a mangled full system tar backup on it that was only half-completed and timestamped around the time the machine had kernel-panicked, implying the machine freaked out while streaming the backup to the network. This strongly suggested the hard disk had seized up at some point during the process. After I did some hardware confidence testing with Snooper and Farallon's diagnostics suite, I concluded that the RAM, cache, CPU, network card and logic board were fine and the hard disk was the only problem.

I have three NetBSD machines right now: this IIci, a Quadra 605 with a full 68040 which I use for disk cloning and system rescue (1.6.2), and a MIPS-based Cobalt RaQ 2 which acts as an NFS server. I briefly considered moving services to the RaQ, but pretty much nothing could replace this machine for offering AppleShare services and I still have many old Mac clients in the house that can only talk to thule, so I chose to reconstruct it. While it still came up in the System 7.1 booter partition (NetBSD/mac68k requires Mac OS to boot), I simply assumed that everything was corrupt and the hard disk was about to die, and that I should rebuild to a brand new device from the last successfully completed backup rather than try to get anything out of the mangled most recent one.

Some people had success with using SCSI2SDs on other NetBSD systems, so I figured this was a good time to try it myself. I had a SCSI2SD v5 and v6 I had recently ordered and a few 8GB microSD cards which would be more than ample space for the entire filesystem and lots of wear-leveling. I tried all of these steps on the v5 and most of them on the v6, both with up-to-date firmware, and at least a couple iterations with different SD media. Here goes.

First attempt: I set up the Q605 to run a block-level restore from an earlier block-level image. Assuming this worked, the next step would have been overlaying the image with changed files from the last good tar backup. After the block-level image restore was complete I put the SCSI2SD into the IIci to make sure it had worked so far. It booted the Mac OS partition, started the NetBSD booter and immediately panicked.

Second attempt: I manually partitioned the SCSI2SD device into the Mac and NetBSD root and swap partitions. That seemed to work, so next I attempted to use the Mac-side Mkfs utility to build a clean UFS filesystem on the NetBSD partition, but Mkfs didn't like the disk geometry and refused to run. After suppressing the urge to find my ball-peen hammer, I decided to reconfigure the SCSI2SD with the geometry from the Quantum Fireball drive it was replacing and it accepted that. I then connected it to the Q605 and restored the files from the last good tar backup to the NetBSD partition. It mounted, unmounted and fscked fine. Triumphantly, I then moved it over to the IIci, where it booted the Mac OS partition, started the NetBSD booter and immediately panicked.

Third attempt: theorizing maybe something about the 1.6 system was making the 1.5.2 system unstable, I decided to create a 1.5.2 miniroot from an old install CD I still had around, boot that on the IIci, and then install on the IIci (and eliminate the Q605 as a factor). I decided to do this as a separate SCSI target, so I configured the SCSI2SD to look like two 2GB drives. Other than an odd SCSI error #5, Mkfs seemed to successfully create a new UFS filesystem on the second target. I then ran the Mac-side Installer to install a clean 1.5.2. The minishell started and seemed to show a nice clean UFS volume, so I attempted to install the package sets (copied to the disk earlier). After a few files from the first install set were written, however, the entire system locked up.

Fourth attempt: maybe it didn't like the second target, and maybe I had to take a break to calm down at this point, maybe. I removed the second target and repartitioned the now single 2GB SCSI target into a Mac, miniroot, and full root and swap partitions. This time, Mkfs refused to newfs the new miniroot no matter what the geometry was set to and no matter how I formatted the partition, and I wasn't in any mood to mess around with block and fragment sizes.

Fifth attempt: screw the miniroot and screw Apple and screw the horse Jean-Louis Gassee rode in on and all his children with a rusty razorblade sideways. I might also have scared the cat around this time. I repartitioned again back to a MacOS, swap and root partition, and Mkfs on the IIci grudgingly newfsed that. I was able to mount this partition on the Q605 and restore from the last good backup. It mounted, unmounted and fscked fine. I then moved it over to the IIci, where it booted the Mac OS partition, started the NetBSD booter and immediately panicked.

At this point I had exceeded my patience and blood pressure, and there was also a non-zero probability the backup itself was bad. There was really only one way to find out. As it happens, I had bought a stack of new-old-stock Silicon Graphics-branded Quantum Atlas II drives to rebuild my SGI Indigo2, so I pulled one of those out of the sealed anti-static wrap, jumpered it to spin up immediately, low-level formatted, partitioned and Mkfsed it on the IIci, connected it to the Q605 and restored from the tar backup. It mounted, unmounted and fscked fine. I then moved it over to the IIci, where it booted the Mac OS partition, started the NetBSD booter and ... the OS came up and, after I was satisfied with some cursory checks in single-user, uneventfully booted into multi-user. The backup was fine and the system was restored.

Since the Atlas II is a 7200rpm drive (the Fireball SE it replaced was 5400rpm) and runs a little hot, I took the lid off and placed it sideways to let it vent a bit better.

As a nice bonus the Atlas II is a noticeably faster drive, not only due to the faster rotation but also the larger cache. There's been no more weird errors, the full system backup overnight yesterday terminated completely and in record time, most system functions are faster overall and even some of the other weird glitchiness of the machine has disappeared. (The device on top of the case is the GSM terminal I use for sending text message commands to the main server.) I've allocated another one of the NOS Atlas drives to the IIci, and I'll find something else for rebuilding the Indigo2. I think I've got some spare SCSI2SDs sitting around ...

So what's the moral of this story?

The moral of the story is that SCSI is awfully hard to get right and compatibility varies greatly on the controller, operating system and device. I've heard of NetBSD/mac68k running fine with SCSI2SD devices on other Quadras, and it appeared to work on my Quadra 605 as well (the infamous "LC" Quadra), but under no circumstances would NetBSD on my IIci want any truck with any hardware revision of it. Despite that, however, the IIci worked fine with it in MacOS except when the NetBSD installer took over (and presumably started using its own SCSI routines instead of the SCSI Manager) or tried to boot the actual operating system. It's possible later versions of NetBSD would have corrected this, but it's unlikely they would be corrected in the Mac-side Mkfs and Installer because of their standalone codebase, and the whole point of a solid-state disk emulator is to run the old OS your system has to run anyway. Of course, all of this assumes my IIci doesn't have some other hardware flaw that only manifests in this particular fashion, but that seems unlikely.

Thus, if you've got a mission-critical or otherwise irreplaceable machine with a dying hard drive, maybe your preservation strategy should simply be to find it another spinning disk. SCSI2SD is perfectly fine for hobbyist applications, and probably works well enough in typical environments, but it's not a panacea for every situation and the further you get away from frequent usage cases the less likely it's been tested in those circumstances. If it doesn't work or you don't want to risk it, you're going to need that spare drive after all.

There's also the issue of how insane it is to use almost 30-year-old hardware for essential services (the IIci was introduced in 1989), but anyone who brings that up in the comments on this particular blog gets to stand in the corner and think about what they just said.

Sunday, November 11, 2018

ICYMI: what's new on Talospace

In the shameless plug category, in case you missed them, two original articles on Talospace, our sister blog: making your Talos II into an IBM pSeries (yes, you can run AIX on a Talos II with Linux KVM), and roadgeeking with the Talos II (because the haters gotta hate and say POWER9 isn't desktop ready, which is just FUD FUD FUD).

Thursday, November 8, 2018

Friday, November 2, 2018

TenFourFox FPR11b1 available

TenFourFox Feature Parity 11 beta 1 is now available (downloads, hashes, release notes). As mentioned, FPR11 and FPR12 will be smaller in scope due to less low-hanging fruit to work on and other competing projects such as the POWER9 JIT, but there are still new features and improvements.

The most notable one is my second attempt to get unique origin for data: URIs to stick (issue 525). This ran aground in FPR10 and had to be turned off because of compatibility issues with the Firefox 45 version of uBlock Origin, which would be too major an add-on for me to ignore breaking. FPR11 now has a shim in it to allow the old behaviour for data URL access initiated by the internal system principal (including add-ons) but use the new behaviour for web content, and seems to properly reject the same test cases while allowing uBlock to run normally. As before, we really need this in the browser to defend against XSS attacks, so please test thoroughly. Once again, if you experience unusual behaviour in this version, please flip security.data_uri.unique_opaque_origin to false and restart the browser. If the behaviour changes, then this was the cause and you should report it in the comments.

FPR11 also has a more comprehensive fix for sites that use Cloudflare Rocket Loader, a minor speedup to the JavaScript JIT, and new support for two JavaScript features (Symbol.hasInstance and Object.getOwnPropertyDescriptors). In addition, for a small additional speed boost, CSS error reporting is now disabled by default except for debug builds (JavaScript error reporting is of course maintained). If you want this back on, set layout.css.report_errors to true.

After a lot of test cases and poring through bug reports, I think that the issue with Citibank is issue 533. Sites that use that combination of front-end deployment tools will not work in Firefox 50 or lower, which worryingly may affect quite a few, and it seems to be due to the large front-end refactor that landed in Firefox 51. The symptom is usually an error that looks like "this is undefined" in the JavaScript console. Like the other two looming JavaScript issues we are starting to face, this requires a large amount of work and is likely to have substantial regressions if I can get it finished (let alone get it building) at all. I'm looking through the changes to see if any obviously affect the scope of this to see if the actual error that is reported can be worked around, but it may just be masking some bigger problem which requires much more surgery and, if so, would not be feasible to fix unfortunately either.

Thursday, November 1, 2018

OS X kernel exploit in ICMP: Power Macs vulnerable?

I've been puzzling over this for a couple hours, and without a known working proof of concept (which has not yet been revealed) I'm mostly guessing, but I don't believe that Power Macs are vulnerable to CVE-2018-4407.

In a nutshell, the exploit works by sending an abnormal and malicious ICMP packet over a local network to a vulnerable Mac. A vulnerable Mac will attempt to send an error back to the sender with a copy of the abnormal packet header, which has been constructed to be oversize and thus overflows the buffer used for the copy, potentially allowing remote code execution with the contents of the malicious packet.

The current and known vulnerable code dates to about OS X Yosemite (warning: long page). In this version additional code was added to compute the length of what to copy and select different types of buffers, and this later revision appears to be missing a critical line from the original BSD source code that would properly limit the length of what was copied. Prior versions of OS X have a different length calculation (warning: same long page) that doesn't seem to be exploitable in that fashion.

This is not to say that this exploit couldn't be made to work on a Power Mac (or, for that matter, 10.9 and earlier), but my best determination is that at worst the exploit wouldn't work as written. Even if it could, there are three mitigations: first, a remote code exploit would need to be PowerPC specific, which seems unlikely in this day and age; second, the attacker must be on the same network as you in most cases; and third, you can easily defend against it anyway, vulnerable or not, by (as I have previously recommended) enabling the firewall and stealth mode under Sharing > Firewall in System Preferences which disables ICMP replies and thus prevents this section of code from even executing in the first place.

Tuesday, October 30, 2018

The Space Grey Mac mini is too little, too late and too much

I've got a stack of G4 minis here, one of which will probably be repurposed to act as a network bridge with NetBSD, and a white C2D mini which is my 10.6 test machine. They were good boxes when I needed them and they still work great, but Apple to its great shame has really let the littlest Mac rot. Now we have the Space Grey mini, ignoring the disappointing and now almost mold-encrusted 2014 refresh which was one step forward and two steps back, starting at $800 with a quadcore i3, 8GB of memory and 128GB of SSD. The pictures on Ars Technica also show Apple's secret sauce T2 chip on-board.

If you really, really, really want an updated mini, well, here's your chance. But with all the delays in production and Apple's bizarrely variable loadouts over the years the mini almost doesn't matter anymore and the price isn't cheap Mac territory anymore either (remember that the first G4 Mac mini started at $500 in 2005 and people even complained that was too much). If you want low-end, then you're going to buy a NUC-type device or an ARM board and jam that into a tiny case, and you can do it for less unless you need a crapload of external storage (the four Thunderbolt 3 ports on the Space Grey mini are admittedly quite compelling). You can even go Power ISA if you want to: the "Tiny Talos" a/k/a Raptor Blackbird is just around the corner, with the same POWER9 goodness of the bigger T2 systems in a single socket and the (in fairness: unofficial) aim is to get it under $700. That's what I'm gonna buy. Heck, if I didn't have the objections I do to x86, I could probably buy a lot more off-the-shelf for $800 and get more out of it since I'm already transitioning to Linux at home anyway. Why would I bother with chaining myself to the sinking ship that is macOS when it's clear Apple's bottom line is all about iOS?

Don't get me wrong: I'm glad to see Apple at least make a token move to take their computer lines seriously and the upgrade, though horribly delayed and notable more for its tardiness than what's actually in it, is truly welcome. And it certainly would build optimism and some much-needed good faith for whatever the next Mac Pro is being more than vapourware. But I've moved on and while I like my old minis, this one wouldn't lure me back.

Friday, October 26, 2018

And now for something completely different: Make your radio station Power Mac your recording radio station Power Mac

So let's say you've turned your Power Mac into your household radio station. If you're like me and you use the Mac as a repeater to transmit a distant station into your house, you might want to record that audio. Now that Clear Channel iHeartMedia is locking down a lot of their podcasts to their app, for example, you can make an end run around their forcible registration crap by just recording content off the air yourself.

My machine uses a RadioSHARK 2 as its repeater source as previously mentioned in our last article on this topic, and if it were simply a matter of recording from that, you could just (duh) use the RadioSHARK's own software for that purpose as designed. But on my machine, I'm using the Shark with my own custom software to tune and play through USB audio; the RadioSHARK software isn't even involved.

The easiest way to skin this cat, especially on a stand-alone machine which isn't doing anything else, is just to AppleScript QuickTime and use that to do the recording. Unfortunately most of the how-tos you'll find to do this don't work for QuickTime 7 because the dictionary became wildly different (better, too, but different). Here's a quick AppleScript to make QuickTime 7 record 60 seconds from the currently set audio input:

tell application "QuickTime Player"
        activate
        close every window saving no

        new audio recording
        start (first document)
        delay 60
        stop (first document)

        close (first document) saving no
        quit
end tell
What this will do is close all open documents (just in case, to have a predictable state), then create a new audio recording, start it, record 60 seconds from the default audio input, stop it, and then save it to the Desktop as an audio-only QuickTime movie named something like Audio.mov or Audio 2.mov, etc. Despite the saving no at the end, the file actually is saved, in fact at the point where the recording is stopped no matter what you actually do at the time you close it.

If you don't like restricting this to the "first document," you can also do something like set new_movie to id of front movie to get the ID of what's recording, and then use start movie id new_movie and so forth to reference it specifically. Modifying this for the general case without having to close windows and so on is left as an exercise for the reader.

On my radio station Mac, I have a cron job that pipes this to osascript (the commandline AppleScript runtime) to record certain radio shows at certain times, and then copies the resulting file off somewhere for me to play later. There doesn't seem to be a way in this version of QuickTime to change the default filename, but since I don't use the system to record any other audio, I always know the file will be stored as ~/Desktop/Audio.mov and can just move that. Best of all, by using QuickTime to do this job while the USB audio streaming daemon is running, I can still listen simultaneously while it records if I like.

Now, if you'll excuse me, I've got some queued up Handel on the Law to listen to, simultaneously the best and worst legal show on radio.