And now for something completely different: SCSI2SD is not a panacea

When well-meaning but underinformed folks hear you keep old Macs alive, without fail they tell you to replace the capacitors, and install a SCSI2SD.

The first bit of "advice" drives me up the wall because while it's true for many Macs, it's now uttered in almost a knee-jerk fashion anywhere almost any vintage computer is mentioned (this kind of mindlessness infests pinball machine restoration as well). To be sure, in my experience 68030-based Macs and some of the surrounding years will almost invariably need a recap. All of my Color Classics, IIcis, IIsis and SE/30s eventually suffered capacitor failures and had to be refurbished, and if you have one of these machines that has not been recapped yet it's only a matter of time.

But this is probably not true for many other systems. Many capacitors have finite lifetimes but good quality components can function for rather long periods especially with regular periodic use; in particular, early microcomputers from the pre-surface mount era (including early Macs) only infrequently need capacitor replacement. My 1987-vintage Mac Plus and my Commodore microcomputers, for example, including my blue foil PET 2001 and a Revision A KIM-1 older than I am, function well and have never needed their capacitors to be serviced. Similarly, newer Macs, even many beige Power Macs, still have long enough residual capacitor lifespan that caps are (at least presently) very unlikely to be the problem, and trying to replace them runs you a greater risk of damaging the logic board than actually fixing anything (the capacitor plague of the early 2000s notwithstanding).

That brings us to the second invariant suggestion, a solid state disk replacement, of which at least for SCSI-based systems the SCSI2SD is probably the best known. Unlike capacitors, this suggestion has much better evidence to support it as a general recommendation as indicated by the SCSI2SD's rather long list of reported working hardware (V5 device compatibility, V6 device compatibility). However, one thing that isn't listed (much) on those pages is whether the functionality is still the same for other OSes these computers might run. That will become relevant in a moment.

My own history with SCSI2SD devices has been generally positive, but there have been issues. My very uncommon IBM ThinkPad "800" (the predecessor of the ThinkPad 850, one of the famous PowerPC IBM ThinkPads) opened its SCSI controller fuse after a SCSI2SD was installed, and so far has refused to boot from any device since then. The jury is still out on whether the ThinkPad's components were iffy to begin with or the SCSI2SD was defective, but the card went to the recycler regardless since I wasn't willing to test it on anything else. On the other hand, my beautiful bright orange plasma screen Solbourne S3000 has used a V5 SCSI2SD since the day I got it without any issues whatsoever, and I installed a 2.5" version in a Blackbird PowerBook 540c which runs OS 7.6.1 more or less stably.

You have already met thule, the long-suffering Macintosh IIci which runs the internal DNS and DHCP and speaks old-school EtherTalk, i.e., AppleTalk over DDP, to my antediluvian Mac (and one MS-DOS PC) clients for file and print services on the internal non-routable network. AFAIK NetBSD is the only maintained OS left that still supports DDP. Although current versions of the OS apparently continue to support it, I had problems even as early as 1.6 with some of my clients, so I keep it running NetBSD 1.5.2 with my custom kernel. However, none of the essential additional software packages such as netatalk are available anywhere obvious for this version, so rebuilding from scratch would be impossible; instead, a full-system network backup runs at regular intervals to archive the entire contents of the OS, which I can restore to the system from one of the other machines (see below). It has a Farallon EtherWave NuBus Ethernet card, 128MB of RAM (the maximum for the IIci) and the IIci 32K L2 cache card. It has run nearly non-stop since 1999. Here it is in the server room:

In 2014 it finally blew its caps and needed a logic board refurbishment (the L1 card was recapped at the same time). Its hard disk has long been marginal and the spare identical drive had nearly as many hours of service time. Rotating between the two drives and blacklisting the bad sectors as they appeared was obviously only a temporary solution. Though my intent was to run the drive(s) completely into the ground before replacing them, bizarre errors like corrupt symbols not being found in libc.so told me the end was near. Last week half of the house wasn't working because nothing could get a DHCP lease. I looked on the console and saw this:

The backup server had a mangled full system tar backup on it that was only half-completed and timestamped around the time the machine had kernel-panicked, implying the machine freaked out while streaming the backup to the network. This strongly suggested the hard disk had seized up at some point during the process. After I did some hardware confidence testing with Snooper and Farallon's diagnostics suite, I concluded that the RAM, cache, CPU, network card and logic board were fine and the hard disk was the only problem.

I have three NetBSD machines right now: this IIci, a Quadra 605 with a full 68040 which I use for disk cloning and system rescue (1.6.2), and a MIPS-based Cobalt RaQ 2 which acts as an NFS server. I briefly considered moving services to the RaQ, but pretty much nothing could replace this machine for offering AppleShare services and I still have many old Mac clients in the house that can only talk to thule, so I chose to reconstruct it. While it still came up in the System 7.1 booter partition (NetBSD/mac68k requires Mac OS to boot), I simply assumed that everything was corrupt and the hard disk was about to die, and that I should rebuild to a brand new device from the last successfully completed backup rather than try to get anything out of the mangled most recent one.

Some people had success with using SCSI2SDs on other NetBSD systems, so I figured this was a good time to try it myself. I had a SCSI2SD v5 and v6 I had recently ordered and a few 8GB microSD cards which would be more than ample space for the entire filesystem and lots of wear-leveling. I tried all of these steps on the v5 and most of them on the v6, both with up-to-date firmware, and at least a couple iterations with different SD media. Here goes.

First attempt: I set up the Q605 to run a block-level restore from an earlier block-level image. Assuming this worked, the next step would have been overlaying the image with changed files from the last good tar backup. After the block-level image restore was complete I put the SCSI2SD into the IIci to make sure it had worked so far. It booted the Mac OS partition, started the NetBSD booter and immediately panicked.

Second attempt: I manually partitioned the SCSI2SD device into the Mac and NetBSD root and swap partitions. That seemed to work, so next I attempted to use the Mac-side Mkfs utility to build a clean UFS filesystem on the NetBSD partition, but Mkfs didn't like the disk geometry and refused to run. After suppressing the urge to find my ball-peen hammer, I decided to reconfigure the SCSI2SD with the geometry from the Quantum Fireball drive it was replacing and it accepted that. I then connected it to the Q605 and restored the files from the last good tar backup to the NetBSD partition. It mounted, unmounted and fscked fine. Triumphantly, I then moved it over to the IIci, where it booted the Mac OS partition, started the NetBSD booter and immediately panicked.

Third attempt: theorizing maybe something about the 1.6 system was making the 1.5.2 system unstable, I decided to create a 1.5.2 miniroot from an old install CD I still had around, boot that on the IIci, and then install on the IIci (and eliminate the Q605 as a factor). I decided to do this as a separate SCSI target, so I configured the SCSI2SD to look like two 2GB drives. Other than an odd SCSI error #5, Mkfs seemed to successfully create a new UFS filesystem on the second target. I then ran the Mac-side Installer to install a clean 1.5.2. The minishell started and seemed to show a nice clean UFS volume, so I attempted to install the package sets (copied to the disk earlier). After a few files from the first install set were written, however, the entire system locked up.

Fourth attempt: maybe it didn't like the second target, and maybe I had to take a break to calm down at this point, maybe. I removed the second target and repartitioned the now single 2GB SCSI target into a Mac, miniroot, and full root and swap partitions. This time, Mkfs refused to newfs the new miniroot no matter what the geometry was set to and no matter how I formatted the partition, and I wasn't in any mood to mess around with block and fragment sizes.

Fifth attempt: screw the miniroot and screw Apple and screw the horse Jean-Louis Gassee rode in on and all his children with a rusty razorblade sideways. I might also have scared the cat around this time. I repartitioned again back to a MacOS, swap and root partition, and Mkfs on the IIci grudgingly newfsed that. I was able to mount this partition on the Q605 and restore from the last good backup. It mounted, unmounted and fscked fine. I then moved it over to the IIci, where it booted the Mac OS partition, started the NetBSD booter and immediately panicked.

At this point I had exceeded my patience and blood pressure, and there was also a non-zero probability the backup itself was bad. There was really only one way to find out. As it happens, I had bought a stack of new-old-stock Silicon Graphics-branded Quantum Atlas II drives to rebuild my SGI Indigo², so I pulled one of those out of the sealed anti-static wrap, jumpered it to spin up immediately, low-level formatted, partitioned and Mkfsed it on the IIci, connected it to the Q605 and restored from the tar backup. It mounted, unmounted and fscked fine. I then moved it over to the IIci, where it booted the Mac OS partition, started the NetBSD booter and ... the OS came up and, after I was satisfied with some cursory checks in single-user, uneventfully booted into multi-user. The backup was fine and the system was restored.

Since the Atlas II is a 7200rpm drive (the Fireball SE it replaced was 5400rpm) and runs a little hot, I took the lid off and placed it sideways to let it vent a bit better.

As a nice bonus the Atlas II is a noticeably faster drive, not only due to the faster rotation but also the larger cache. There's been no more weird errors, the full system backup overnight yesterday terminated completely and in record time, most system functions are faster overall and even some of the other weird glitchiness of the machine has disappeared. (The device on top of the case is the GSM terminal I use for sending text message commands to the main server.) I've allocated another one of the NOS Atlas drives to the IIci, and I'll find something else for rebuilding the Indigo². I think I've got some spare SCSI2SDs sitting around ...

So what's the moral of this story?

The moral of the story is that SCSI is awfully hard to get right and compatibility varies greatly on the controller, operating system and device. I've heard of NetBSD/mac68k running fine with SCSI2SD devices on other Quadras, and it appeared to work on my Quadra 605 as well (the infamous "LC" Quadra), but under no circumstances would NetBSD on my IIci want any truck with any hardware revision of it. Despite that, however, the IIci worked fine with it in MacOS except when the NetBSD installer took over (and presumably started using its own SCSI routines instead of the SCSI Manager) or tried to boot the actual operating system. It's possible later versions of NetBSD would have corrected this, but it's unlikely they would be corrected in the Mac-side Mkfs and Installer because of their standalone codebase, and the whole point of a solid-state disk emulator is to run the old OS your system has to run anyway. Of course, all of this assumes my IIci doesn't have some other hardware flaw that only manifests in this particular fashion, but that seems unlikely.

Thus, if you've got a mission-critical or otherwise irreplaceable machine with a dying hard drive, maybe your preservation strategy should simply be to find it another spinning disk. SCSI2SD is perfectly fine for hobbyist applications, and probably works well enough in typical environments, but it's not a panacea for every situation and the further you get away from frequent usage cases the less likely it's been tested in those circumstances. If it doesn't work or you don't want to risk it, you're going to need that spare drive after all.

There's also the issue of how insane it is to use almost 30-year-old hardware for essential services (the IIci was introduced in 1989), but anyone who brings that up in the comments on this particular blog gets to stand in the corner and think about what they just said.