Friday, June 28, 2019

And now for something completely different: NetBSD on the last G4 Mac mini (and making the kernel power failure proof)

(First, as a public service message, if you're running Linux on a G5 you may wish to update the kernel.)

I'm a big fan of NetBSD. I've run it since 2000 on a Mac IIci (of course it's still running it) and I ran it for several years on a Power Mac 7300 with a G3 card which was the second incarnation of the Floodgap gopher server. Today I also still run it on a MIPS-based Cobalt RaQ 2 and an HP Jornada 690. I think NetBSD is a better match for smaller or underpowered systems than current-day Linux, and is fairly easy to harden and keep secure even though none of these systems are exposed to the outside world.

Recently I had a need to set up a bridge system that would be fast enough to connect two networks and I happened to have two of the "secret" last-of-the-line 1.5GHz G4 Mac minis sitting on the shelf doing nothing. Yes, they're probably outclassed by later Raspberry Pi models, but I don't have to buy anything and I like putting old hardware to good use. So here it is, doing serious business, with the total outlay being the cost of one weekend afternoon:

NetBSD/macppc is a fairly mature port, but that doesn't mean it doesn't have bugs. And oddly there do seem to still be some in the install process, at least of the 8.1 release I used, on this last and mightiest of the PowerPC miniatures. Still, once it got set up it's been working great since, so here's a few pointers on getting the 1.5 mini (and current Power Macs generally) running as little NetBSD servers. As most of my readers are Power Mac users and likely to have multiple Power Macs that can aid this process, I will orient this guide to them with some sidebar notes for people trying to pull the mini up all by itself. This machine is configured with 1GB of RAM, the standard 2.5" PATA spinning disk and optical drive, USB 2.0, FireWire 400, Bluetooth and WiFi, using the onboard Radeon 9200 GPU as the console.

The working configuration, hit upon by Sevan Janiyan, is to have an HFS+ partition with the bootloader (called ofwboot.xcf) and the kernel, and then a separate partition for NetBSD's root volume. For some reason the mini goes berserk when trying to boot from a kernel on a NetBSD native partition, but works fine from an HFS+ one. Unfortunately, since the NetBSD installer cannot actually initialize an HFS+ volume, you'll need to do some of the partitioning work in Mac OS X, copy the files there, and then finish the rest. There's a couple ways of skinning that cat, but for many of you this means you'll need not only the NetBSD installer CD, but also a bootable copy of Mac OS X either on disc (i.e., an installer) or a bootable external drive, and some means of copying files.

And that brings us to our first pro tip: the G4 Mac minis had absolutely crap optical drives that would fail if you looked at them crossways. This drive was no exception; it would read commercially pressed CDs but only certain media types of burned CDs and wouldn't read any DVD at all. That means it wouldn't boot from my usual OS X Tiger 10.4.6 install DVD, and the last generation of G4 minis require 10.4, so none of my previous OS X CDs would work.

As it happens, the minimum requirement for the G4/1.5 minis is not actually 10.4.2, yet another Apple lie; it's actually 10.4.0 (though note that some devices like Bluetooth may not work properly). This is fortunate because 10.4.0 was available in CD format and I was able to boot Disk Utility off that Tiger CD instead. Your other option is to bring up the mini in Target Disk Mode (connect over FireWire; hold T down as you turn the mini on until you see a yellow logo on a blue background) from another Power Mac and do the formatting there. In fact, we'll be using Target Disk Mode in a minute, but here I just booted from the CD instead.

In Disk Utility (whether you're doing this on the machine from the Tiger installer or on another machine over FireWire), wipe the mini's current partition scheme and create two new partitions. The first is your HFS+ volume for booting. This particular machine will only run NetBSD, so I made it 512MB to have enough room for multiple kernels and for other files I might need, but if you want a dual-boot system you can make this larger. The second partition will be for NetBSD; I allocated everything else and created it as a UFS ("UNIX File System") partition, though we will divvy it up later. The formatting scheme should look more or less like these screenshots. Shut down the mini when you're done.

Now we boot the NetBSD installer. Bring up the machine in OpenFirmware mode -- all New World Macs use OpenFirmware 3 -- by holding down Command-Option-O-F while powering it on (I advise doing this from a directly-attached USB keyboard). This will bring up the OpenFirmware interface. When you are commanded to do so, release the keys and you will drop to the famous ok prompt. If you're lucky and the evil spirits in your optical drive have been placated by an offering of peanut M&Ms and a young maiden at midnight, you can simply insert the NetBSD install disc and type

boot cd:,\ofwboot.xcf netbsd.macppc

Note the backslash, not a forward slash! If this works properly, then the screen will go black (you don't go back) and enter the Installer proper.

If you get weird errors or OpenFirmware complains the disc is not readable, the optical drive is probably whacked. My drive wouldn't read burned Fujifilm CD-R media (that everything else did), but would read burned Maxell media. If you can't even control for that, you may be able to connect a FireWire CD/DVD reader and boot from it instead. The command would be "something like"

boot fw/node/sbp-2/disk:,\ofwboot.xcf netbsd.macppc

If this didn't work, you may need to snoop around the OpenFirmware device tree to figure out where the device is actually attached, though this should basically work for the G4 mini's single port. Alternatively, you could also try a USB CD-ROM drive, or dding the install image to a USB drive on another computer and booting the mini from that, but the boot string will vary based on which port you connect it to (use dev usb0 and ls to show everything under that port, then dev usb1, etc.). Make sure it is directly connected to the mini. Once you find a device that shows a disk, then "something like" this will work (let's say it was found under usb1):

boot usb1/disk:,\ofwboot.xcf netbsd.macppc

If even that won't work, there are some other ways like netbooting, but this is rather more complicated and I won't talk about it here. Or you could actually fix the drive, I guess ...

When the Installer starts up, choose the option to drop to the shell when it is offered. We will now finish the partitioning from the NetBSD side; we do not use the Installer's built-in partition tool as it will run out of memory. At the shell prompt, type

pdisk /dev/wd0c

When it asks you for a command, type a capital letter P and press RETURN. This will print out the current partition map, which if your mini is similar to mine, should show 4 partitions: the Apple partition map itself, followed by the HFS+ partition, and then by a tiny Apple_Boot partition that is made whenever a UFS volume appears to be the boot volume. (Silly Mac OS X.) You can remove it if you want, but this seemed like more trouble than it was worth for a measly 8.5 megabytes. After that is the space for NetBSD. On my G4 mini, this was partition number 4. Delete this partition by typing a lower-case d, press RETURN, and type 4. Be sure of this number! I will use it in the examples below.

First we will formally create the swap. This is done with the capital letter C command (as shown in the screenshot). Indicate the first block is 4p (i.e., starting at partition 4), for 4194304 blocks (2GB), type Apple_UNIX_SVR2 (don't forget the underscores!), and slice b.

Next is the actual NetBSD root: capital letter C, then saying the first block was 5p (i.e., starting at partition 5, the unallocated section), soaking up the rest of the blocks (however many you see listed under Apple_Free), type Apple_UNIX_SVR2 (don't forget the underscores!), and slice a.

If you did all this right, your screen should look more or less like this:

Verify the partition map one more time with the capital letter P command, then write it out with lower-case w, answering y(es), and then quit with lower-case q. At the shell prompt, return to the installer by typing sysinst and when asked, indicate you will "Use existing partition sizes." The installer will then install the appropriate packages and you can do the initial setup for your clock, the root password, etc. When this is all done, reboot your mini with the left mouse button held down; it will eject the CD (and fail to find a boot volume if you do not have an OS X installation). Shut down the mini.

Before the mini will boot NetBSD, we must copy the kernel and the bootloader to the HFS+ partition. This is where Target Disk Mode comes in handy, because you can just copy directly. Here is my iBook G4 copying a custom kernel (more in a moment):

On the iBook G4, I put in the NetBSD install CD and copied off ofwboot.xcf and netbsd-GENERIC.gz, or you can download them from here and here. They should be copied to the root of the mini's HFS+ volume for the command below to work. For good measure I also uncompressed the gzipped kernel as a failsafe and put a copy of the installation kernel there too, though this isn't necessary. Once the files are copied, eject the mini's drive on the FireWire machine, unplug the FireWire and power the mini off.

If you don't have another Mac around that can talk to the mini over FireWire, you can do this from NetBSD itself, but it's a bit more involved.

Either way, re-enter OpenFirmware with Cmd-Opt-O-F while powering it back up. It's time to boot your new NetBSD machine.

You can see from the screenshot here that the HFS+ volume is considered partition 2, as we left it in pdisk. That means your boot string is

boot hd:,\ofwboot.xcf hd:2/netbsd-GENERIC.gz

Yes, the path to ofwboot still has a backslash, but the argument to ofwboot actually needs a forward slash. NetBSD will start immediately.

There are several minor and one rather obnoxious bug with NetBSD's current support. You will notice a few strange messages on startup as part of the huge mass of text:

oea_startup: failed to allocate DEAD ZONE: error=12
pmu0: power-mgt not configured
pmu0: pmu-pwm-fans not configured
WARNING: 3 errors while detecting hardware; check system log.
bwi0: firmware_open failed on v3/ucode5.fw

I don't know what the first three are, but they appear to be harmless, and appear in many otherwise working dmesg archives (see also this report). The summary WARNING thus can also be politely ignored.

However, the last message is rather obnoxious. Per Sevan the built-in Broadcom WiFi in the Mac mini (detected as bwi0) doesn't work right in NetBSD with more than 512MB of memory, which I refuse to downgrade to, and NetBSD doesn't come with the firmware anyway. Even if you copy it off some other system that does, you won't be able to bring the interface up in the configuration here (you'll just see weird errors about wrong firmware version, etc.).

Since this machine is a bridge and sometimes needs to connect to a test WiFi, I went with a USB WiFi dongle instead (I also use a USB dongle when bridging Ethernet to Ethernet, but pretty much any Ethernet-USB dongle will work too). The one I had on the shelf that I'd bought for something else and then forgot about was a Belkin Wireless G. They sell a number of chipsets under this name, but the model F5D7050 I have here is based on a Ralink RT2501USB chipset that NetBSD sees as rum0, and works fine with wpa_supplicant.

Last but not least was making it "failsafe," with a solid power supply and making it autostarting. Although the G4 mini came with an 85W power supply, I stole the 110W from my 2007 Intel mini and used that so it wouldn't run anywhere near the PSU's capacity and hopefully lengthen its lifetime. As it turns out, this may not have been a problem anyway; most of the time this system is using just 21W on the Kill-A-Watt, maybe 40ish when it's booting.

To autostart NetBSD, ordinarily you would go into OpenFirmware and set boot-device to the bootloader and boot-file to the kernel, as the picture below shows.

However, you'll end up with a black screen or at minimum no console at all on an OpenFirmware 3 system if that's all you do. The magic sauce is to emit some text to the screen before loading the bootloader. Thus, the OpenFirmware settings are (each setenv command is one line):

setenv auto-boot? true
setenv boot-device hd:,\ofwboot.xcf
setenv boot-file hd:2/netbsd-GENERIC.gz (note that I used a different kernel in the screenshot: more in a second)
setenv boot-command ." hi there" cr " screen" output boot
reset-all

The boot-command spacing is especially critical. There is a space after the ." and the quote mark before screen" and after cr is also separated by spaces. The reset-all just tells OpenFirmware to write those settings to NVRAM. If you zap the mini's PRAM with Command-Option-P-R later, you may need to re-enter these.

In this configuration your mini will now start NetBSD automatically when it's turned on (just hold down Command-Option-O-F when starting it up to abort to OpenFirmware). However, this won't bring the machine up automatically after a power failure. While FreeBSD allows starting up after a power failure, this code apparently never made it over to NetBSD. Happily, supporting it merely requires a relatively simple kernel hack. Based on the FreeBSD pmu(4) driver, I created a patch that will automatically reboot any PMU-based NetBSD Power Mac after a power failure.

You should be comfortable with compiling your own kernels in NetBSD; not only is it just good to do for auditing purposes, but you can slim the kernel down substantially or enable other less common features. It's especially easy for NetBSD because all the tools to build it come with a standard installation. All you need to do is download the source and run the build process.

To use this patch, download the source to your home directory on the NetBSD box (you want syssrc.tgz) and download the patch and have it in your home directory as pmu.diff. If you don't have a working curl on your install yet (pkg_add curl, pkg_add mozilla-rootcerts, mozilla-rootcerts install), you may want to download it somewhere else and use scp, sftp or ftp to retrieve it. Then, adjusting as necessary for username and path,

cd /
tar zxf ~/syssrc.tgz
cd /usr/src/sys/arch/macppc/dev
patch -p0 < ~/pmu.diff

Then follow the instructions to make the kernel. I have a pre-built one of 8.1-GENERIC (I call it POWERON) on the gopher server, but you should really roll your own so that you get security fixes, since I may only maintain that kernel intermittently. That build is the one I'm using on the machine currently and on the screenshot above. With this custom kernel installed, when the power is abruptly cut while the machine is powered up it will automatically reboot when power is reapplied, just as the analogous option does in Mac OS X. Copy it to the HFS+ partition and remember to change boot-file to point to it once you've confirmed it works.

Overall, I think the G4 mini makes a fine little server. I wouldn't use it as a client except in Mac OS X itself, and I am forced to admit that even that is becoming less practical these days. But as a little machine to do important back-office tasks and do so reliably, I think NetBSD on the mini is a good choice. Once all the kinks with the installation got ironed out, so far it's been solid and performant especially considering this machine is about 13 years old (though I'm happy with its performance even on thirty-year-old machines). Rather than buying something new, if your needs are small it's probable you've got some old machine around that could do those tasks instead of oozing toxins from its circuit board into a waste dump in Rwanda. And since I had two on the shelf, it has an instant spare. I'll probably be usefully running it for as long as I've run my other NetBSD systems, and that's the highest compliment I think I can pay it.

Saturday, June 22, 2019

TenFourFox FPR15b1 available

TenFourFox Feature Parity Release 15 beta 1 is now available (downloads, hashes, release notes).

In honour of New Coke's temporary return to the market (by the way, I say it tastes like Pepsi and my father says it tastes like RC), I failed again with this release to get some sort of async/await support off the ground, and we are still plagued by issue 533. The second should be possible to fix, but I don't know exactly what's wrong. The first is not possible to fix without major changes because it reaches up into the browser event loop, but should be still able to get parsing and thus enable at least partial functionality from the sites that depend on it. That part didn't work either. A smaller hack, though, did make it into this release with test changes. Its semantics aren't quite right, but they're good enough for what requires it and does fix some parts of Github and other sites.

However, there are some other feature improvements, including expanded blocking of cryptominers when basic adblock is enabled (from the same list Mozilla uses for enhanced privacy in mainstream Firefox), and updated internationalization support with upgraded timezones and locales such as the new Japanese Reiwa era (for fun, look at Is it Reiwa yet? in FPR14.1 before you download FPR15b1). The usual maintenance and security fixes are (will be) also included (in final). In the meantime, I'm going to take a different pass at the async/await problem for FPR16. If even that doesn't work, we'll have to see where we're at then for parity purposes, since while the majority of websites still work well in TenFourFox's heavily patched-up engine there are an increasing number of major ones that don't. It's hard to maintain a browser engine on your own. :(

Meanwhile, if you'd like the next generation of PowerPC but couldn't afford a Talos II, maybe you can afford a Blackbird. Here's what I thought of it. (See also the followup.)

Tuesday, June 18, 2019

Stand by for FPR14 SPR1 chemspill

Mozilla has shipped a fix for MFSA2019-18 in Firefox 67.0.3 and 60.7.1. This exploit has been detected in the wild, and while my analysis indicates it would require a PowerPC-specific attack to be exploitable in official TenFourFox builds (the Intel versions may be directly exploited, however), it could probably cause drive-by crashes and we should therefore ship an urgent fix as well. The chemspill is currently undergoing confidence tests and I'm shooting to release builds before the weekend. For builders, the only change in FPR14 SPR1 is the patch for bug 1544386, which I will be pushing to the repo just as soon as I have confirmed the fix causes no regressions.

This chemspill also holds up the FPR15 beta which was actually scheduled for today. Unfortunately, the big JavaScript update I've been trying to make for the last couple cycles also ran aground and will not be in FPR15 either. There is a smaller one and some other improvements, so this is not an empty release, but I'll talk more about that in a few days.

Tuesday, June 4, 2019

Make Cheese Grating Great Again

The Mac Pro, like New Coke, is back. In a miniature cheese grater you could make a mean quesadilla with. A million Power Mac G5s and O.G. Mac Pros are singing out, "we told you so! We told you so!"

But if you're going to buy one of these things (starting at $6000), you don't get to complain how much a Talos II costs.