Long life computing (plus: Quad G5 CPU swapping for dummies)

We are now at the point where the youngest quad G5 is almost eight years old, and TenFourFox works on any Mac that can run 10.4 and has sufficient memory, meaning theoretically even a beige 9500 with a G3 card and a full RAM loadout could be out there (and if there is, I'd love to hear about it) pushing 19 years young.

I'm committed to getting a full useful decade out of my quad G5 systems, and frankly I don't see any reason to stop there. The Quad handles all my tasks except taxes and Android development (I use an Intel mini for Eclipse and TurboTax), including Photoshop, Microsoft Office and HD video; I'd have to rebuy all that just for the dubious privilege of using an Intel Mac. When it gets to the point that TenFourFox's rendering core can't be evolved further, then I'll just throw a Linux box in the server room and run a browser over remote X11. Then I can have my cake and eat it too.

Like me, I imagine most people reading this blog will want to keep their vintage Power Macs operating for a long time as well, especially if you use your Power Mac as your daily driver like me. There are some general things I do with all of my regular use systems (my daily drivers are my Quad G5, a 1GHz iMac G4 and an iBook/G4 12" 1.33GHz, all running 10.4) and these suggestions should help you also. Plus, those of us rocking liquid-cooled G5s are probably starting to notice that the fans are coming on harder, so we're going to talk about servicing the Quad G5 LCS at the end.

Software security. If you're running TenFourFox, you're already ahead of the curve. Congratulations. Here are other security issues to consider with old versions of Mac OS X (as well as our note about OS X's internal SSL).

Save a port, ride an extension. Wait, what? Logic boards and daughterboards are expensive and precious as well as a pain in the rear to fix, and most ports are directly soldered to the logic board, so when a port wears out it can be an expensive and time-consuming swap job. For USB and FireWire, especially for the front ports on my G5 which are constantly being connected and disconnected from my mobile phone and HD video camera, I connected extensions to them and I plug the camera and phone and etc. into the extension cables; I'd rather be replacing a $5 or so extension cable than wrecking my front port daughterboard. Similarly, my Bluetooth dongle connects with its own extension cable (though this is partially so I can find the damn thing).

Rotate your hard drives. Virtually all of our Power Macs are beyond their crib death stage and the next likely failing device will be spinning hard disks. Some of you have solved this problem with solid-state drives, which are a decent if pricey upgrade and certainly immune to mechanical if not electronic failure, but not even the SATA bus in a G5 will get anywhere near the maximum throughput possible; plus, no version of PPC OS X supports TRIM, which means write speed will gradually degenerate. (Linux users might get better benefit.) In my case, I have a stock of SATA I and II spinning drives which I bought new or NOS, and every four or five years I clone the disks (boot off the Tiger DVD and use Disk Utility) to a new set and swap them out. I'm using WD Caviar Blacks which are good solid 7200rpm drives; I did a swap in 2010, my next swap is due for 2015 and I have enough disks in stock to do one more swap in 2020. The swapped out drives are still working, of course, and if the G5 is still trucking in 2025 I might start a more rapid rotation schedule with those old disks depending on how heavily it's being used.

If you're on a system using the older PATA drives, stock up now; NOS stocks are disappearing -- or find something like a Sonnet PCI SATA card, which is bootable and works fine with OS 9 and OS X.

A common question is whether SATA III (6.0Gbps) drives are compatible with G5s and PCI SATA cards, and the answer is, of course, it depends. All Power Mac compatible SATA controllers I have encountered were SATA I (1.5Gbps), including the ones in all the G5s up through the Quad, and SATA II (3.0Gbps) drives work fine with SATA I. If the SATA III drive can be jumpered to run at SATA II speeds, it will work just fine; if it does not have a SATA II jumper, it depends on the drive electronics. Some users have reported that the SATA III Hitachi Deskstars work fine with SATA I, but many SATA III SSDs will not and this can be an expensive thing to explain to vendor support.

Speaking of spare hard disks ...

Have spare RAM, spare cards and spare parts. Consider a body double. Get used to doing basic repair on your machine. Besides hard disks, RAM ages too; both my iMac G4 and my quad G5 blew DIMM sticks around the same time. The good news is that every New World Mac uses industry-standard RAM and I was even able to find brand new PC100 sticks recently for my old Sawtooth G4 file server on Amazon, 14 years after it was taken off the market. RAM is cheap. Have a stock on hand. Beige Mac RAM can often be found from used Mac vendors like MemoryX.

Stock up on PRAM batteries. Later machines like the Power Mac G5 use standard coin cells but earlier G4 systems used 1/2AA 3.6V lithium batteries which may need to be specially ordered.

Another frequent consumable is video cards on those systems with discrete video and PCI/AGP slots. Fans are often the culprit -- the nVidia 7800GT's fan quit in my G5 a year or so ago when I was out of town and I returned to discover the GPU was at a finger-searing 105 C. Amazingly, it was still working and I put a new CPU cooler on it. I have a stock of a couple spare 7800GTs since they're a decent video card that doesn't require two slots in my G5.

Regular PCI cards die as well, though admittedly not as frequently. One of my PCIe FireWire/USB cards is now refusing to recognize FireWire devices plugged into it. Time to swap that out; it's just not worth doing any sort of component level repair on these things anymore. I have a couple in stock too. Sonnet still sells many of these peripheral cards new.

Certain machines have notoriously high frequency failures in certain components. Besides the issues with many early Power Mac G5 models, which I'll talk about in a moment, the MDD G4s are infamous for having iffy power supplies. Since my MDD is my OS 9 workstation of choice, I have three MDD power supplies in stock (I've already had to replace the power supply twice on it since I bought it new in 2002, not counting the power supply replacement under Apple's recall).

If you are unlucky to have a model with a history of logic board failures, take heart in that logic board death is usually an early finding and most of the surviving Power Macs will have a better track record. However, it's often better to just keep an entire body double on hand for your mission critical systems -- a(n as much as possibly) identically configured system, waiting in storage ready to swap in when you need it. This gets you spares for everything, even the logic board, power supply and case; if you have a major electronics failure, just move everything to the body double and work on the original. I kept a spare Apple Network Server 700 in the closet to replace my Apple Network Server 500, which it had to do for a period of a few weeks while I debugged a hardware issue, and my body double quad G5 is now my current quad and the original one is now the spare (with a new CPU and LCS). I also have body doubles for my iBook, my iMac G4 and my Sawtooth G4 and MDD G4.

Prices on Power Macs are starting to bottom out as they become rarer and more interesting as collector's items. This may well be the last chance to stock up on them at a decent cost before it becomes a seller's market.

Have a strategy to recover from catastrophic failure. Besides my body double and spare disks, I have bootable Power Mac compatible discs for Alsoft DiskWarrior, Prosoft Data Rescue 3 and of course retail Tiger 10.4.6, which will boot any compatible Power Mac including the Quad G5. Alsoft even still sells a DiskWarrior for OS 9 systems, and Data Rescue 3 still works with G4 and G5 computers. Support companies that still support the Power Mac; please don't pirate. I'm not an affiliate of either company, merely a satisfied customer.

Throttle your CPU: heat (and heat caused by higher power consumption) is the enemy. Heat diminishes the lifetime of components. Heat dries up thermal compound on the heatsinks. Run your Mac in reduced mode if at all possible, particularly for G5 systems which run comparatively hot to begin with -- my G5 is usually in Reduced mode, and I only flip it to Highest if I have a task I need done quicker or a slow-to-render web page. A tool like Fast and Slow (free) will do nicely; I like CPU Speed Menu, which is a little more configurable, but it's shareware and it doesn't seem its author supports it anymore.

Power Mac G5 Special Section: Swapping the Quad G5 CPU

I have long observed that there are only two G5 computers worth owning: the air-cooled PCI dual processor "DP" 2.3GHz (the later dual core "DC" PCIe 2.3 is also air-cooled, and about 5% faster depending on the benchmark, but has comparatively lower reliability scores), and the liquid-cooled Quad G5, the fastest Power Mac ever made and the one with the best liquid cooling system. The iMac G5s are a big bag of hurt to work on, the early G5s are not substantially faster than the G4s they replaced, and the others have logic board, power supply or liquid cooling leakage issues.

Now, again, by this time in the Power Mac's lifespan most early failures have already wound up in the landfill. Similarly, most of the liquid cooling systems that were going to leak probably already have. However, liquid cooling systems, like the automotive radiators the G5 systems are based on, wear out. There are small evapourative losses from the hoses carrying the coolant, which in these systems are water-based, and thermal paste also slowly loses its moisture and starts to crumble. The very last Quad G5 to roll off the production line is almost eight years old, and if it's still on its original CPU and heat sink it is very likely it's suffering from both these problems.

At this point we should probably talk about what thermal calibration actually involves. All G5 (a/k/a PowerPC 970) systems use fan software controlled by the operating system which needs to know how effective the current cooling system is. According to IBM's technical documentation on the 970, each CPU has temperature sensors and a calibration ROM which stores data about the sensors' set points and fan speeds. Thermal calibration does not actually calibrate the sensors; thermal calibration data for the on-chip thermal diodes is burned in at the factory (IBM calls this information part of the "fuse code data") and if these sensors are bad or incorrect the only solution is to replace the CPU. Instead, what it does is try to find the fan (and, on LCSes, pump) RPM ranges that will keep the CPU in a normal temperature range even under maximal power draw, and writes this data to the calibration ROM. (This is why losing your PRAM battery doesn't screw up your fans. By the way, the PMU/SMU, contrary to popular belief, does not manage the fans either. Resetting the PMU/SMU to fix a fan problem won't help and may scramble the PMU/SMU.)

The upshot is that whenever you do anything to change the cooling characteristics of the system, such as replacing or servicing the CPUs, you need to run thermal calibration again to make sure that the computer knows which RPMs and pump speeds will work. You can't "thermal calibrate" your way out of a failing CPU assembly -- the assembly either passes or it doesn't, and on some earlier versions of the calibration tool you ran the risk of corrupting the calibration ROM if you ran it repeatedly on an iffy unit. Thermal calibration is run from the Open Firmware diagnostic tools on the Apple Service Diagnostic disc. Most people will want to use 2.6.3 with both these models of G5, and the Quad requires it, but 2.5.8 will also work fine with the DP 2.3. To avoid Apple's wrath I won't link these disc images here, but they are not difficult to find. The ambient temperature in the room when you run calibration must be below 77 F (25 C).

A utility like Temperature Monitor or iStat Pro will be useful to know if you're getting close to the edge. I threw together a little tool to monitor your G5 temperature and (max) fan or pump speed in the menu bar. (On my G4 laptop, it didn't install in the menu bar correctly and I don't know why. If it doesn't appear to work, make sure the monitoring process isn't running alone in the background soaking up CPU cycles. I'll look into this later.) The menu bar tool displays the highest temp of the CPUs and the highest RPM of the non-fixed pumps or fans; you can also use it as a command line tool (see the page for how). On single pump LCSes, the same speed is reported twice.

At least on the Quad, ordinarily the intakes idle at 970rpm (ahaha), the exhausts and drive bay fan at 1000rpm, the backside fans at 1100rpm and the PCI slots at apparently a fixed 1560rpm. The LCS pumps, for their part, idle at 1250rpm. As heat increases the CPU fan speeds will increase, as well as the liquid cooling system pumps. Under load the fastest RPM in the system is usually the LCS pumps which are rev-limited by the computer to 3600rpm.

At idle, the CPUs should be below 50 C, especially in Reduced power mode, and the CPUs should be roughly within 10 C of each other or less. (In fact, earlier this spring, my G5 observed a 35 C core, the lowest it's been in years.) The CPUs should not exceed 70 C even under load on any core. If your Quad G5 is seeing core temperatures over 70 C even when the pump speed is pegged at 3600rpm, your system can no longer cool effectively and it's time to service the processor assembly. Resist the temptation to run thermal calibration because it probably won't help you for long if it helps you at all. Switch your G5 into Reduced mode if it wasn't already; assuming you're not much over the 70 C mark, you've still got some time to get all your parts together.

Servicing the DP 2.3 is a matter of unmounting the heatsink, applying new thermal compound (Shin-Etsu X23-7783D is the typical recommendation, though others will work) and putting the CPUs back. Fortunately, most DP 2.3 systems if well-taken care of won't even need that; do so only if your Mac has random sudden sleeping episodes or you see complaints about being "overtemp" in your OS X system log.

The Quad is a different story -- the easiest way, and the only way Apple ever allowed, is to just find a new CPU and heatsink assembly. There is a lot of confusion over which will work and what LCS the Quads came with. Most systems I've encountered have a so-called "version 1" Delphi assembly with a single pump. It looks like this:

For a period of time Apple used a so-called "version 2" manufactured by Cooligy with twin Delphi pumps; despite the apparently later revision the "version 2" LCS actually has the poorer track record. The Apple service manual for the Quad does not say anything about requiring a special logic board for the version 2 LCS, and they are comparatively unusual. I will not discuss them further here. Although Apple used Panasonic LCSes on some older G5s, they never used them on the Quads (even I got this wrong until I did my research). There appears to be no problem with replacing a version 2 LCS with a version 1 LCS; in fact, a version 1 may be the only one you can get anymore.

The version 1 CPU and heatsink should come as an item. If you are buying from an Apple Certified reseller, this is the only way Apple ever sold them. Do not buy them separately unless you know what you're doing, or you're totally crazy; you will have to reassemble them. The part number for the combined CPU and heatsink is 661-3729. When shopping around for these, don't pay more than a couple hundred dollars -- I now have three spares, and the most I spent for any one of them was $135. The magic words you want to hear from the reseller are "Apple Certified Refurbished" (or, less optimally, "passes thermal calibration"). If you don't hear these phrases from them, do not buy. Prefer the Certified Refurbished parts, because these came from Apple with new thermal paste and new coolant and they'll last for years more if you treat them right.

(If you can't buy a decent used CPU, all is not lost. There are ways to service the liquid cooling system and the CPUs by hand if you're willing to do a bit of grunt work and are comfortable with draining and replacing the coolant manually. Heikki Lindholm offered many great tips while I was writing this and I direct you to his excellent treatises on servicing the Quad G5 cooling system and pump assemblies.)

In addition to the CPU, you will also need:

• An X-acto knife or small flathead screwdriver to use as a pry tool.
• A Philips 0 screwdriver.
• A 3mm long-handle flathead hex driver (I used an Eklind part# 54930).
• A 4mm long-handle ballhead hex driver (I used a Bondhus part# 13160). Both are available from Amazon and most special-order tool retailers. The tool shaft needs to be at least 8" long; shorter ones will not be able to get to the screws.
• A grabber of some sort to go chasing screws; you can use a magnetic one as long as you keep away from the hard disks.
• A copy of the ASD for your G5; burn it to disc and have it handy (remember that ASD 2.6.3 requires a DVD).

Power down the Mac and unplug it. Let it cool down: not only is the radiator potentially hot enough to burn you, but working on the G5 logic board when it's hot may damage the logic board. Flip the lever on the back, remove the side panel and remove the air deflector, and put them aside. Pull the front fans straight out, in front of the radiator.

The processors are covered by a heat shield with a G5 logo. If you've never serviced the CPUs before (or Apple serviced them), this shield is locked in place by a little plastic pin on top. Pry this very tenacious pin out with the X-acto knife or small flathead screwdriver verrrrrry carefully so that you don't go damaging anything else. This will destroy the pin most likely, but all our Macs are out of warranty and you may need to do stuff with the processor again in the future. The heat shield will now become very loose and mobile. Shift the shield to the left and remove it. In front of the front radiator is a little plastic intake frame; rotate it clockwise from its mooring around the radiator so it can come free from its standoffs and pull it out.

Now it's time to get the old CPU assembly out. Put the computer flat on its side. If you don't do this, the screws will fall out when you remove them and end up in the power supply compartment, and that really sucks! Follow Apple's steps exactly and loosen/remove the screws exactly in numerical order (note: the captive float plate screws are not really captive and can come out, so be careful):

(The ballhead screws in the last step may have to be accessed at an angle; that's why they're ballheads and not flat.)

Once the screws are loosened and/or removed as directed, disconnect the black fan control linkage by pulling the two halves apart, and grab the centre of the metal frame. The CPU should lift straight up and out with very little force:

This is what you should see with the CPU out. Yes, it's dusty. Clean the dust out and in particular make sure there are no dust bunnies floating over where the CPUs attach (those arrays of little square pins).

It is particularly critical not to lose any screws: in particular, the screws attaching the CPUs to the logic board are actually an integral part of the power supply connection to the processors and the computer may not be able to even power up without them.

If the screws despite your best effort do get into the power supply compartment (and don't ask me how I have experience with this), you'll need to give them an escape path. Pull the CPU out and also remove the hard drives because you're going to have to shake the computer around. Unmoor the rear exhaust fan assembly by pressing in the tabs facing you so it can move freely and won't block any flying screws (you don't have to disconnect it unless you want to). Pry up the adhesive pad on the bottom of the partition between the CPUs and the power supply (just enough to get to the screws; don't toss it) and undo both Philips-head screws to remove the partition. Unplug the power cable near the PRAM battery and thread it free as you lift the partition. The partition is flexible and you may need to bend it a bit to get it out. Turn the G5 upside down and remove one or both of the screws on the underside. This will loosen the power supply so the screws can escape. Now it's time for calisthenics with your Quad:

Get all of the screws out! Do not leave any of them loose in the machine! I had to literally put my Quad over my head and shake it to get them loose enough to fish for with the grab. Once you've got them all back, replace the screws in the underside, put the partition back (rethreading the power connector through and plugging it back into the logic board), put the screws back in the partition, replace the adhesive pad and put the rear fan assembly back into its original position, making sure the tabs lock into their little slots.

Now get the new CPU straight up out of its box, lifting it up by its metal bridge (it should be stored such that the heatsink is on top, which uses gravity to maintain the seal between the heatsink and the actual processors). Do not lift it up by the radiators or heat pipes or you will bend them and reduce the assembly's cooling performance. Used assemblies may or may not come with caps over the processor connectors; remove them if they are present, and loosen the float plate screws on the new CPU assembly like you did in the first step of removing the old CPU. Next, put the screws in their holes before installing the assembly: it's much harder to put the screws in when the CPU is already in the case. Carefully align the CPU over the standoffs and put it straight down into the computer:

The CPU should go in nice and level. Press down on the bridge only to make sure the CPU is seated; don't press anywhere else. It should give slightly with a little thunk. Do not apply significant force or you will damage the CPU, the logic board or both.

With the CPU now in place, tighten the four captive ballhead screws (the ones in the last step you did when loosening the screws before pulling the old CPU out), going in order from top to bottom, then middle top and middle bottom. Do not overtighten -- tighten only up to mild resistance. If the ballhead screws are not engaging well, the CPU is probably not seated correctly: make sure the float screws are loose, carefully pull the CPU out and reseat it. You can also try tightening the top, middle top, bottom and middle bottom in a different order; this may enable the CPU to move around a bit into a more favourable position.

Once the ballhead screws are engaged and gently tightened:

• Tighten the remainder of the screws in the reverse order you loosened and/or removed them, finishing with tightening the captive float plate screws. Again: do not overtighten.
• Connect the black fan control cable.
• Put back the plastic radiator intake frame; guide its rear tab to the back around the processor standoffs and rotate it so that its clips grip the front radiator.
• Replace the heat shield only if you want to.
• Replace the front fans and put back the clear plastic deflector.

Connect your G5 and plug it in, cross your fingers, and hold down Opt-Cmd-O-F while turning it on from the power switch. If you did this right, one of the front red LEDs will briefly light and then turn off, the G5 will bong and you will enter the Open Firmware prompt. (If you don't get a bong, the CPU is not in place correctly or, horrors, is damaged. Loosen it and remove it. See if it will boot with the old CPU.) Release the keys, type eject cd to open the optical drive, put your ASD disk in, and close the optical drive. On my ASD 2.6.3 disk, I had to type boot cd:,\diags (yes, you need the colon, the comma and the backslash) to start the diagnostic program, the only time your G5 will ever "appear" to boot Mac OS 9. :) The fans will come on full blast; this is normal and the fans will remain loud during the entire diagnostic process.

After the diagnostic program finishes loading, select Thermal Calibration from the pull down menu. Make sure the room is below 77 F/25 C and that the front of the machine is not facing any heat source or heat exhaust. Once you begin thermal calibration on the new CPU assembly and it succeeds, you may cause issues with your Mac if you put the old failing CPU assembly back in later. This is the point of no return. If you're lucky, this is the result:

Congratulations. Now put the side panel back on, run the rest of the self-tests and keep on trucking.

(And marvel about how quiet it is now when you reboot into OS X.)

Meanwhile, I'm pulling 29 beta to transfer the changesets over for a TenFourFox 29 unstable release. Still a couple of moderate issues to fix, but it's getting there and is mostly in a state I think you'll be able to play with. More about that probably next week.