Tuesday, October 29, 2019

SourceForge download issues (and Github issues issues)

There are two high-priority problems currently affecting TenFourFox's download and development infrastructure. Please don't open any more Tenderapp tickets on these: I am painfully aware of them and am currently trying to devise workarounds, and the more tickets get opened the more time I spend redirecting people instead of actually working on fixes.

The first one is that the hack we use to relax JavaScript syntax to get Github working (somewhat) is now causing the browser to go into an infinite error loop on Github issue reports and certain other Github pages, presumably due to changes in code on their end. Unfortunately we use Github heavily for the wiki and issues tracker, so this is a major problem. The temporary workaround is, of course, a hack to relax JavaScript syntax even further. This hack is disgusting and breaks a lot of tests but is simple and does seem to work, so if I can't come up with something better it will be in FPR17. Most regular users won't be affected by this.

However, the one that is now starting to bite people is that SourceForge has upgraded its cipher suite to require a new cipher variant TenFourFox doesn't support. Unfortunately all that happens is links to download TenFourFox don't work; there is no error message, no warning and no explanation. That is a bug in and of itself but this leaves people dead in the water being pinged to install an update they can't access directly.

Naturally you can download the browser on another computer but this doesn't help users who only have a Power Mac. Fortunately, the good old TenFourFox Downloader works fine. As a temporary measure I have started offering the Downloader to all users from the TenFourFox main page, or you can get it from this Tenderapp issue. I will be restricting it to only Power Mac users regardless of browser very soon because it's not small and serves no purpose on Intel Macs (we don't support them) or other current browsers (they don't need it), but I want to test that the download gate works properly first and I'm not at my G5 right this second, so this will at least unblock people for the time being.

For regression and stability reasons I don't want to do a full NSPR/NSS update but I think I can hack this cipher in as we did with another one earlier. I will be speeding up my timetable for FPR17 so that people can test the changes (including some other site compatibility updates), but beta users are forewarned that you will need to use another computer to download the beta. Sorry about that.

Friday, October 18, 2019

TenFourFox FPR16 SPR1 available

TenFourFox Feature Parity Release "16.1" (SPR 1) is now available for testing (downloads, hashes, release notes). As noted, this is a pure security update and there are no user-facing changes; the big under-the-hood change of those is that we are now pulling entirely from 68ESR, including locale data, certificate roots and so forth. There is also a small update to the ATSUI font blacklist. Assuming no issues, it will go live Monday evening Pacific time as usual.

Saturday, October 12, 2019

Chrome users gloriously freed from obviously treacherous and unsafe uBlock Origin

Thank you, O Great Chrome Web Store, for saving us from the clearly hazardous, manifestly unscrupulous, overtly duplicitous uBlock Origin. Because, doubtlessly, this open-source ad-block extension by its very existence and nature could never "have a single purpose that is clear to users." I mean, it's an ad-blocker. Those are bad.

Really, this is an incredible own goal on Google's part. Although I won't resist the opportunity to rag on them, I also grudgingly admit that this is probably incompetence rather than malice and likely yet another instance of something falling through the cracks in Google's all-powerful, rarely examined automatic algorithms (though there is circumstantial evidence to the contrary). Having a human examine these choices costs money in engineering time, and frankly when the automated systems are misjudging something that will probably cost Google's ad business money as well, there's just no incentive to do anything about it. But it's a bad look, especially with how two-faced the policy on Manifest V3 has turned out to be and its effect on ad-blocker options for Chrome.

UPDATE: I hate always being right. Peter Kasting, a big wheel and original member of the Chrome team, escalated the issue and the extension is back, but for how long? And will it happen again? And what if you're not a squeaky enough wheel to gain enough attention to your plight?

It is important to note that this block is for Chrome rather than Chromium-based browsers (like Edge, Opera, Brave, etc.). That said, Chrome is clearly the one-ton gorilla, and Google doesn't like you sideloading extensions either. While Mozilla reviews extensions too, and there have been controversial rejections on their part, speaking as an add-on author of over a decade there is at least a human on the other end even if once in a while the human is a butthead. (A volunteer butthead, to be sure, but still a butthead.) So far I think they've reached a reasonable compromise between safety and user choice even if sometimes the efforts don't scale. On the other hand, Google clearly hasn't by any metric.

This is a good time to remind people who may not know that TenFourFox has built-in basic adblock, targeted at the JavaScript-based nuisances that are most pernicious on our older systems. It's not only an integral part of the browser but it's also actually written in C++, so it's faster than a JavaScript-based add-on and works at a much lower level. It can also be combined with Private Browsing and other adblocker add-ons for even more comprehensive protection.

You may have suspected by the relative lack of activity on this blog and at Github that there aren't going to be any new features in the next TenFourFox release, and you'd be right. Between my wife and I actually being in the same hemisphere for a couple weeks, an incredible amount of work at the dayjob and work on the POWER9 side for mainline Firefox I've just been too short-handed to do much development this cycle. It will instead be numbered FPR16 SPR1 with security patches only and I'll use the opportunity to change our upstream certificate source to 68ESR. Watch for it sometime next week.

Sunday, September 22, 2019

A quick note for 64-bit PowerPC Firefox builders

If you build Firefox on 64-bit Linux, *BSD, etc. for your G5, you may want to check out this Talospace article for an upcoming low-level fix especially as we need to ensure big-endian systems work fine with it. The problem never affected OS X Firefox for Power Macs because those builds were only ever 32-bit, and even TenFourFox is 32-bit through and through even on the G5 largely for reasons of Carbon compatibility which we need for some pieces of the widget code. Since this is syndicated on Planet Mozilla let me give a big thanks to Ted Campbell for figuring out the root cause, which turned out to be a long-standing problem I don't think anyone ever noticed before.

I have not decided what to land on TenFourFox FPR17 mostly because this fix took up a fair bit of time; it's possible FPR17 may be a security-only stopgap release. In a related vein, the recent shift to a 4-week cadence for future Firefox releases starting in January will unfortunately increase my workload and may change how I choose to roll out additional features generally. Build day on the G5 is, in fact, literally a day or sometimes close to two (with the G5 in Reduced performance to cut down on fan noise and power consumption it takes about 20 hours to generate all four CPU-optimized releases, plus another 6 hours to regenerate the debug build for development testing; if there are JavaScript changes, I usually kick off a round each on the debug, G4/7450 and G5 builds through the 20,000+ item test suite and this adds another ten hours). Although the build and test process is about 2/3rds automated, it still needs intervention if it goes awry; plus, uploading to SourceForge is currently a manual process, and of course the documentation doesn't write itself. I don't have any easy means of cross-building TenFourFox on the Talos II (which, by the way, with dual 4-core CPUs for 32 threads builds Firefox in about half an hour), so I need to figure out how to balance this additional time requirement with the time I personally have available. While I do intend to continue supporting TenFourFox for those occasions I need to use a Power Mac, this Talos II is undeniably my daily driver, and fixing bugs in the mainline Firefox build I use every day is unavoidably a higher priority.

Monday, September 2, 2019

The deformed yet thoughtful offspring of AppleScript and Greasemonkey

Ah, AppleScript. I can't be the only person who's thinking Apple plans to replace AppleScript with Swift because it's not new and sexy anymore. And it certainly has its many rough edges and Apple really hasn't done much to improve this, which are clear signs it's headed for a room-temperature feet-first exit.

But, hey! If you're using TenFourFox, you're immune to Apple's latest self-stimulatory bright ideas. And while I'm trying to make progress on TenFourFox's various deficiencies, you still have the power to make sites work the way you want thanks to TenFourFox's AppleScript-to-JavaScript "bridge." The bridge lets you run JavaScript within the page and sample or expose data back to AppleScript. With AppleScript's other great powers, like even running arbitrary shell scripts, you can connect TenFourFox to anything else on the other end with AppleScript.

Here's a trivial example. Go to any Github wiki page, like, I dunno, the one for TenFourFox's AppleScript support. If there's a link there for more wiki entries, go ahead and click on it. It doesn't work (because of issue 521). Let's fix that!

You can either cut and paste the below examples directly into Script Editor and click the Run button to run them, or you can cut and paste them into a text file and run them from the command line with osascript filename, or you can be a totally lazy git and just download them from SourceForge. Unzip them and double click the file to open them in Script Editor.

In the below examples, change TenFourFoxWhatever to the name of your TenFourFox executable (TenFourFoxG5, etc.). Every line highlighted in the same colour is a continuous line. Don't forget the double quotes!

Here's the script for Github's wiki.

tell application "TenFourFoxWhatever"
    tell current tab of front browser window to run JavaScript "
 
    // Don't run if not on github wiki.
    if (!window.location.href.match(/\\/\\/github.com\\//i) ||
        !window.location.href.match(/\\/wiki\\//)) {
        window.alert('not a Github wiki page');
        return;
    }
 
    // Display the hidden links
    let nwiki=document.getElementById('wiki-pages-box').getElementsByClassName('wiki-more-pages');
    while (nwiki.length > 0) {
        nwiki.item(0).classList.remove('wiki-more-pages');
    }
 
    // Remove the 'more pages' link (should be only one)
    let jwiki=document.getElementById('wiki-pages-box').getElementsByClassName('wiki-more-pages-link');
    if (jwiki.length > 0)
        jwiki.item(0).style.display = 'none';
 
    "
end tell

Now, have the current tab on any Github wiki page. Run the script. Poof! More links! (If you run it on a page that isn't Github, it will give you an error box.)

Most of you didn't care about that. Some of you use your Power Macs for extremely serious business like YouTube videos. I ain't judging. Let me help you get rid of the crap, starting with Weird Al's anthem to alumin(i)um foil.

With comments in the five figures from every egoist fruitbat on the interwebs with an opinion on Weird Al, that's gonna take your poor Power Mac a while to process. Plus all those suggested videos! Let's make those go away!

tell application "TenFourFoxWhatever"
    tell current tab of front browser window to run JavaScript "
 
    // Don't run if not on youtube.
    if (!window.location.href.match(/\\.youtube.com\\//i) ||
        !window.location.href.match(/\\/watch/)) {
        window.alert('not a YouTube video page');
        return;
    }
 
    // Remove secondary column and comments.
    // Wrap in try blocks in case the elements don't exist yet.
    try {
        document.getElementById('secondary').innerHTML = '';
        document.getElementById('secondary').style.display = 'none';
    } catch(e) { }
    try {
        document.getElementById('comments').innerHTML = '';
        document.getElementById('comments').style.display = 'none';
    } catch(e) { }
 
    "
end tell

This script not only makes those areas invisible, it even nukes their internal contents. This persists from video to video unless you reload the page.

As an interesting side effect, you'll notice that the video area scaled to meet the new width of the browser, but the actual video didn't. I consider this a feature rather than a bug because the browser isn't trying to enable a higher-resolution stream or scale up the video for display, so the video "just plays better." Just make sure you keep the mouse pointer out of this larger area or the browser will now have to composite the playback controls.

You can add things to a page, too, instead of just taking things away. Issue 533 has been one of our long-playing issues which has been a particular challenge because it requires a large parser update to fix. Fortunately, Webpack has been moving away from uglify and as sites upgrade their support (Citibank recently did so), this problem should start disappearing. Unfortunately UPS doesn't upgrade as diligently, so right now you can't track packages with TenFourFox from the web interface; you just get this:

Let's fix it! This script is a little longer, so you will need to download it. Here are the major guts though:

    // Attempt to extract tracking number.
    let results = window.location.href.match(
        /^https:..www.ups.com.track.loc=([a-zA-Z_]+)\\&tracknum=([a-zA-Z0-9]+)\\&/
    );
    if (!results || results.length != 3) {
        window.alert('Unable to determine UPS tracking number.');
        return;
    }
 
    // Construct payload.
    let locale = results[1];
    let tn = results[2];
    let payload = JSON.stringify({'Locale':locale,'TrackingNumber':[tn]});

A bit of snooping on UPS's site from the Network tab in Firefox 69 on my Talos II shows that it uses an internal JSON API. We can inject script to complete the request that TenFourFox can't yet make. Best of all, it will look to UPS like it's coming from inside the house the browser ... because it is. Even the cookies are passed. When we get the JSON response back, we can process that and display it:

    // For each element, display delivery date and status.
    // You can add more fields here from the JSON.
    output.innerHTML = '';
    data.trackDetails.forEach(function(pkg) {
        output.innerHTML += (pkg.trackingNumber+' '+
            pkg.packageStatus+' '+
            pkg.scheduledDeliveryDate+'<p>');
        });
    }

So we just hit Run on the script, and ...

... my package arrives tomorrow.

Some of you will remember a related concept in Classilla called stelae, which were scraps of JavaScript that would automatically execute when you browse to a site the stela covers. I chose not to implement those in precisely that fashion in TenFourFox because the check imposes its own overhead on the browser on every request, and manipulating the DOM is a lot easier (and in TenFourFox, a lot more profitable) than manipulating the actual raw HTML and CSS that a stela does (and Classilla, because of its poorer DOM support, must). Plus, by being AppleScript, you can run them from anywhere at any time (even from the command line), including the very-convenient ever-present-if-you-enable-it Script menu, and they run only when you actually run them.

The next question I bet some of you will ask is, that's all fine for you because you're a super genius™, but how do us hoi polloi know the magic JavaScript incantations to chant? I wrote these examples to give you general templates. If you want to make a portion of the page disappear, you can work with the YouTube example and have a look with TenFourFox's built-in Inspector to find the ID or class of the element to nuke. Then, getElementById('x') will find the element with id="x", or getElementsByClassName('y') will find all elements with y in their class="..." (see the Github example). Make those changes and you can basically make it work. Remove the block limiting it to certain URLs if you don't care about it. If you do it wrong, look at the Browser Console window for the actual error message from JavaScript if you get an error back.

For adding functionality, though, this requires looking at what Firefox does on a later system. On my Talos II I had the Network tab in the Inspector open and ran a query for the tracking number and was able to see what came back, and then compared it with what TenFourFox was doing to find what was missing. I then simulated the missing request. This took about 15 minutes to do, granted given that I understood what was going on, but the script will still give you a template for how to do these kinds of specialized requests. (Be careful, though, about importing data from iffy sites that could be hacked or violating the same-origin policy. The script bridge has special privileges and assumes you know what you're doing.) Or, if you need more fields than the UPS script is providing, just look at the request the AppleScript sends and study the JSON object the response passes back, then add the missing fields you want to the block above. Tinker with the formatting. Sex it up a bit. It's your browser!

One last note. You will have noticed the scripts in the screen shot (and the ones you download) look a little different. That's because they use a little magic to figure out what TenFourFox you're actually running. It looks like this:

set tenfourfox to do shell script "ps x | perl -ne '/(TenFourFox[^.]+)\.app/ && print($1) && exit 0'"
if {tenfourfox starts with "TenFourFox"} then
    tell application tenfourfox
        tell «class pCTb» of front «class BWin» to «event 104FxrJS» given «class jscp»:"

This group of commands runs a quick script through Perl to find the first TenFourFox instance running (another reason to start TenFourFox before running foxboxes). However, because we dynamically decide the application we'll send AppleEvents to (i.e., "tell-by-variable"), the Script Editor doesn't have a dictionary available, so we have to actually provide the raw classes and events the dictionary would ordinarily map to. Otherwise it is exactly identical to tell current tab of front browser window to run JavaScript " and this is actually the true underlying AppleEvent that gets sent to TenFourFox. If TenFourFox isn't actually found, then we can give you a nice error message instead of the annoying "Where is ?" window that AppleScript will give you for orphaned events. Again, if you don't want to type these scripts in, grab them here.

No, I'm not interested in porting this to mainline Firefox, but the source code is in our tree if someone else wants to. At least until Apple decides that all other scripting languages than the One True Swift Language, even AppleScript, must die.

Saturday, August 31, 2019

TenFourFox FPR16 available

TenFourFox Feature Parity Release 16 final is now available for testing (downloads, hashes, release notes). This final version has a correctness fix to the VMX text fragment scanner found while upstreaming it to mainline Firefox for the Talos II, as well as minor outstanding security updates. Assuming no issues, it will become live on Monday afternoon-evening Pacific time (because I'm working on Labor Day).

Saturday, August 24, 2019

TenFourFox FPR16b1 available

TenFourFox Feature Parity Release 16 beta 1 is now available (downloads, hashes, release notes). In addition, the official FAQ has been updated, along with the tech notes.

FPR16 got delayed because I really tried very hard to make some progress on our two biggest JavaScript deficiencies, the infamous issues 521 (async and await) and 533 (this is undefined). Unfortunately, not only did I make little progress on either, but the speculative fix I tried for issue 533 turned out to be the patch that unsettled the optimized build and had to be backed out. There is some partial work on issue 521, though, including a fully working parser patch. The problem is plumbing this into the browser runtime which is ripe for all kinds of regressions and is not currently implemented (instead, for compatibility, async functions get turned into a bytecode of null throw null return, essentially making any call to an async function throw an exception because it wouldn't have worked in the first place).

This wouldn't seem very useful except that effectively what the whole shebang does is convert a compile-time error into a runtime warning, such that other functions that previously might not have been able to load because of the error can now be parsed and hopefully run. With luck this should improve the functionality of sites using these functions even if everything still doesn't fully work, as a down payment hopefully on a future implementation. It may not be technically possible but it's a start.

Which reminds me, and since this blog is syndicated on Planet Mozilla: hey, front end devs, if you don't have to minify your source, how about you don't? Issue 533, in fact, is entirely caused because uglify took some fast and loose shortcuts that barf on older parsers, and it is nearly impossible to unwind errors that occur in minified code (this is now changing as sites slowly update, so perhaps this will be self-limited in the end, but in the meantime it's as annoying as Andy Samberg on crack). This is particularly acute given that the only thing fixing it in the regression range is a 2.5 megabyte patch that I'm only a small amount of the way through reading. On the flip side, I was able to find and fix several parser edge cases because Bugzilla itself was triggering them and the source file that was involved was not minified. That means I could actually read it and get error reports that made sense! Help support us lonely independent browser makers by keeping our lives a little less complicated. Thank you for your consideration!

Meanwhile, I have the parser changes on by default to see if it induces any regressions. Sites may or may not work any differently, but they should not work worse. If you find a site that seems to be behaving adversely in the beta, please toggle javascript.options.asyncfuncs to false and restart the browser, which will turn the warning back into an error. If even that doesn't fix it, make sure nothing on the site changed (like, I dunno, checking it in FPR15) before reporting it in the comments.

This version also "repairs" Firefox Sync support by connecting the browser back up to the right endpoints. You are reminded, however, that like add-on support Firefox Sync is only supported at a "best effort" level because I have no control over the backend server. I'll make reasonable attempts to keep it working, but things can break at any time, and it is possible that it will stay broken for good (and be removed from the UI) if data structures or the protocol change in a way I can't control for. There's a new FAQ entry for this I suggest you read.

Finally, there are performance improvements for HTML5 and URL parsing from later versions of Firefox as well as a minor update to same-site cookie support, plus a fix for a stupid bug with SVG backgrounds that I caused and Olga found, updates to basic adblock with new bad hosts, updates to the font blacklist with new bad fonts, and the usual security and stability updates from the ESRs.

I realize the delay means there won't be a great deal of time to test this, so let me know deficiencies as quickly as possible so they can be addressed before it goes live on or about September 2 Pacific time.