Configuring ancient ReadyNAS devices in the modern era

I once bought a pair of Netgear ReadyNAS server boxes. They’re terrible, but here we are more than a decade later and they’re still working. Slowly.

Thing is, the drives in them are so far over their design lives it’s just not funny, and while I don’t necessarily need much of the data they store… mmm, most of it is rushes and project files from decade-old video productions, it represents the best part of a decade of my work. I’m loathe to simply throw it away. And letting it bit-rot is just throwing it away without making a conscious decision to do so. I’m not clear that would be better.

Every now and then I try to do the sums for pushing everything into, say, Amazon Glacier. But I should really sort it out first, and that’s hard given how slow these bloody things are.

Upshot: I now have a third NAS in the house. A new-fangled Synology thing, which is so much nicer than the Netgears it’s hard to describe. Though it is made of plastic and lacks the natty little blue LCD display of the Netgears which show things like boot confirmation messages and the current IP address and so on. OK, so perhaps it’s not as nice as such, but it is considerably faster. And it has a trio of 12Tb drives in it, so it’s neatly larger than the eight drives its replacing. Gosh.

Right, next: copying stuff off the old drives. Oh.

Oh my.

This is going to take a while. Hours days weeks. Oh bothers. Well, I’m not tying my MacBook up for weeks on end, this is the sort of job Pis are made for.

OK: mount the old and new shares on a Pi, and run a massive rsync job to dump the contents across in a resumable sort of way. Great. Solid plan. Step one: mount the shaaaaaaaaaa oh.

Problem.

I’ve had issues before with mounting the old ReadyNAS shares over SMB, because the SMB version is so old modern Linux tends to have a bit of a hissy fit. Good old NFS works just fine… but not all the old shares are configured to export NFS volumes, or whatever the terminology is. They’re probably ‘clients’, or is X the only unixy protocol which does that backwards?

Whatever, it’s easy, surely: log into the NAS admin panel, select the share, click ‘NFS,’ dunzo. Only… the admin panel is a web interface. Served straight from 2010. Over https. Using that there super-modern TLS version… ah crap. 1.0.

The TLS apocalypse is properly upon us. Will Chrome open that admin page? Nope. Firefox? No. Safari? Not a chance. Edge? Geez, no. Opera? I had to download it, nope. Vivaldi? C’mon. Brave? Hahaha.

The impasse lasted a couple of days. I seriously considered dusting off my old PowerMac G4 just to run some shitty ancient browser. But then… then…

Frikkin’ Omniweb. OmniWeb. Let’s just list the things this does:

  1. Exist. It still exists.
  2. Be updated. It was last updated not last decade, not last year, but last month. For real.
  3. Run on Apple Silicon.
    1. Natively.
  4. Have vertical tabs, from before whichever time this is where somebody’s claimed to have invented them (again).
  5. Happily load shit old TLS1.0 sites without a hint of complaint.

All hail OmniWeb. You may be a bizarre footnote in browser history, but you had some great ideas and when nothing else ran on Mac OS X, you were there. You’re on your second (?third?) rendering engine and your third fourth fifth(?) processor architecture. When all of this is ocean and we’re all accessing Extended Reality Pro via our HeadCanon displays while breathing through MuskLungs™, you’ll still be there. Being merrily weird, probably quite slow, and somehow retro and modern in equal measure.

All hail Omniweb.

Packaging a Python guizero app with PyInstaller

TL;DR: Yes, you can, and yes, it ‘just works.’

Longer answer:

This week I realised I needed a small piece of tooling for some of my collaborators on an IoT project. I’ve been running debug and test stuff using Mosquitto‘s handy mosquitto_sub and mosquitto_pub command-line tools, but it’s not exactly reasonable for me to inflict those on others.

So I reached for JavaScript and started whipping up a quick little web app. I thought it’d make a nice change to stumble around committing travesties in a language that wasn’t C++, for a change. Alas, configuration and implementation details made that less straightforward than I’d hoped, so I ended up bailing and reaching for Python. Meh, sometimes you have to know when to quit.

I’ve used Laura Sach and Martin O’Hanlon‘s guizero library a few times before and rather liked it – it’s a minimal-configuration layer on top of… actually, I’m not sure if it’s Qt or Tkinter or what, but that’s rather the point. A few lines of code and you have a basic GUI working, call it done and move on with your life. And I’ve used the Paho MQTT libraries in Python before, so that bit was comfortable. Or so I thought.

Let’s take a look at what I built, some of the mistakes I made along the way, and how I fixed or worked around the problems I encountered.

Pretty quickly I had something like:

import paho.mqtt.client as mqtt
from guizero import App, TextBox
import config

def on_connect(client, userdata, flags, rc):
    client.subscribe("#")

def on_message(client, userdata, msg):
    payload = str(msg.payload.decode("utf-8"))
    mqttMessageBox.append(payload)

def mqttLoop():
    client.loop()

client = mqtt.Client()
client.on_connect = on_connect
client.on_message = on_message
client.username_pw_set(config.mqttUsername, config.mqttPassword)
client.connect(config.mqttHostname, config.mqttPort, 60)

app = App(title="My App Name")
mqttMessageBox = TextBox(app, width='fill', height='fill', multiline=True, scrollbar=True, text="Starting up...")

app.repeat(100, mqttLoop)
app.display()

Most of this will look familiar if you’ve done a guizero tutorial. The mundane bits are:

  • This basic app presents a window with a TextBox object, which it populates with messages received from an MQTT server.
  • I’ve omitted a bunch of stuff for clarity: my app has buttons for sending messages as well as the receiving pane.
  • The MQTT connection details are in a config.py file, which is listed in my .gitignore, so I don’t accidentally commit my security credentials to a public repository. Again.
  • We make an MQTT object client, then a guizero application and window, app.
  • Line 9 is needed because pretty much all the example MQTT code out there assumes Python 2.x and fails to mention that the returned message is a byte string, not a (unicode) Python string. So when you output it in Python3, you get something like b'The string you expect'. This little dance avoids that. Sigh.

Where it gets a little interesting is in the event and loop handling. Paho/MQTT example code typically calls client.loop_forever(), which in this case would conflict with guizero’s model, where the app.display() line also loops forever.

Line 24 is guizero’s way of dealing with this: we register a repeating callback on a guizero object (in this case the App itself, though it could be a widget like the TextBox). Every 100 milliseconds that calls my mqttLoop() function, which in turn calls client.loop(). Done.

guizero/MQTT update loop collisions

…and that works. Sort-of. Turns out something really quite nasty happens with all the looping, and guizero’s window becomes only sporadically responsive. The TextBox tended to update every few seconds at best, or perhaps only when I dragged the window between monitors. Button controls added to the window worked, sort-of, but were somewhat unresponsive and didn’t visually register clicks.

Upping the repeat time (ie. app.repeat(1000, mqttLoop)) helped the window, but the MQTT handling became janky. I fiddled for a while but a looming sense of ‘this isn’t going to work’ forced a rethink.

The fix turns out to be obvious in the Paho library: forget my mqttLoop() function and replace the guizero repeat line with:

client.loop_start()

Paho integrates some sort of threading model, and its updates and callbacks then run independently of whatever guizero is doing. It all just works, and the app behaves smoothly. As you’d expect, only… somehow, not how I’d expected: these are highly abstracted libraries, designed for ease of use in simplistic circumstances, and they’re both working around what as far as I’m aware is still a single-threaded runtime model in Python. And it works. Cool.

A few minutes later I had something a little more fully-featured, which allows my colleagues to send test messages across the device network, and to inspect what the network messaging is doing:

I really do mean ‘a few minutes’ – I’m working in VS Code with the beta of GitHub Copilot, and it’s flat-out amazing in terms of suggesting whole blocks of code for you. Often from the example or tutorial you’re following along with, which is a bit spooky.

Anyway, let me be clear that this app is not going to win any Apple Design Awards. But hey, it’s utility tooling for internal use. I need it soon, cross-platform, and ‘good enough’. The above is good enough.

Autoscrolling TextBox in guizero

I had two issues remaining. One was that the messages scroll off the bottom of the TextBox, the fix for which is in a comment on this Stack Overflow query. After line 10 above, add:

mqttMessageBox.tk.see('end')

We’re reaching through guizero to the underlying GUI library here (ah, look – it’s Tkinter after all!), and prompting it to scroll down. The API call seems weird, but it works so let’s gloss over that and move on.

Packaging (PyInstaller)

The last remaining issue: how do I give people this app? I can’t very well ask them to navigate to the right directory in a shell and type python3 vet.py every time.

The solution I’ve wanted an excuse to play with for a while is PyInstaller. One thing which has occasionally put me off is that the ‘Quickstart’ docs say:

pip install pyinstaller
pyinstaller yourprogram.py

…and that’s it. Which seems implausible to the point of being ridiculous.

Except: that pretty much was it, for me. Some minor notes:

  • For pip read pip3. One underlying theme you might spot in this post is that we are not yet done with the Python3 transition. Oooooh, no.
  • I’m not sure where PyQt comes into this, but it does… and for a while at the start of the year PyQt was one of those things that really didn’t play nicely with the new-fangled Apple Silicon Macs. Like, er, the one I’m working on. Those messy days are happily behind us, but I had to debug some legacy cruft in my system. That pretty much boiled down to brew link pyqt@5 after I’d removed an old shared library object that was still lying around – brew told me what to do when I tried brew install pyqt@5.
  • To make an executable application package, the incantation you need is pyinstaller --windowed yourprogram.py.

Now, PyInstaller isn’t perfect. I’d need to jump through some hoops if I wanted to sign the Mac app, but more importantly it’s not a cross-compiler: to package for Windows, you need to run it on Windows. And then there are, well, complications. I’ll try that tomorrow, maybe.

Wrapping up

The big take-home for me is this: being able to build bespoke bits of tooling is extremely helpful, and the faffing time involved in getting something like this out of the door is really very short. It’s genuinely taken me longer to write this post. You do need to be aware of what a fairly large range of moving parts are called, but once you can name things you can google them, and the chances are really very good that they’ll play nicely together.

The biggest challenge I typically come across is that gap between the trivial-case ‘quick start’ sort of guide, and the full-on API docs. This is where Stack Overflow is useful, but test cases there tend to be so specific it can be challenging to find the right one which relates to your code, or to find anything at all if you’re not sure where to start.

I suspect I’m far from alone in being broadly marginally competent, but still finding API docs hard to read. What’s lacking, I think, is intermediate-level worked examples. Hence this blog post, even if it boils down to ‘Packaging guizero with PyInstaller: yes, that works.’

If you’d like to see the code for the ‘full’ app – such as it is – it’s in this GitHub repository. It’s also worth checking out Laura and Martin’s recent guizero book, which is a fine example of… oh, intermediate-level worked examples. Well, fancy that.

Update 2021-09-03: Windows 10 package

Turned out I didn’t even have Python installed on my Windows box, so that was fun (hurray for Chocolatey, for that). Then I had some fumbling around with PowerShell, because I’m really not familiar with the Windows command line. I keep Git Bash around because while it’s weird and rather slow, if I squint a little it’s more familiar for me than anything PowerShell does. I’ve pretty much zero interest in learning Windows admin stuff at this point.

Finally, I had some dancing around to do to find the correct incantation to build my executable package. For a while I was doing the right thing, but something in the vet.spec file PyInstaller generates was forcing the wrong thing. Deleting the build and dist directories and the .spec file, then re-running pyinstaller vet.py -w --onefile produced what I want, which is a single .exe file I can bung on a file share and invite project collaborators to download and run.

It works! It’s even uglier than the Mac version, but it works and it’s mine. I’ll need to repeat the dance on a Raspberry Pi at some point, but for now: utility done, on with the next task.

Coda: somewhat astonishingly, the Windows .exe (built on a 12 year-old Mac Pro, seriously) also works on Windows 11 in emulation under arm64, as virtualised on my ARM MacBook. So many people in so many places working so hard, so muppets like me can build things that… ‘just work.’ Maybe the world can be a better place after all?

Movable Type 6.2

I may have moved this blog to WordPress, but my opinion of the system hasn’t really changed; I still think Movable Type was a better-engineered system right from the start, and it’s only really licensing and product direction that have steered me to join the mainstream. Well, that and WordPress having matured to the point of no longer annoying me quite enough to prevent me from seeing past its obvious attractions.

MT had its quirks, though. One of which was file uploads, which went… ach, basically anywhere. Ten years into an MT install and you’d have media files strewn in every damn directly on the server, it seemed. Trust me, I just cleaned out this server.

No more! Last week saw the release of Movable Type 6.2, which includes such innovations as:

DEFAULT UPLOAD SETTINGS

Website administrator […] can configure default settings for upload that including default upload destination.

Well, gosh. Maybe there’s life in the old Perl dog yet.

Blurred books

I find it surprisingly difficult to browse second-hand books on market stalls. Too often the serried ranks look like this — I can discern book-like shapes, but as I try to make out the authors or titles the world starts drawing in around the periphery. Even when I can read the words I don’t recognise any of them.

The first time I built my own PC I found myself sitting in the car park, terrified the list of components I’d agonised over would elicit knowing grimaces from the testosterone-laden atmosphere inside. I even had an uncomfortable time in a bicycle shop some years ago as I struggled to work out how much of my intimate knowledge of late-80s velotech was still relevant.

It seems obvious to promote and advocate for bookshops against the encroachment of Amazon, but we shouldn’t forget that to the uninitiated — and even to the unpracticed — they’re alien, vaguely threatening environments. There are reasons other than convenience for the steady rise of online shopping.

The main thing physical shops have going for them is human contact. Very, very few shops capitalise on that advantage. But then, dealing with humans is hard.

Domestic drones

Back in university, twenty years ago, I wrote a not-very-good sketch about a company who repurposed surplus cruise missiles. We’d all seen video footage from the (first) Gulf War showing Tomahawks diving through the windows of buildings, and it made some warped sort of mildly-satirical sense to think of them being used for… er… pizza delivery.

And now this:

Rrrright.

The punchline of the sketch was ‘minimum cholesterol damage.’ I said it wasn’t a very good sketch.

Tumblr, blogging, and all that jazz

Where Tumblr Came From – Anil Dash

many of us who were familiar with blogs already saw tumblelogs as “just a simple blogging template”, similar to what we were already doing on Movable Type or WordPress at the time, rather than a fundamentally different medium.

Despite that myopia, there was a lot of momentum around simplified, media-rich blogging at that moment in history.

Just read the whole thing. Blogging: it’s not as simple as it seems, and history is littered with the corpses not just of dead blogs, but of dead blogging systems.

(– via everyone)

Let’s hear it for Bundler

Turns out the solution to my broken Staticmatic install was as simple as:

bundle install

Yeah. That simple. Durr. But big respect for Bundler, a simple tool that solves a subtle problem – in this case downgrading the versions of a bunch of gems, rolling them back to whatever I built this system with in the first place, but doing all that only within this project.

Yeah, I know that’s what it does, but I didn’t really know that’s what it does. I should probably RTFM for some of this stuff.

Back in the Saddle III – Octopress

For giggles, another exploration in this series is a completely different animal, namely Octopress. Now, I’m not a complete stranger to command-line static site generators – I built StoryCog.com in StaticMatic, later adding a blog using the same templates and stylesheet driven by Melody (the open-source Movable Type 4 fork). Both are now dead projects, so I’ve some interest in revamping that site too.

Problems strike immediately, as the Octopress install instructions require a recent Git (currently: some old crap), and RVM (broken, somewhere). These are easy enough to fix, but in installing Ruby 1.9.something under a new RVM I seem to have nuked my gem set. Which means I’ve now completely broken StaticMatic. Oh, drat. This, incidentally, is why normal, sane, well-adjusted people shy away from command-line tools. If you live in them every day then all is well, but if you only sort-of understand them there are so many ways of screwing things up by accident, it’s just not funny.

Aaanyway: with the prerequisites sorted, time to move onto Octopress itself. And it works. OK, so I had some path problems with the configuration system, but once I’d got those sorted rsync deploy locally worked well, and the default output is pretty nice. I’m a big fan of the prebuilt video player, too. That’s the sort of thing that makes my life an awful lot simpler.

The rake/Jekyll import from WordPress worked well, and in principle I could redeploy my blog on Octopress almost immediately. So why haven’t I? Well, I may yet, but my hesitations at the moment are about time.

Publish time is an issue. On my Mac Pro, rebuilding after adding a new post takes about four minutes. It’s a single-threaded sort of thing, so I suspect my laptop would be slightly quicker (SSD, and all that), but I’m concerned that’s long enough to discourage quick posting. I note with interest that one of my favourite bloggers, having jumped to Octopress, set up a Kirby blog for quick posts. Well, to replace a Tumblr, but the point remains.

My other time issue is tinkering and learning time. While the default theme looks nice enough, it’s not completely to my taste. Delving into it is where the wheels start to come off, for me – hence my brief post yesterday. Notably, there’s precious little documentation and very few code comments on what the heck is going on in the templates and stylesheets. I love Compass, but I need help getting my head around someone else’s code of this complexity, and I suspect this is why so many Octopress blogs have stuck with the default.

Now, they’ve also stuck with the defaults because there’s plain good decision-making involved here. Octopress is opinionated software, and while I don’t agree with the choice made by the Hibari folks, most of Octopress slots nicely into my thinking about blogging. Which is cool.

I doubt I’m going to take the plunge just yet, but it’s good to know the option exists. It’s a radical platform, but I can see why so many geeks are enjoying it.