Packaging a Python guizero app with PyInstaller

TL;DR: Yes, you can, and yes, it ‘just works.’

Longer answer:

This week I realised I needed a small piece of tooling for some of my collaborators on an IoT project. I’ve been running debug and test stuff using Mosquitto‘s handy mosquitto_sub and mosquitto_pub command-line tools, but it’s not exactly reasonable for me to inflict those on others.

So I reached for JavaScript and started whipping up a quick little web app. I thought it’d make a nice change to stumble around committing travesties in a language that wasn’t C++, for a change. Alas, configuration and implementation details made that less straightforward than I’d hoped, so I ended up bailing and reaching for Python. Meh, sometimes you have to know when to quit.

I’ve used Laura Sach and Martin O’Hanlon‘s guizero library a few times before and rather liked it – it’s a minimal-configuration layer on top of… actually, I’m not sure if it’s Qt or Tkinter or what, but that’s rather the point. A few lines of code and you have a basic GUI working, call it done and move on with your life. And I’ve used the Paho MQTT libraries in Python before, so that bit was comfortable. Or so I thought.

Let’s take a look at what I built, some of the mistakes I made along the way, and how I fixed or worked around the problems I encountered.

Pretty quickly I had something like:

import paho.mqtt.client as mqtt
from guizero import App, TextBox
import config

def on_connect(client, userdata, flags, rc):
    client.subscribe("#")

def on_message(client, userdata, msg):
    payload = str(msg.payload.decode("utf-8"))
    mqttMessageBox.append(payload)

def mqttLoop():
    client.loop()

client = mqtt.Client()
client.on_connect = on_connect
client.on_message = on_message
client.username_pw_set(config.mqttUsername, config.mqttPassword)
client.connect(config.mqttHostname, config.mqttPort, 60)

app = App(title="My App Name")
mqttMessageBox = TextBox(app, width='fill', height='fill', multiline=True, scrollbar=True, text="Starting up...")

app.repeat(100, mqttLoop)
app.display()

Most of this will look familiar if you’ve done a guizero tutorial. The mundane bits are:

  • This basic app presents a window with a TextBox object, which it populates with messages received from an MQTT server.
  • I’ve omitted a bunch of stuff for clarity: my app has buttons for sending messages as well as the receiving pane.
  • The MQTT connection details are in a config.py file, which is listed in my .gitignore, so I don’t accidentally commit my security credentials to a public repository. Again.
  • We make an MQTT object client, then a guizero application and window, app.
  • Line 9 is needed because pretty much all the example MQTT code out there assumes Python 2.x and fails to mention that the returned message is a byte string, not a (unicode) Python string. So when you output it in Python3, you get something like b'The string you expect'. This little dance avoids that. Sigh.

Where it gets a little interesting is in the event and loop handling. Paho/MQTT example code typically calls client.loop_forever(), which in this case would conflict with guizero’s model, where the app.display() line also loops forever.

Line 24 is guizero’s way of dealing with this: we register a repeating callback on a guizero object (in this case the App itself, though it could be a widget like the TextBox). Every 100 milliseconds that calls my mqttLoop() function, which in turn calls client.loop(). Done.

guizero/MQTT update loop collisions

…and that works. Sort-of. Turns out something really quite nasty happens with all the looping, and guizero’s window becomes only sporadically responsive. The TextBox tended to update every few seconds at best, or perhaps only when I dragged the window between monitors. Button controls added to the window worked, sort-of, but were somewhat unresponsive and didn’t visually register clicks.

Upping the repeat time (ie. app.repeat(1000, mqttLoop)) helped the window, but the MQTT handling became janky. I fiddled for a while but a looming sense of ‘this isn’t going to work’ forced a rethink.

The fix turns out to be obvious in the Paho library: forget my mqttLoop() function and replace the guizero repeat line with:

client.loop_start()

Paho integrates some sort of threading model, and its updates and callbacks then run independently of whatever guizero is doing. It all just works, and the app behaves smoothly. As you’d expect, only… somehow, not how I’d expected: these are highly abstracted libraries, designed for ease of use in simplistic circumstances, and they’re both working around what as far as I’m aware is still a single-threaded runtime model in Python. And it works. Cool.

A few minutes later I had something a little more fully-featured, which allows my colleagues to send test messages across the device network, and to inspect what the network messaging is doing:

I really do mean ‘a few minutes’ – I’m working in VS Code with the beta of GitHub Copilot, and it’s flat-out amazing in terms of suggesting whole blocks of code for you. Often from the example or tutorial you’re following along with, which is a bit spooky.

Anyway, let me be clear that this app is not going to win any Apple Design Awards. But hey, it’s utility tooling for internal use. I need it soon, cross-platform, and ‘good enough’. The above is good enough.

Autoscrolling TextBox in guizero

I had two issues remaining. One was that the messages scroll off the bottom of the TextBox, the fix for which is in a comment on this Stack Overflow query. After line 10 above, add:

mqttMessageBox.tk.see('end')

We’re reaching through guizero to the underlying GUI library here (ah, look – it’s Tkinter after all!), and prompting it to scroll down. The API call seems weird, but it works so let’s gloss over that and move on.

Packaging (PyInstaller)

The last remaining issue: how do I give people this app? I can’t very well ask them to navigate to the right directory in a shell and type python3 vet.py every time.

The solution I’ve wanted an excuse to play with for a while is PyInstaller. One thing which has occasionally put me off is that the ‘Quickstart’ docs say:

pip install pyinstaller
pyinstaller yourprogram.py

…and that’s it. Which seems implausible to the point of being ridiculous.

Except: that pretty much was it, for me. Some minor notes:

  • For pip read pip3. One underlying theme you might spot in this post is that we are not yet done with the Python3 transition. Oooooh, no.
  • I’m not sure where PyQt comes into this, but it does… and for a while at the start of the year PyQt was one of those things that really didn’t play nicely with the new-fangled Apple Silicon Macs. Like, er, the one I’m working on. Those messy days are happily behind us, but I had to debug some legacy cruft in my system. That pretty much boiled down to brew link pyqt@5 after I’d removed an old shared library object that was still lying around – brew told me what to do when I tried brew install pyqt@5.
  • To make an executable application package, the incantation you need is pyinstaller --windowed yourprogram.py.

Now, PyInstaller isn’t perfect. I’d need to jump through some hoops if I wanted to sign the Mac app, but more importantly it’s not a cross-compiler: to package for Windows, you need to run it on Windows. And then there are, well, complications. I’ll try that tomorrow, maybe.

Wrapping up

The big take-home for me is this: being able to build bespoke bits of tooling is extremely helpful, and the faffing time involved in getting something like this out of the door is really very short. It’s genuinely taken me longer to write this post. You do need to be aware of what a fairly large range of moving parts are called, but once you can name things you can google them, and the chances are really very good that they’ll play nicely together.

The biggest challenge I typically come across is that gap between the trivial-case ‘quick start’ sort of guide, and the full-on API docs. This is where Stack Overflow is useful, but test cases there tend to be so specific it can be challenging to find the right one which relates to your code, or to find anything at all if you’re not sure where to start.

I suspect I’m far from alone in being broadly marginally competent, but still finding API docs hard to read. What’s lacking, I think, is intermediate-level worked examples. Hence this blog post, even if it boils down to ‘Packaging guizero with PyInstaller: yes, that works.’

If you’d like to see the code for the ‘full’ app – such as it is – it’s in this GitHub repository. It’s also worth checking out Laura and Martin’s recent guizero book, which is a fine example of… oh, intermediate-level worked examples. Well, fancy that.

Update 2021-09-03: Windows 10 package

Turned out I didn’t even have Python installed on my Windows box, so that was fun (hurray for Chocolatey, for that). Then I had some fumbling around with PowerShell, because I’m really not familiar with the Windows command line. I keep Git Bash around because while it’s weird and rather slow, if I squint a little it’s more familiar for me than anything PowerShell does. I’ve pretty much zero interest in learning Windows admin stuff at this point.

Finally, I had some dancing around to do to find the correct incantation to build my executable package. For a while I was doing the right thing, but something in the vet.spec file PyInstaller generates was forcing the wrong thing. Deleting the build and dist directories and the .spec file, then re-running pyinstaller vet.py -w --onefile produced what I want, which is a single .exe file I can bung on a file share and invite project collaborators to download and run.

It works! It’s even uglier than the Mac version, but it works and it’s mine. I’ll need to repeat the dance on a Raspberry Pi at some point, but for now: utility done, on with the next task.

Coda: somewhat astonishingly, the Windows .exe (built on a 12 year-old Mac Pro, seriously) also works on Windows 11 in emulation under arm64, as virtualised on my ARM MacBook. So many people in so many places working so hard, so muppets like me can build things that… ‘just work.’ Maybe the world can be a better place after all?

Movable Type 6.2

I may have moved this blog to WordPress, but my opinion of the system hasn’t really changed; I still think Movable Type was a better-engineered system right from the start, and it’s only really licensing and product direction that have steered me to join the mainstream. Well, that and WordPress having matured to the point of no longer annoying me quite enough to prevent me from seeing past its obvious attractions.

MT had its quirks, though. One of which was file uploads, which went… ach, basically anywhere. Ten years into an MT install and you’d have media files strewn in every damn directly on the server, it seemed. Trust me, I just cleaned out this server.

No more! Last week saw the release of Movable Type 6.2, which includes such innovations as:

DEFAULT UPLOAD SETTINGS

Website administrator […] can configure default settings for upload that including default upload destination.

Well, gosh. Maybe there’s life in the old Perl dog yet.

Blurred books

I find it surprisingly difficult to browse second-hand books on market stalls. Too often the serried ranks look like this — I can discern book-like shapes, but as I try to make out the authors or titles the world starts drawing in around the periphery. Even when I can read the words I don’t recognise any of them.

The first time I built my own PC I found myself sitting in the car park, terrified the list of components I’d agonised over would elicit knowing grimaces from the testosterone-laden atmosphere inside. I even had an uncomfortable time in a bicycle shop some years ago as I struggled to work out how much of my intimate knowledge of late-80s velotech was still relevant.

It seems obvious to promote and advocate for bookshops against the encroachment of Amazon, but we shouldn’t forget that to the uninitiated — and even to the unpracticed — they’re alien, vaguely threatening environments. There are reasons other than convenience for the steady rise of online shopping.

The main thing physical shops have going for them is human contact. Very, very few shops capitalise on that advantage. But then, dealing with humans is hard.

Domestic drones

Back in university, twenty years ago, I wrote a not-very-good sketch about a company who repurposed surplus cruise missiles. We’d all seen video footage from the (first) Gulf War showing Tomahawks diving through the windows of buildings, and it made some warped sort of mildly-satirical sense to think of them being used for… er… pizza delivery.

And now this:

Rrrright.

The punchline of the sketch was ‘minimum cholesterol damage.’ I said it wasn’t a very good sketch.

Tumblr, blogging, and all that jazz

Where Tumblr Came From – Anil Dash

many of us who were familiar with blogs already saw tumblelogs as “just a simple blogging template”, similar to what we were already doing on Movable Type or WordPress at the time, rather than a fundamentally different medium.

Despite that myopia, there was a lot of momentum around simplified, media-rich blogging at that moment in history.

Just read the whole thing. Blogging: it’s not as simple as it seems, and history is littered with the corpses not just of dead blogs, but of dead blogging systems.

(– via everyone)

Let’s hear it for Bundler

Turns out the solution to my broken Staticmatic install was as simple as:

bundle install

Yeah. That simple. Durr. But big respect for Bundler, a simple tool that solves a subtle problem – in this case downgrading the versions of a bunch of gems, rolling them back to whatever I built this system with in the first place, but doing all that only within this project.

Yeah, I know that’s what it does, but I didn’t really know that’s what it does. I should probably RTFM for some of this stuff.

Back in the Saddle III – Octopress

For giggles, another exploration in this series is a completely different animal, namely Octopress. Now, I’m not a complete stranger to command-line static site generators – I built StoryCog.com in StaticMatic, later adding a blog using the same templates and stylesheet driven by Melody (the open-source Movable Type 4 fork). Both are now dead projects, so I’ve some interest in revamping that site too.

Problems strike immediately, as the Octopress install instructions require a recent Git (currently: some old crap), and RVM (broken, somewhere). These are easy enough to fix, but in installing Ruby 1.9.something under a new RVM I seem to have nuked my gem set. Which means I’ve now completely broken StaticMatic. Oh, drat. This, incidentally, is why normal, sane, well-adjusted people shy away from command-line tools. If you live in them every day then all is well, but if you only sort-of understand them there are so many ways of screwing things up by accident, it’s just not funny.

Aaanyway: with the prerequisites sorted, time to move onto Octopress itself. And it works. OK, so I had some path problems with the configuration system, but once I’d got those sorted rsync deploy locally worked well, and the default output is pretty nice. I’m a big fan of the prebuilt video player, too. That’s the sort of thing that makes my life an awful lot simpler.

The rake/Jekyll import from WordPress worked well, and in principle I could redeploy my blog on Octopress almost immediately. So why haven’t I? Well, I may yet, but my hesitations at the moment are about time.

Publish time is an issue. On my Mac Pro, rebuilding after adding a new post takes about four minutes. It’s a single-threaded sort of thing, so I suspect my laptop would be slightly quicker (SSD, and all that), but I’m concerned that’s long enough to discourage quick posting. I note with interest that one of my favourite bloggers, having jumped to Octopress, set up a Kirby blog for quick posts. Well, to replace a Tumblr, but the point remains.

My other time issue is tinkering and learning time. While the default theme looks nice enough, it’s not completely to my taste. Delving into it is where the wheels start to come off, for me – hence my brief post yesterday. Notably, there’s precious little documentation and very few code comments on what the heck is going on in the templates and stylesheets. I love Compass, but I need help getting my head around someone else’s code of this complexity, and I suspect this is why so many Octopress blogs have stuck with the default.

Now, they’ve also stuck with the defaults because there’s plain good decision-making involved here. Octopress is opinionated software, and while I don’t agree with the choice made by the Hibari folks, most of Octopress slots nicely into my thinking about blogging. Which is cool.

I doubt I’m going to take the plunge just yet, but it’s good to know the option exists. It’s a radical platform, but I can see why so many geeks are enjoying it.

Back in the Saddle II – Habari

My assumption, when embarking on this little series of posts, was that I’d jot a few brief notes about the blog engine alternatives I explored by way of review. Pretty much, that would be that.

Trouble is, each of the projects I’m trying is more-or-less freeware, built and maintained by volunteers. They’ve made decisions based on how they want to use their product, and to a some extent any ‘review’ I might offer would be more a comparison between my preferences and intentions and theirs. Which is of stuff all use to anyone else.

So the short version of this post is: I wanted to like Habari, I really did. But I’m not using it because… well… I don’t. I’m sure it’s great. But it’s not for me. Here’s an example:

habari-menuThis is the top-level Habari menu, and pretty much the main bit of interface offered up by the system. I thought this sort of menu was cool when I first saw it in NeXTSTEP circa. 1992, but in practice I find the waggling-the-mouse-like-a-gear-lever dog-leg manoeuvre annoying as anything.

The apparent goal is to present a thoroughly minimal interface, to get out of the way as much as possible. That’s admirable, but there’s always a trade-off involved in simplicity, and for me this falls too far on the side of ‘absent’ rather than ‘simplified.’ Besides, poke only a little deeper and you quickly stumble across modal dialogue pop-ups. Habari is not, at this stage, impressing me as a system which aligns with my ideas of good taste.

Importing my blog archive (from a WordPress database) was seamless and trivial, which is great. But can I find an off-the-shelf theme I like? No. Not even vaguely. Now, again, I have very particular tastes here, but almost everything in the theme repository looks like dodgy ports from WordPress circa 2009. It’s not at all clear which remain maintained.

OK, so let’s take a brief look at how themes are built. I’ve never been a fan of WordPress’ ‘pepper PHP throughout the template’ approach, much preferring Movable Type’s template tags. Intriguingly, Habari offers both alternatives, the former via RawPHPEngine, the latter via HiEngine. But:

If you’re looking to start building themes in Habari, and you’re not accustomed to building templates using a syntax similar to this, then you should most definitely not use HiEngine. Instead, you should look into the native PHP support provided by RawPHPEngine, which is faster and better at teaching you real PHP, which can be useful when creating more complex themes.

–(From the project wiki).

Ouch. Yeah, we’re not going to agree on that. Heck, the last few sites I’ve built I’ve done pretty much in HAML, I’ve really no interest in going back to ?php if ( blah ) ?whatever?php? nonsense.

Upshot: Habari might be great, but I’m not the right user for it.