The Daily Grind

TL;DR: Yes, you can, and yes, it ‘just works.’

Longer answer:

This week I realised I needed a small piece of tooling for some of my collaborators on an IoT project. I’ve been running debug and test stuff using Mosquitto‘s handy mosquitto_sub and mosquitto_pub command-line tools, but it’s not exactly reasonable for me to inflict those on others.

So I reached for JavaScript and started whipping up a quick little web app. I thought it’d make a nice change to stumble around committing travesties in a language that wasn’t C++, for a change. Alas, configuration and implementation details made that less straightforward than I’d hoped, so I ended up bailing and reaching for Python. Meh, sometimes you have to know when to quit.

I’ve used Laura Sach and Martin O’Hanlon‘s guizero library a few times before and rather liked it – it’s a minimal-configuration layer on top of… actually, I’m not sure if it’s Qt or Tkinter or what, but that’s rather the point. A few lines of code and you have a basic GUI working, call it done and move on with your life. And I’ve used the Paho MQTT libraries in Python before, so that bit was comfortable. Or so I thought.

Let’s take a look at what I built, some of the mistakes I made along the way, and how I fixed or worked around the problems I encountered.

Pretty quickly I had something like:

import paho.mqtt.client as mqtt
from guizero import App, TextBox
import config

def on_connect(client, userdata, flags, rc):
    client.subscribe("#")

def on_message(client, userdata, msg):
    payload = str(msg.payload.decode("utf-8"))
    mqttMessageBox.append(payload)

def mqttLoop():
    client.loop()

client = mqtt.Client()
client.on_connect = on_connect
client.on_message = on_message
client.username_pw_set(config.mqttUsername, config.mqttPassword)
client.connect(config.mqttHostname, config.mqttPort, 60)

app = App(title="My App Name")
mqttMessageBox = TextBox(app, width='fill', height='fill', multiline=True, scrollbar=True, text="Starting up...")

app.repeat(100, mqttLoop)
app.display()

Most of this will look familiar if you’ve done a guizero tutorial. The mundane bits are:

• This basic app presents a window with a TextBox object, which it populates with messages received from an MQTT server.
• I’ve omitted a bunch of stuff for clarity: my app has buttons for sending messages as well as the receiving pane.
• The MQTT connection details are in a config.py file, which is listed in my .gitignore, so I don’t accidentally commit my security credentials to a public repository. Again.
• We make an MQTT object client, then a guizero application and window, app.
• Line 9 is needed because pretty much all the example MQTT code out there assumes Python 2.x and fails to mention that the returned message is a byte string, not a (unicode) Python string. So when you output it in Python3, you get something like b'The string you expect'. This little dance avoids that. Sigh.

Where it gets a little interesting is in the event and loop handling. Paho/MQTT example code typically calls client.loop_forever(), which in this case would conflict with guizero’s model, where the app.display() line also loops forever.

Line 24 is guizero’s way of dealing with this: we register a repeating callback on a guizero object (in this case the App itself, though it could be a widget like the TextBox). Every 100 milliseconds that calls my mqttLoop() function, which in turn calls client.loop(). Done.

guizero/MQTT update loop collisions

…and that works. Sort-of. Turns out something really quite nasty happens with all the looping, and guizero’s window becomes only sporadically responsive. The TextBox tended to update every few seconds at best, or perhaps only when I dragged the window between monitors. Button controls added to the window worked, sort-of, but were somewhat unresponsive and didn’t visually register clicks.

Upping the repeat time (ie. app.repeat(1000, mqttLoop)) helped the window, but the MQTT handling became janky. I fiddled for a while but a looming sense of ‘this isn’t going to work’ forced a rethink.

The fix turns out to be obvious in the Paho library: forget my mqttLoop() function and replace the guizero repeat line with:

client.loop_start()

Paho integrates some sort of threading model, and its updates and callbacks then run independently of whatever guizero is doing. It all just works, and the app behaves smoothly. As you’d expect, only… somehow, not how I’d expected: these are highly abstracted libraries, designed for ease of use in simplistic circumstances, and they’re both working around what as far as I’m aware is still a single-threaded runtime model in Python. And it works. Cool.

A few minutes later I had something a little more fully-featured, which allows my colleagues to send test messages across the device network, and to inspect what the network messaging is doing:

I really do mean ‘a few minutes’ – I’m working in VS Code with the beta of GitHub Copilot, and it’s flat-out amazing in terms of suggesting whole blocks of code for you. Often from the example or tutorial you’re following along with, which is a bit spooky.

Anyway, let me be clear that this app is not going to win any Apple Design Awards. But hey, it’s utility tooling for internal use. I need it soon, cross-platform, and ‘good enough’. The above is good enough.

Autoscrolling TextBox in guizero

I had two issues remaining. One was that the messages scroll off the bottom of the TextBox, the fix for which is in a comment on this Stack Overflow query. After line 10 above, add:

mqttMessageBox.tk.see('end')

We’re reaching through guizero to the underlying GUI library here (ah, look – it’s Tkinter after all!), and prompting it to scroll down. The API call seems weird, but it works so let’s gloss over that and move on.

Packaging (PyInstaller)

The last remaining issue: how do I give people this app? I can’t very well ask them to navigate to the right directory in a shell and type python3 vet.py every time.

The solution I’ve wanted an excuse to play with for a while is PyInstaller. One thing which has occasionally put me off is that the ‘Quickstart’ docs say:

pip install pyinstaller
pyinstaller yourprogram.py

…and that’s it. Which seems implausible to the point of being ridiculous.

Except: that pretty much was it, for me. Some minor notes:

• For pip read pip3. One underlying theme you might spot in this post is that we are not yet done with the Python3 transition. Oooooh, no.
• I’m not sure where PyQt comes into this, but it does… and for a while at the start of the year PyQt was one of those things that really didn’t play nicely with the new-fangled Apple Silicon Macs. Like, er, the one I’m working on. Those messy days are happily behind us, but I had to debug some legacy cruft in my system. That pretty much boiled down to brew link pyqt@5 after I’d removed an old shared library object that was still lying around – brew told me what to do when I tried brew install pyqt@5.
• To make an executable application package, the incantation you need is pyinstaller --windowed yourprogram.py.

Now, PyInstaller isn’t perfect. I’d need to jump through some hoops if I wanted to sign the Mac app, but more importantly it’s not a cross-compiler: to package for Windows, you need to run it on Windows. And then there are, well, complications. I’ll try that tomorrow, maybe.

Wrapping up

The big take-home for me is this: being able to build bespoke bits of tooling is extremely helpful, and the faffing time involved in getting something like this out of the door is really very short. It’s genuinely taken me longer to write this post. You do need to be aware of what a fairly large range of moving parts are called, but once you can name things you can google them, and the chances are really very good that they’ll play nicely together.

The biggest challenge I typically come across is that gap between the trivial-case ‘quick start’ sort of guide, and the full-on API docs. This is where Stack Overflow is useful, but test cases there tend to be so specific it can be challenging to find the right one which relates to your code, or to find anything at all if you’re not sure where to start.

I suspect I’m far from alone in being broadly marginally competent, but still finding API docs hard to read. What’s lacking, I think, is intermediate-level worked examples. Hence this blog post, even if it boils down to ‘Packaging guizero with PyInstaller: yes, that works.’

If you’d like to see the code for the ‘full’ app – such as it is – it’s in this GitHub repository. It’s also worth checking out Laura and Martin’s recent guizero book, which is a fine example of… oh, intermediate-level worked examples. Well, fancy that.

Update 2021-09-03: Windows 10 package

Turned out I didn’t even have Python installed on my Windows box, so that was fun (hurray for Chocolatey, for that). Then I had some fumbling around with PowerShell, because I’m really not familiar with the Windows command line. I keep Git Bash around because while it’s weird and rather slow, if I squint a little it’s more familiar for me than anything PowerShell does. I’ve pretty much zero interest in learning Windows admin stuff at this point.

Finally, I had some dancing around to do to find the correct incantation to build my executable package. For a while I was doing the right thing, but something in the vet.spec file PyInstaller generates was forcing the wrong thing. Deleting the build and dist directories and the .spec file, then re-running pyinstaller vet.py -w --onefile produced what I want, which is a single .exe file I can bung on a file share and invite project collaborators to download and run.

It works! It’s even uglier than the Mac version, but it works and it’s mine. I’ll need to repeat the dance on a Raspberry Pi at some point, but for now: utility done, on with the next task.

Coda: somewhat astonishingly, the Windows .exe (built on a 12 year-old Mac Pro, seriously) also works on Windows 11 in emulation under arm64, as virtualised on my ARM MacBook. So many people in so many places working so hard, so muppets like me can build things that… ‘just work.’ Maybe the world can be a better place after all?