Ubuntu do-release-upgrade failure recovery

do-release-upgrade is a handy but error-prone tool to upgrade your linux distribution. Today I was trying to get from Ubuntu 22.04 to 24.04, but it failed with missing libglib2 (possibly because I removed something annoying like apport that in turn removed ubuntu desktop).

Processing was halted because there were too many errors
The upgrade has completed but there were errors during the upgrade process.

This broke the GUI bad enough I couldn’t even open a new terminal window.

Before rebooting, in the same terminal I ran:
sudo apt --fix-broken install

That brought the GUI back, and I was able to restart into the new distro. I still needed to update and upgrade, but it seems operational. Another reinstall averted!

Linux self-rescue with init=/bin/bash

If you accidentally lock yourself out of a linux system, but can still get to GRUB, you can fix a lot of stuff by editing the normal GRUB kernel command line:

linux ... ro quiet splash $vt_handoff

with

linux ... rw init=/bin/bash

This drops you into a powerful root shell, which runs as PID 1 (init, normally systemd).

The “rw” makes your root filesystem writeable. You can then “nano /etc/passwd” (or /etc/sudoers) or “passwd <username>” or otherwise fix your system.

You won’t be able to cleanly shutdown, so “umount -r /” and “sync” when you’re done before power cycling.

Limitations: you need to be able to get to GRUB (tap “shift” at startup just after BIOS/UEFI handoff?) and edit the boot command line. This is a little trickier if your root filesystem is encrypted.

Arduino Uninitialized RAM

A wacky Arduino Uno R3 mystery: uninitialized heap data is not zero, and it’s often different in a few places, but it is generally consistent every time a given Arduino boots up. These two images show all of Arduino RAM (all 2KB!) for two separate physical Arduino 328P devices over 3 boots. The bottom are the registers and strings (predictable), but I was expecting black (zeros) in the middle, and it’s clearly not that!

In the pictures above, RGB pixel colors are the next 3 adjacent bytes in Arduino RAM, with addr 0 at the bottom in the obsolete style of memory maps (I like 0 at the top now, so it matches a hex dump, but I need to update my data viz tool). OK, but where is this data coming from?

Turns out, the Arduino Uno bootloader just … doesn’t initialize the RAM. There isn’t an operating system, and the Arduino runtime library doesn’t either. So this data is coming straight from the on-chip SRAM.

But why is it semi-random? I think it’s due to fabrication variations in the individual SRAM bit circuits! Theoretically an SRAM’s contents will be random at startup (it’s a bistable circuit), and a few bits seem really random, but many bits seem slightly biased toward 1 or 0.

To try it yourself, just read from a pointer just beyond your current heap data, for an Uno R3 this is around:
unsigned char *dragons = (unsigned char *)600;
(Or “dragons=new char;” for a more reliable boring version!)

The weird part about this data: it’s often the same bit pattern (less a few truly balanced bits) for a given chip, but normally quite different on different physical chips, even from the same manufacturer and time.

Folks have looked at using this for “Physically Unclonable Functions (PUF)”, such as for cryptography: https://intrinsic-id.com/wp-content/uploads/2017/05/SRAM-Start-Up.pdf (Who knew random garbage could be useful lol!)

Alaska’s “Dark Winter” vs “Light Winter”

Interior Alaska’s first winter, which I call “dark winter”, runs from October 15 through January 15, and typically gets basically zero direct sunlight unless you’re on top of a hill or ridge near solar noon.

After that you get another three months of “light winter” through approximately April 15 when breakup begins (plus or minus a few weeks). Direct sunlight is possible during light winter in an increasing number of places, for an increasing number of hours per day, and it feels glorious.

Both winters are consistently below freezing and have snow on the ground, but I enjoyed the direct sunshine today … as I snowshoed around at -20F!

The Power of Audience Questions

In grad school, I was at a small local conference along with maybe 30 other people, listening to a talk about the “Interior Penalty method”, a term I’d never heard. But I was a lowly grad student, and there were several Famous Professors in the audience, so I wasn’t brave enough to say anything. About 15 minutes in, somebody in the back asks “What is the Interior Penalty method?” and I perked up, because I wanted to know as well.

Now, the speaker clearly didn’t want to derail his slides, so he did a quick poll: “How many people here know what the Interior Penalty method is?” (He sounded dismissive, like he assumed everybody would clearly know the Interior Penalty method.)

NOBODY raised their hand, not the Famous Professors and especially not me!

The speaker was clearly taken aback, so quickly outlines something about it being a finite element approach that uses discontinous shape functions. A Famous Professor asked “Is that Discontinous Galerkin?” and the speaker agreed yes, some people call it that. Even I had seen a little Discontinuous Galerkin FEM, so actually followed some of the second half of the speaker’s talk.

This event sticks with me 20+ years later because (1) for the entire first half of this talk, *nobody* in the audience knew what the speaker talking about, yet (2) *nobody* spoke up about it!

As a speaker, this has made me really appreciate audience questions more, even “I have no idea what you’re talking about”. And a speaker can encourage questions in several non-obvious ways, like following “Are there any questions?” with an *awkward* period of silence, like 30 seconds, to let people get the courage to speak up. A simple “That’s a great question!” adds a lot too, even if it’s followed by “We’ll talk about that next week.”

Talking is easy, but communicating is hard.

Unicode Vision: UTF viewer

I cover UTF-8 encoding in my computer hardware course, so built this little web tool to see the UTF-8 bytes and UTF-32 chars from any string you paste in:

https://lawlorcode.com/2024/uni.html

(Mostly posting this here so I can find it myself!)

UAF On-Campus Wifi Setup

Fall 2023 classes are starting soon, and people are coming back on the University of Alaska Fairbanks (UAF) campus and need wifi!

Your options are:

• UAGuest: a pain to set up every time, and then capped at 10 megabit (the other options are 20x faster in a non-congested area!).
• eduroam: uses certificates, the easiest setup is the eduroam CAT (configuration assistant tool). Runs at up to 200 megabit, and works at almost any university around the world.
• UAlaska: uses your UA single sign-on password. This seems to be the most reliable connection, except you may need to manually switch to “PEAP” instead of the default TLS authentication, like this:

The flip side of this is to access university resources from off campus, you need the OIT VPN, which recently broke on Linux and I haven’t gotten it running again.

Starlink Economics

A written response to Thunderf00t / Phil Mason’s Jul 13, 2021 video (“Starlink: BUSTED!!”), which I’m not linking to because nobody else should be forced to endure his mocking, derisive tone.

Phil: Falcon 9 launch cost to SpaceX is $60M.

Actual: that’s what they charge external customers. Actual internal cost is like 1/2 (CNBC) to 1/4 (Musk) that number.

Phil: Starlink ground stations will pay $1B/year for bandwidth.

Actual: Tier 1 ISP peers do not pay for bandwidth, since they transit each other’s traffic. (Starlink isn’t Tier 1 yet, but offers connection scope that will eventually get it there.)

Phil: Customers require 20 Mbps, so a 20 Gbps sat can only serve 1,000 customers.

Actual: US fixed broadband customers average less than 3 Mb/sec during prime time.
(So a 20 Gbps sat can serve over 6,000 customers.)

Phil: A fully-deployed 10,000 satellite Starlink network can only support 3 million customers “at best”.

Actual: With 3,580 satellites Starlink is currently supporting over 1 million customers, despite huge countries like India and Indonesia not having service yet.

Phil’s general approach here is to fudge all his numbers a few-fold worse than reality, and to no one’s surprise, the business case then doesn’t make sense. Using today’s numbers, 1 million residential customers @ $110/month = $1.32 billion/year gross revenue; 50 launches/year @ $15M(?) internal cost = $0.75 billion/year launch cost, so the system economics hinge on the per-satellite cost and longevity. That’s not counting the extremely lucrative business, airline, maritime, and military customers, growth potential in many countries (esp. middle east, south Asia), and Starship enabling much more capable satellites.

I hope Starlink succeeds, it’s 10x faster than my other options available here in Alaska, and I love it!

Get a phone onto LTE Network

I’ve had an AT&T prepaid account for decades, but a recent phone upgrade only worked for texts, not voice (I get “Emergency Service Only” when I try to make calls). AT&T shut down their 3G network in February, so phones need to use LTE to work at all now.

I took the phone to an AT&T store, and they installed a slightly dodgy-looking app “ForceLTE”, but couldn’t actually get the phone to make calls on AT&T (despite the phone model number showing up in AT&T’s list of tested and working devices).

Reading the app instructions, it basically just uses the standard secret dialing code *#*#4636#*#* and picks “Device Information”. I tried setting the Preferred Network Type to “LTE only”, and the phone immediately connected to the AT&T network successfully.

I can now make voice calls and use data!

Palo Alto GlobalProtect on Linux via OpenConnect

My university (UAF) just migrated to Palo Alto GlobalProtect as their VPN. When I log into the VPN web interface it shows binary download links for Windows and Mac, but not Linux. Evidently there is a Palo Alto GlobalProtect client for Linux hidden somewhere behind their customer support portal, but the portal won’t let you create an account without a device serial number or sales order number (which is ridiculous!).

There’s a decent little quirky open source project to talk to several VPN servers called “openconnect“, by David Woodhouse, who wrote JFFS2 and several other kernel features. I grabbed the git version, which built and installed fine on my Ubuntu machine like this:

sudo apt-get install git wget gcc libtool m4 automake autotools-dev
git clone https://gitlab.com/openconnect/openconnect/
cd openconnect

# Seems to need a "vpnc-script" before config works
wget http://git.infradead.org/users/dwmw2/vpnc-scripts.git/blob_plain/HEAD:/vpnc-script
chmod +x vpnc-script
sudo mkdir /etc/vpnc
sudo cp vpnc-script /etc/vpnc/vpnc-script

# OK, normal config from here
./autogen.sh
./configure
make check
# FAIL: bad_dtls_test is OK, we're not using Cisco
sudo make install
sudo ldconfig

(Don’t worry about the “CSD trojan” comments during install, he’s just making fun of Cisco’s post-connect downloaded program by calling it a Trojan horse.)

Once it was installed, I’m now able to connect to the UA VPN like this:

sudo openconnect  --allow-insecure-crypto --protocol=gp  https://vpn.alaska.edu

You log in with your normal UA credentials, and as soon as it connects you’re on the UA VPN. Press control-C to disconnect. I made a script out of this, and pass the “–user=” option so I only need to enter my password. Slick!

(As of 2023, I now get a two-factor(?) challenge “SAML POST authentication is required via…”. If anybody can figure out what to do with this, please leave a comment or email me!)