Dr. Lawlor's Code, Robots, & Things

November 8, 2017

Making chroot jails

Filed under: Linux, Sysadmin — Dr. Lawlor @ 8:20 pm

A chroot jail is a UNIX way run a dangerous application, like a network server, inside its own limited subset of the filesystem.  This cuts off access to recurring security holes like setuid executables in /bin or /usr/bin, kernel device files in places like /proc and /dev, and it lets you build your own restricted or sanitized runtime environment including libraries and config files.  Docker or rkt containers use chroot as one of the ways they sandbox containerized applications, and the fact all the libraries are included makes containers portable across systems.

A maximum security chroot jail consists of the prisoner process, and the bare minimum shared libraries necessary for it to run.  You can figure out which libraries are needed using “ldd ./prisoner”:

root@5a2c7cc2357f:/tmp/jailhouse# cp /bin/date ./prisoner
root@5a2c7cc2357f:/tmp/jailhouse# ldd ./prisoner
  linux-vdso.so.1 =>  (0x00007ffdf433d000)
  libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f39e8d18000)
  /lib64/ld-linux-x86-64.so.2 (0x000055751140c000)

(Or, since ldd is just a shell script wrapper around the dynamic linker library /lib64/ld-linux-x86-64.so.2, you can also run “/lib64/ld-linux-x86-64.so.2 –list ./prisoner”, although “ldd” is easier to remember.)

You can ignore linux-vdso, which is injected by the kernel, but if you copy the remaining shared libraries into the jail, you can run the prisoner process inside the jail using chroot.  Here, we just need ld-linux and libc:

root@5a2c7cc2357f:/tmp/jailhouse# mkdir lib64
root@5a2c7cc2357f:/tmp/jailhouse# cp /lib64/ld-linux-x86-64.so.2 lib64/
root@5a2c7cc2357f:/tmp/jailhouse# mkdir lib
root@5a2c7cc2357f:/tmp/jailhouse# mkdir lib/x86_64-linux-gnu
root@5a2c7cc2357f:/tmp/jailhouse# cp /lib/x86_64-linux-gnu/libc.so.6 lib/x86_64-linux-gnu/
root@5a2c7cc2357f:/tmp/jailhouse# chroot --userspec 12345:6789 . ./prisoner
Thu Nov 9 04:43:38 UTC 2017

Here we ran the prisoner as user ID 12345, in group ID 6789, due to the “–userspec 12345:6789” in the chroot call.  Without a userspec, the process runs in the jail as root, which is very bad (see chw00t)!  Neither this user nor group exist in my /etc/passwd or /etc/group files, which means the prisoner only gets a number, not a name.  You can add the username to either the system /etc/passwd or the chroot jailhouse/etc/passwd to get symbolic names, although do remember to set the permissions on jailhouse/etc/passwd carefully, because overwriting passwd with “jailbird:x:0:0::/home/jailbird:/bin/bash” could makes the jailbird user run as root within the jail.

Often the process you want to run inside the jail is more complex, and so it needs quite a few libraries to work.  Sometimes those libraries load more files and libraries at runtime, and they’re not always well documented.  In these cases, “strace ./prisoner” is quite useful, because it shows you every system call, including calls like open (finding config files), exec (calling sub-programs), or mmap (usually loading shared libraries or allocating memory).  I often copy strace into the chroot so I can “chroot . strace ./prisoner” and watch what the program is doing inside the jail.

A particularly complex program may need /dev entries created (CUDA in a chroot needed basically all of /dev/nvidia*), or things like /proc mounted into the chroot using “mount -o bind /proc proc”.  These sorts of comforts make the jail more like a halfway house, in that it allows the prisoners more functionality but does pose more of a danger to society.

I’ve built chroot jails packed with ancient library versions so I can run ancient programs in their old environment.

I’ve also built chroot jails to contain student code (e.g., in NetRun).

Further reading:

Advertisements

April 26, 2016

Terrain Rendering in 3D

Filed under: Graphics, Linux, Programming, Random Thoughts — Dr. Lawlor @ 4:49 pm

Back in 2003 I worked on a terrain simplification algorithm, a modification of Lindstrom’s method, to allow interactive exploration of big terrains in 3D.

A binary distribution is available for Windows (.zip) or Linux (.tar.gz), and the binaries should work, and let you view your own binary DEM and JPEG texture.  I made an attempt to include the source in there, although I built both of the binaries using my custom build system, so it’s likely to take quite a bit more work, and possibly even some missing custom libraries, to get it to compile.

September 15, 2015

Rescuing Data from a Banished Daemon

Filed under: Linux, Programming — Dr. Lawlor @ 9:12 pm

Last week we managed to accidentally delete the entire directory housing our custom database server (superstar), including the on-disk backups and historical backups of its robot info database.  Not good.

Amazingly, ps showed the server was still running, and poking around we could even see the in-memory copy of the database was still fine, but it was trapped inside this banished daemon.

# ps
...
no_priv 5149 0.1 0.0 14932 1640 ? S 18:04 0:08 ./superstar

Checking /proc/5149 shows the directory and exe have been deleted, which make a lot of tools break:

# ls -al /proc/5149
...
lrwxrwxrwx 1 no_priv no_priv 0 Sep 11 18:09 cwd -> /home/itest/git3/superstar (deleted)
-r-------- 1 no_priv no_priv 0 Sep 11 18:20 environ
lrwxrwxrwx 1 no_priv no_priv 0 Sep 11 18:09 exe -> /home/itest/git3/superstar/superstar (deleted)

We tried for a while to recreate or change the cwd, so we could tickle the daemon into backing up its database somewhere accessible–relinking won’t seem to change this directory, although possibly inode-level reconstruction of the old directory could have worked.

But I could still copy off the running executable for analysis:

# cp /proc/5149/exe /tmp/necrodaemon
# nm /tmp/necrodaemon | grep back
000000000061ca40 b _ZL15backup_filename

This is the in-memory address of the string holding the filename where the daemon tries to back up its in-memory database.  Currently it reads “db.bak”, a relative path, which is a problem since the process’s current working directory is gone.

Now just attach to the daemon in gdb, and we can see and change what’s inside it.

# gdb -p 5149

Of course, the daemon wasn’t compiled with debug symbols, so we can’t call methods or even functions.  So step one is verifying there’s anything actually at that “backup_filename” memory address–it’s a std::string, which has a single pointer to a separate refcounted object that actually holds the bytes of the string.  We then overwrote the bytes of this string to make the relative path into an absolute path, “/a.bak”, a file which we’d previously created and given the correct permissions.

(gdb) p *(long *)0x61ca40
$1 = 27717720
(gdb) p *(char **)0x61ca40
$7 = 0x1a6f058 "db.bak"
(gdb) set {char}0x1a6f058 = '/'
(gdb) p *(char **)0x61ca40
$8 = 0x1a6f058 "/b.bak"
(gdb) set {char}0x1a6f059 = 'a'
(gdb) p *(char **)0x61ca40
$9 = 0x1a6f058 "/a.bak"
(gdb) detach
Detaching from program: , process 5149
(gdb) quit

As soon as the backup ran, the daemon had written its in-memory database to the newly created file “/a.bak”, so our data was saved–daemon necromancy to the rescue!

August 12, 2015

Parrot “Rolling Spider” UAV Hacking: Dumping the Filesystem

Filed under: Hardware, Linux, Programming, Rolling Spider — Dr. Lawlor @ 11:53 pm

I just got a new tiny UAV, the Parrot “Rolling Spider” ($80), which is very fun to fly via bluetooth with my phone.  But it’s also a linux-based computer, so it’s also fun to hack!

The easiest way to get a root shell is to just plug it in via the USB cable, which not only shows up as a removable USB drive, it also shows up as a network device (at least, as of the 1.99.2 firmware version).  This means you can immediately get a root shell with:

telnet 192.168.2.1

That was easy!  Now to dump the filesystem, to netcat on port 1234.  (The ^p avoids /proc, which has infinite recursive root links; the ^y avoids /sys, which has files that change in size.)

tar cpf - [^p][^y]* | nc -l -p 1234

To get the filesystem as a file on your desktop computer, now just:

nc 192.168.2.1 1234 > rootfs.tar

This has *everything*, from:

drwxr-xr-x root/root 0 1969-12-31 14:00 bin/
lrwxrwxrwx root/root 0 1969-12-31 14:00 bin/getopt -> busybox
lrwxrwxrwx root/root 0 1969-12-31 14:00 bin/dd -> busybox
lrwxrwxrwx root/root 0 1969-12-31 14:00 bin/cp -> busybox
lrwxrwxrwx root/root 0 1969-12-31 14:00 bin/df -> busybox
lrwxrwxrwx root/root 0 1969-12-31 14:00 bin/ip -> busybox
-rwxrwxr-x root/root 35 1969-12-31 14:00 bin/kk
lrwxrwxrwx root/root 0 1969-12-31 14:00 bin/ln -> busybox
lrwxrwxrwx root/root 0 1969-12-31 14:00 bin/ls -> busybox
lrwxrwxrwx root/root 0 1969-12-31 14:00 bin/mv -> busybox
lrwxrwxrwx root/root 0 1969-12-31 14:00 bin/ps -> busybox
lrwxrwxrwx root/root 0 1969-12-31 14:00 bin/rm -> busybox
lrwxrwxrwx root/root 0 1969-12-31 14:00 bin/sh -> busybox
lrwxrwxrwx root/root 0 1969-12-31 14:00 bin/vi -> busybox
-rwxrwxr-x root/root 305 1969-12-31 14:00 bin/blink_led_orangeleft.sh
lrwxrwxrwx root/root 0 1969-12-31 14:00 bin/ash -> busybox

through:

drwxr-xr-x root/root 0 1969-12-31 14:00 usr/share/avahi/
-rw-r--r-- root/root 560 1969-12-31 14:00 usr/share/avahi/avahi-service.dtd
-rw-r--r-- root/root 5104 1969-12-31 14:00 usr/share/avahi/service-types
drwxr-xr-x root/root 0 1969-12-31 14:00 var/
lrwxrwxrwx root/root 0 1969-12-31 14:00 var/log -> /tmp/
-rw-rw-r-- root/root 7 1969-12-31 14:00 version.txt
drwxr-xr-x root/root 0 1969-12-31 14:00 www/
-rw-rw-r-- root/root 485 1969-12-31 14:00 www/index.html

For example, now I can see the contents of the control shell scripts:

$ cat bin/set_led_greenleft.sh 
#!/bin/sh


# temp behaviour : red light right on
gpio 33 -d ho 1
# temp behaviour : red light left off
gpio 30 -d ho 0

#green light right off
gpio 31 -d ho 0

#green light left on
gpio 32 -d ho 1

I can also see the details of how the code was built:

$ file bin/busybox
busybox: ELF 32-bit LSB executable, ARM, EABI5 version 1 (SYSV), dynamically linked (uses shared libs), for GNU/Linux 2.6.16, stripped

Of course, eventually I’ll want to permanently modify this filesystem, by re-flashing the UAV with a reverse engineered PLF firmware file, which is similar to the Parrot AR Drone PLF format.  I’m nearly there with “plftool -e raw -i rollingspider_update.plf -o .”, but each resulting file has the filename prepended in some sort of fixed-length binary header.

Stay tuned!

August 20, 2014

Setting up a HTTPS server via Apache mod_ssl: start to finish

Filed under: Linux, Servers, Sysadmin — Dr. Lawlor @ 1:42 am

If you’re like me, you’re already running a Linux web server.  It uses port 80 by default, which sends everything unencrypted across the network.  This is a bad idea, because anybody watching your users’ network traffic can see everything sent in either direction.  So you want to use HTTPS, which runs over TCP port 443.  Here’s how to set it up!

(more…)

August 18, 2014

UAF Eduroam setup in Linux

Filed under: Linux — Dr. Lawlor @ 5:23 pm

Eduroam provides simple, no-login roaming wifi at a bunch of higher educational institutions.  It’s an IEEE 802.1X authentication setup, where you set up credentials at your home institution, and they can be used worldwide.

First go to https://nah.alaska.edu/eduroam/, and log in with your UA credentials (same as blackboard, UAOnline, etc).  Download the Root certificate (rootCA.crt) and your “PKCS12 with intermediate and root” .p12 file, and save them to a fixed location on your machine.

Select the “eduroam” network from Network-Manager applet.
Should auto-identify as WPA/WPA2 Enterprise.
Authentication is TLS.  (It’s NOT Tunneled TLS or PEAP here, like it is most other places.)
Identity is “YOURLOGIN@alaska.edu”.
User certificate is left blank (for some reason).
CA certificate is rootCA.crt file downloaded above.
Private key is your .p12 file, again from above.
Private key password is YOURLOGIN.
Overall, it should look like this browser and network manager setup.

This shows Eduroam working correctly in Linux.

This shows Eduroam working correctly in Linux.

Now connect to the eduroam wifi!

It should connect within 10 seconds; if it lags for half a minute or more something’s messed up.  You need to download new certificates every year, so keep this post handy!

May 21, 2014

Tolerating Electrical Interference: reconnecting USB ports

Filed under: Arduino, Hardware, Linux — Dr. Lawlor @ 6:18 pm

UAF’s Aurora Robotics team entry in the NASA Robotic Mining Competition is using a standard Arduino Mega as the central microcontroller, which has worked well except for one intermittent disconnection problem, which we believe is caused by electrical interference from our big motors.

When it disconnects, the Linux kernel shows a dmesg error like “ehci_hub: USB disconnect on port 3 (EMI?)”.  We had previously built a system to tolerate this, by simply waiting a few seconds and reopening the USB serial connection, but this was only a partial fix since eventually the kernel disables the port completely.  This is bad because we could only fix it with a complete restart.  During our second practice mission yesterday, the kernel disabled all USB ports just as we began the run, resulting a completely scrubbed practice mission.

John Pender found a very simple way to re-enable the USB ports by running these commands as root:

echo "0000:00:1a.0" > /sys/bus/pci/drivers/ehci_hcd/unbind
echo "0000:00:1d.0" > /sys/bus/pci/drivers/ehci_hcd/unbind
echo "0000:00:1a.0" > /sys/bus/pci/drivers/ehci_hcd/bind
echo "0000:00:1d.0" > /sys/bus/pci/drivers/ehci_hcd/bind

This basically just turns off the kernel’s EHCI USB driver, then turns it back on, allowing the USB ports to be used again. Use “lspci | grep USB” to verify those addresses are correct for your machine (they seem right for most laptops).

The long-term fix for electrical interference issues on Arduino is using opto isolators on each digital control channel connected to a noisy source, but it’s often difficult to fully separate the ground planes in real robotics designs, so this is a simple fast way to get your robot moving again!

February 27, 2014

Ubuntu USB serial circuit boards vs “modem-man”

Filed under: Arduino, Hardware, Linux — Dr. Lawlor @ 9:41 pm

For CYBER-Alaska we built a custom Arduino-style circuit board (“Rovoduino”) that connects to the PC via USB.  It generally worked, but there was always a weird half-minute delay when you plugged it into a Linux PC before anything would start to work.  Tonight I was testing a new sensor on an Arduino Mega, a similar board, and noticed the plug-in-to-work delay there was only a little over one *second* (due to the bootloader).

Back to my custom board, trying a “sudo lsof | grep ttyUSB” during the half-minute delay shows the unexpected process “modem-man” spending this time sending commands like “AT+++” and trying to convince my circuit board that it is, in fact, a modem.

It’s not a modem.

You can explicitly blacklist your custom USB device from modem-man on Ubuntu in /lib/udev/rules.d/77-mm-usb-device-blacklist.rules, but it’s more reliable to just remove the darn thing entirely:

sudo apt-get remove modemmanager

Because seriously, who uses a modem?

This makes plugging and working on my USB serial board a lot snappier!

Blog at WordPress.com.