Dr. Lawlor's Code, Robots, & Things

September 28, 2015

Earthquake P-wave and nighttime anxiety

Filed under: Random Thoughts — Dr. Lawlor @ 9:24 pm

About 2am last night, I woke up feeling extremely anxious, and the hairs on my arms were standing up–neither of which is at all typical for me!  A few seconds later, I felt the bed start shaking very gently, and I realized it was probably an earthquake.

Checking the USGS Earthquake Map, there was indeed a magnitude 3.1 earthquake in our hills at this time, about 60km away and 11km deep.  Clearly something about the earthquake woke me up, but it’s surprising I subconsciously managed to detect the P wave while sound asleep, considering the later S wave (typically 2-3 times larger) could barely be felt.

September 15, 2015

Rescuing Data from a Banished Daemon

Filed under: Linux, Programming — Dr. Lawlor @ 9:12 pm

Last week we managed to accidentally delete the entire directory housing our custom database server (superstar), including the on-disk backups and historical backups of its robot info database.  Not good.

Amazingly, ps showed the server was still running, and poking around we could even see the in-memory copy of the database was still fine, but it was trapped inside this banished daemon.

# ps
...
no_priv 5149 0.1 0.0 14932 1640 ? S 18:04 0:08 ./superstar

Checking /proc/5149 shows the directory and exe have been deleted, which make a lot of tools break:

# ls -al /proc/5149
...
lrwxrwxrwx 1 no_priv no_priv 0 Sep 11 18:09 cwd -> /home/itest/git3/superstar (deleted)
-r-------- 1 no_priv no_priv 0 Sep 11 18:20 environ
lrwxrwxrwx 1 no_priv no_priv 0 Sep 11 18:09 exe -> /home/itest/git3/superstar/superstar (deleted)

We tried for a while to recreate or change the cwd, so we could tickle the daemon into backing up its database somewhere accessible–relinking won’t seem to change this directory, although possibly inode-level reconstruction of the old directory could have worked.

But I could still copy off the running executable for analysis:

# cp /proc/5149/exe /tmp/necrodaemon
# nm /tmp/necrodaemon | grep back
000000000061ca40 b _ZL15backup_filename

This is the in-memory address of the string holding the filename where the daemon tries to back up its in-memory database.  Currently it reads “db.bak”, a relative path, which is a problem since the process’s current working directory is gone.

Now just attach to the daemon in gdb, and we can see and change what’s inside it.

# gdb -p 5149

Of course, the daemon wasn’t compiled with debug symbols, so we can’t call methods or even functions.  So step one is verifying there’s anything actually at that “backup_filename” memory address–it’s a std::string, which has a single pointer to a separate refcounted object that actually holds the bytes of the string.  We then overwrote the bytes of this string to make the relative path into an absolute path, “/a.bak”, a file which we’d previously created and given the correct permissions.

(gdb) p *(long *)0x61ca40
$1 = 27717720
(gdb) p *(char **)0x61ca40
$7 = 0x1a6f058 "db.bak"
(gdb) set {char}0x1a6f058 = '/'
(gdb) p *(char **)0x61ca40
$8 = 0x1a6f058 "/b.bak"
(gdb) set {char}0x1a6f059 = 'a'
(gdb) p *(char **)0x61ca40
$9 = 0x1a6f058 "/a.bak"
(gdb) detach
Detaching from program: , process 5149
(gdb) quit

As soon as the backup ran, the daemon had written its in-memory database to the newly created file “/a.bak”, so our data was saved–daemon necromancy to the rescue!

Blog at WordPress.com.