[Technology] Accessing files.web.uoguelph.ca from Linux when off campus

Until a few weeks ago, I was able to login remotely to my University's web hosting server.  Now, I need to connect via VPN first.



On Fedora Linux, I made sure I had vpnc installed and NetworkManager-vpnc.   Then I went to Network Manager's settings/preferences, added a new connection (+ in the bottom left), chose VNC, and chose the vpnc client.  Then I used the settings from this page:



https://www.uoguelph.ca/ccs/internet-phones/virtual-private-network-vpn/using-vpn



It beats using the creepy script from vpn.uoguelph.ca at least.

[Technology] Automating my life away

A decade ago, part of a job was to help automate creation
of a report that normally involved a human scanning through dozens of
spreadsheets each year, copying and pasting numbers and generating
graphs and tables, manually.

The place where I worked
predominantly used Perl.  (It was a science laboratory.)   My solution
was to use OpenOffice's GUI to bulk-convert the Microsoft Excel
spreadsheets (binary XLS files) to whichever file format OpenOffice used
at the time that was actually a collection of XML in a zipped
directory. (Was that .sdc?)

I then used Perl's libxml
wrapper (probably wrapping libxml2) to parse the XML into an SQLite
database.  (Or was it PostgeSQL like everything else there?)  I had
tried to use some other Perl XML library, but it was Super Slow compared
to the snappy libxml2 (written in C).  Like, a difference in many
minutes to run versus one second.

From there, I wrote
some Perl CGI scripts to automatically generate most of the tables and
graphics into HTML form, in some fashion that looked nice when imported
into OpenOffice. (Really?!)  The person who was responsible for the
report, who normally took a few weeks to do, said that it would save him
so much of the time, that it would take him now a couple days to look
at the data and write conclusions and maybe do some cleaning.  He
marveled at automation and how quickly he could be replaced.  The
automation revolution in action.

Today I used LibreOffice to convert about a hundred spreadsheets into CSV files with this one command:

find . -name "*.xls" | while read line; do
  (cd "$(dirname "$line")" &&
   libreoffice --headless --convert-to csv "$(basename "$line")");
done

I
am not sure that OpenOffice could do headless conversion back then.  I
was probably overwhelmed by all the potential everywhere to discover the
functionality if it exists.  This would have helped free a few hours of
my life.

Automation, a revolution.



UPDATE:

Converting to CSV only does the 1st sheet tab.  You apparently can't specify a different sheet to be the active sheet.  I also had to convert some other old spreadsheets for easy parsing (mostly just grep'ing through them to find data).   In that case, I used --convert-to html which was beautiful, even if one of the resultant HTML files was 138MB large.

[Technology] Starting Firefox Faster

summary: if you use pinned tabs, and they're all loading at start, despite setting tabs to not load until selected, this post is for you.

I have two hard disks, a 24GB SSD (fast!) and a spinner.  I noticed that to fully load Firefox and wait for everything to settle, it was taking about a minute and a half, back when my root directory was on the HDD.  Recently (see earlier posts), I moved my root onto the SSD.  That improved things marginally.  Firefox was now settling in around 1m12s.  I tried moving my profile onto the SSD.  Now I was down around 1m00s.  (These are averaged from multiple warm runs.)  That is still abysmal.



One thing I notice is that a private web app I wrote (look up My Daily in past posts maybe) is guarded by an .htaccess file, and I have it as a pinned tab.  In that full minute, before things finish settling, I get prompted to authenticate myself.  I then recalled that I had enabled the Firefox option to load tabs when clicked (Preferences > General > "Don't load tabs until selected"), so why is it loading?



All the pinned tabs actually seem to be loading at startup.  Grr.  Perhaps that is tunable?



I went to about:config, and sure enough, typing in "pinned" filtered down to this setting:

browser.sessionstore.restore_pinned_tabs_on_demand

defaulting to "false".  I set it to "true", and now Firefox takes <3s to open and settle.  Sure, clicking on a pinned tab now takes a normal time to load the tab, but it's better than loading GMail, Google Calendar, Blogger, Google Drive, Google Keep and my web app ALL AT ONCE.

[Technology] Resizing my encrypted /home partition

I just wrote a post about why I reinstalled Fedora 22: because I wanted root to be on the SSD, and I needed to purge a lot of old installed software anyway. :P  And how that led to trying out the open source b43 wireless driver.



This is more interesting than my wireless problem. 
Because I no longer needed a / partition on my HDD, I wanted to extend
my home partition.  It isn't as dead-simple as back a decade ago.  Back
then, I would use fdisk to delete and recreate partitions and resize2fs
to resize the file system.



Now I have layers upon
layers of useful abstraction.  I'm calling each something potentially
inaccurate, and the tool I needed in brackets. (Those with wildcards
match multiple tools with the same beginning.)



Top

• ext4 file system (resize2fs)


• Logical volume (lv*)


• Volume group (vg*)


• Physical volume (pv*)


• partitions (fdisk)

Bottom



At first I was almost a little annoyed
at the apparent complexity, but it's actually straight forward.  My
drive is still divided into multiple partitions in a partition table. 
(/dev/sda1, /dev/sda2, /dev/sda3, etc; I have 8 for various reasons).



Physical
volumes seem to correspond to those partitions, but mostly just the
useful storage ones (e.g. I don't have them for EFI or swap
partitions).  pvcreate lets me define them.



The
physical volumes are collected into volume groups.  For example,
/dev/sdb1 (my intended root) might be a part of "pantaloons-fedora-ssd"
(if I named my machine pantaloons), and /dev/sda2 and /dev/sda4 might
both be part of "pantaloons-fedora-hdd" (two large areas for my /home
that I'd like to combine later, but which are currently separated by some other, annoying partition).



The
volume group kind of looks like a single contiguous (!) entity (despite
being a collection of disparate physical partitions).  Then, I can
define logical volumes atop the volume groups.



I can
have both / and swap as separate logical volumes atop
"pantaloons-fedora-ssd" (talk about cramped - maybe I don't want to
hibernate after all), and have a single logical volume for /home atop
the "pantaloons-fedora-hdd" volume group.



This is where things get a little trippy though.  Where does the encryption happen?



For my root file system, it looks like it happens like:

partition /dev/sdb1 > LUKS > physical volume > EXT4 file system



For my home file system, it looks like this:

partitions
/dev/sda2,/dev/sda4 > two physical volumes > 1 volume group > 1
logical volume > LUKS > EXT4 file system

FS	Ext4		Ext4
(LUKS)			/dev/mapper/luks-BAR
LVs	root	swap	/home
VGs	pantaloons-fedora-ssd		pantaloons-fedora-hdd
PVs	/dev/mapper/luks-FOO		/dev/sda2	/dev/sda4
(LUKS)	/dev/mapper/luks-FOO		/dev/sda2	/dev/sda4
partitions	/dev/sdb3
disks	/dev/sdb (SSD)		/dev/sda (HDD)

The home partition and its LUKS (using
cryptsetup/dm-crypt) was originally configured through whichever tool
Fedora provided years ago (preupgrade? fedup? anaconda?), while the
configuration on my SSD was what was recommended in F22.



Basically, I had to (with steps that actually increase space emboldened):

• define a new partition in the partition table, 


• define a new Physical Volume (PV) over top it, 


• extend my Volume Group (VG) to include the new Physical Volume


• unmount the file system I was going to extend, 


• close its LUKS setup


• extend the file system's Logical Volume (LV) overtop the new space in the Volume Group


• reopen the LUKS setup


• resize the LUKS setup (are these two out of order? hope not!), and


• resize the Ext4 partition

A bit more detail, here are the steps I used on my system (from
my bash_history, probably missing some steps; I think I also used
lvremove
or pvremove to remove LVs and PVs related to the old home partition, oh
well!):

• # cfdisk, to delete the old root on the /dev/sda4 partition and recreate it


• # pvcreate /dev/sda4 (creates a physical volume)


• # vgextend pantaloons-fedora-hdd /dev/sda4 (add the PV into the VG increasing its overall size)


• make sure /home isn't mounted


• # fsck.ext4 -C 0 -f /dev/mapper/luks-BAR (let's make sure the inodes are fine on our file system before messing with it)


• # cryptsetup luksClose luks-BAR (close LUKS while we change the underlying LV)


• # lvextend -L +61G /dev/pantaloons-fedora-hdd/home (increase the size of our LV within the VG)


• # lvextend -L +242M /dev/pantaloons-fedora-hdd/home (catch a little more space)


• there is probably a way to just extend it to the full space available, maybe read the man page


• # cryptsetup luksOpen /dev/pantaloons-fedora-home luks-BAR


• # cryptsetup resize luks-BAR (resizing the LUKS setup)


• # fsck.ext4 -C 0 -f /dev/mapper/luks-BAR (for paranoia maybe)


• # resize2fs /dev/mapper/luks-BAR (finally, let's resize our Ext4 file system)

Sadly, I can't cite the sources I pieced some parts together from, because I did this form a live USB key which is supposed to keep history as persistent storage but the browser history was empty when I tried checking it later :S  However, man pages were also useful.

[Technology] How to Mount the Overlay Filesystem of a Live USB

So, I
wanted to demonstrate the commands in the previous post used for
resizing my file system, but of course I did that from a live USB (since
I wanted my file systems unmounted). However, I did not want to reboot
my computer just now to copy root's bash history, so I looked up how to
mount the overlay using these instructions:



https://fedoraproject.org/wiki/LiveOS_image



The simplified version would be:

1. insert USB key into computer


2. identify it
   $ journalctl --system
   mine was created as /dev/sdc1


3. mount it, creating target directories as you go:
   # mount /dev/sdc1 /tmp/sdc1


4. mount the read-only squashed file system:
   # mount /tmp/sdc1/LiveOS/squashfs.img /tmp/squashfs


5. create a block device for its ext3fs image:
   # losetup /dev/loop1 /tmp/squashfs/LiveOS/ext3fs.img -r


6. create a block device for the overlay:
   # losetup /dev/loop2 /tmp/sdc1/LiveOS/overlay-LIVE-foo


7. calculate size of loop1
   # blockdev -q --getsz /dev/loop1mine was 12582912
   


8. define a device combining the two
   # dmsetup create myliveos --table "0 12582912 snapshot 7:1 7:2 P 8"
   myliveos: my own name for the device, call it what you will
   12582912: the # of blocks I acquired from step 7, put in yours
   7:1 references loop1
   7:2 references loop2


9. mount the combined device
   # mount /dev/mapper/myliveos /tmp/myliveos

Now I can cd into /tmp/myliveos, and get my .bash_history!

[Technology] Installing Fedora 22, again: wireless Internet and resizing file systems

I have a 24GB SSD and a 500GB HDD in my computer.



A few months ago, every time my computer hibernated or slept, my file system on my SSD got corrupted, and I had to fsck it to repair it.  I ended up just creating a new partition on my HDD and installing my OS there.



Recently, I've grown reckless, and decided I wanted to have my OS on the SSD after all.  I did some SMART tests and my SSD seems to be fine, so pretending to forget about the earlier drama, I reformatted the SSD and re-installed Fedora 22.

Wireless Internet

Of course, the Internet doesn't work out of the box.  That's because of Broadcom's proprietary drivers and firmware.  Previously, I used the wl driver, using akmod-wl and kmod-wl.  My device:

03:00.0 Network controller: Broadcom Corporation BCM43228 802.11a/b/g/n

I looked at the open source driver, b43, though, and it claimed to now support it(!).  I apparently needed the firmware from the Windows driver, though.  Sigh.  Indeed, journalctl --system (instead of dmesg) reported this:

[  162.191310] b43-phy0: Broadcom 43228 WLAN found (core revision 30)

[  162.191764] b43-phy0: Found PHY: Analog 9, Type 4 (N), Revision 16

[  162.191776] b43-phy0: Found Radio: Manuf 0x17F, ID 0x2057, Revision 9, Version 1

[  162.192544] Broadcom 43xx driver loaded [ Features: PMNLS ]

[  162.192570] b43 bcma0:1: Direct firmware load for b43/ucode30_mimo.fw failed with error -2

[  162.192596] b43 bcma0:1: Direct firmware load for b43/ucode30_mimo.fw failed with error -2

[  162.192995] b43 bcma0:1: Direct firmware load for b43-open/ucode30_mimo.fw failed with error -2

[  162.193015] b43 bcma0:1: Direct firmware load for b43-open/ucode30_mimo.fw failed with error -2

[  162.193019] b43-phy0 ERROR: Firmware file "b43/ucode30_mimo.fw" not found

[  162.193022] b43-phy0 ERROR: Firmware file "b43-open/ucode30_mimo.fw" not found

[  162.193024] b43-phy0 ERROR: You must go to http://wireless.kernel.org/en/users/Drivers/b43#devicefirmware and download the correct firmware for this driver version. Please carefully read all instructions on this website.

This lead me to http://linuxwireless.sipsolutions.net/en/users/Drivers/b43/#Fedora, which has a broken link to http://linuxwireless.sipsolutions.net/en/users/Drivers/b43#other_distros further down the page.





In the end, I had to do these things

$ dnf install b43-fwcutter wget     # dnf instead of yum

$ export FIRMWARE_INSTALL_DIR="/lib/firmware"

$ wget http://www.lwfinger.com/b43-firmware/broadcom-wl-5.100.138.tar.bz2

$ tar xjf broadcom-wl-5.100.138.tar.bz2

$ sudo b43-fwcutter -w "$FIRMWARE_INSTALL_DIR" broadcom-wl-5.100.138/linux/wl_apsta.o

$ modprobe b43

And it worked.

Mounting encrypted /home partition

I cheated and used gnome-disks to make my encrypted /home partition auto-mount.  Sorry.

[Technology] GoogleCL, OAuth2, and bitrot

I have a fancy process for managing photos I take, curating them, and uploading them to PicasaWeb/Google+ Photos/Google Photos/whatever you want to call it.  It partially relies on googlecl (github, google code (old)).



However, it only authenticates using OAuth1, and that was deprecated recently.  Maybe when I'm done with school I'll try to take it over.  In the meantime, I'll be resurrecting an old libgdata tool I wrote 6 years ago on the other side of the planet.  Yay!

[Technology] Memory leaks, vala, and copying delegates

Some of the Gee data structures, like a TreeSet, can get a comparator, a GLib.CompareDataFunc comparator. 



If I define a compliant static function and hand it over to the TreeSet's constructor, I get a warning that copying delegates is not supported.  I can't really change how TreeSet handles it, and I couldn't find how I was supposed to pass it elsewhere, so I passed it over like this (not exact code):



class ApplePie : Pie {

  private GLib.CompareDataFunc get_cmp () {

    return (a,b) => { return a.foo - b.foo; };

  }

}

class CherryPie : Pie {

  private GLib.CompareDataFunc get_cmp () {

    return (a,b) => { return a.bar - b.bar; };

  }

}

class Pie {

  TreeSet ts;

  public Pie () {

    ts = new TreeSet (this.get_cmp ());

  }

}



You see, I needed different comparators for different subclasses, but I couldn't just hand over static functions.  However, somehow, passing a function instantiated in this manner to new TreeSet () resulted in an increase to the reference count for pie!   And, in fact, I had 10 different collections that made use of 3 different comparators like this.  And I went through 10,000s of pies!  I think you can start to see my problem.



The functions were causing reference cycles so none of my pies were being freed when I was finished with them, and I quickly ran out of memory. :D  Hooray!



Right now, I'm doing what I'm not supposed to, and passing static functions (copying delegates) to the new TreeSet constructor (they're passed from the subclass to the superclass through base (), so that might be a part of the problem) and things are working alright.



I'm grateful for Malloc.mallinfo ().uordblks, and for GLib.Object.ref_count for helping me see into problem today.

[Technology] emacs and vala-mode: set-difference

When using emacs 24.5 and vala-mode, I started getting this error:

"Symbol's function definition is void: set-difference"



After some googling, it turns out to be related to cc-mode using cl (CommonLisp).  The not-ideal workaround for now is to apparently include

(require 'cl)

In your ~/.emacs file.

[Technology] Bitrotting brain

TL;DR

• have an encrypted /home


• hard shut down of computer


• wouldn't boot back up to the login screen


• couldn't mount /home from emergency mode


• had to boot a Live USB key and do fsck.ext4 on / (yes, /)


• problems with mounting /home were probably more to do with emergency mode and what gets enabled for different targets via systemd


• I need to update myself on my system's work-abouts :D

Premise



I know a lot about my computer and computers.  There are a lot of people who know a lot more than me.  Relative to the rest of the population, there was a time 3.5 years ago when my knowledge was probably ranked at a higher percentile.



I haven't had the same time and motivation these past few years, though.  One of the largest complications has been my Masters.  Sure, I know a lot more about machine learning, linguistics, natural language processing, etc., but that hasn't helped me really understand my computer much more.  I was fortunate enough to participate in the Google Summer of Code 3 summers in a row, but my time was always tight for that, and full-use was probably not made of those opportunities working on GNOME.



However, today was a harsh wake-up call about the consequences of idleness.  My knowledge of init and rc and Linux's old boot sequence is still fairly strong, having grown up in Slackware on the command-line.  However, while I've read about evolving configurations and utilities since, I haven't gotten much practise in them. 



And so, it was with horror when I realised that my system would not fully boot today and I could not immediately address the issue.



Today's Predicament



My /home directory is an ext4 filesystem on an encrypted LUKS partition.  Perhaps you can immediately see what is going to go wrong.  Earlier today, my computer froze up.  That's not uncommon.  As much as I love Linux, I am never surprised when it spontaneously freezes (especially on a laptop (what with having its Internet connection and video mode and power supply changing constantly)).  After trying to get it to respond for 15s, I just held down the power button as usual and then turned it back on.



Plymouth came up, I saw the fedora logo start to fill, it was interrupted by the password prompt for my /home partition, I entered the password, and then the fedora logo filled up and - nothing.  It just waited there.  It should switch to gdm's login screen, but it wouldn't.  I could go see the terminal in the background, but I couldn't access any shells.  Just the start up log.  There was nothing obviously wrong.



Sometimes I would see this message, "EXT4-fs error (device dm-0): ext4_mb_generate_buddy:757: group 83, block bitmap and bg descriptor inconsistent: 21682 vs 21673 free clusers", but dm-0 is my root directory.  dm-3 is my home directory.



I tried to boot into rescue mode to see if I could still access all my files.  No.  After supplying my password for decryption, I would see this error message "device-mapper: table: 253:3: crypt: unknown target type

Failed to activate: Invalid argument".



That sounds bad.  I entered the emergency mode shell, and checked the journal.  This is something I only do occasionally using journalctl, as I used to directly check logs or dmesg.  Uh oh.  Here it was failing to mount /home, and it was failing at the step of "/usr/lib/systemd/systemd-cryptsetup attach luks-1496...e527 none" [LUKS name contracted].  Uh oh.



At this point I started to suspect that my encrypted partition was corrupted from my hard shut-down.  There's not much else I can do with my computer when it freezes up, but I haven't suffered from it being left in a volatile, at least until now.



I tried to manually mount it, but then realised I have almost never manually mounted a file system encrypted using LUKS.  I've used encfs, but - uh, now what?  Also, lvm2 and systemd I'm only passingly familiar with.  How am I supposed to be confident in my computer's reliability if I don't even understand the fundamental tools that house my data any more?



I double-checked that my back-up was still working (I've had back-up hard drives die on me before!) using another old computer and thankfully it was OK.  However, I didn't want to have to back-up from it if I didn't have to, as I had done work in the past 24 hours that I didn't want to lose, damn it.  (I'm so glad that I've gotten the knack of backing up every 24 hours at least!).  I also wasn't completely convinced that my data was inaccessible.  On plain text file systems, it's easy to grep or scan to reclaim many/most files, even if the file system itself is corrupted or the partition boundaries have been lost.  You can't readily do that when the file system is encrypted, but perhaps there was a way to repair it?



Also, this made me wonder, why was the default multi-user graphical mode stalling out?  If it couldn't mount /home, it should still be able to display the login screen.  I don't need a /home for that.  I tried to launch gdm from emergency mode and couldn't because of failed dependencies, one of which was for binary formats.  That seemed weird.  I didn't remember so many things failing to start back when I was booting in the normal way.  So I tried the normal way again and ... it claimed that /home mounted cleanly and even gave a plausible count of files.  (!)



I then decided to boot in single-user mode (edit the grub command for the kernel I'm booting into and add 'single' on the kernel line).  This took me through the same process as usual, which seemed to mount /home correctly, but instead of going all the way to the point where it was stalling, it stops successfully before at a command-line for root.  Hooray!



Once there, I poked around.  I couldn't immediately tell why from emergency mode I couldn't successfully mount my file systems, but there's an egregious lack of familiarity in me regarding how systemd, lvm2, and luks interact, so that's no big surprise.  I did a passive run of fsck.ext4 on my / directory (remember how dm-0 had errors reported earlier?) and yes, there seemed to be quite a lot.  I grabbed my Fedora live USB key from my drawer, re-booted into it, ran fsck on my computer's root partition, and let it fix all the errors.  This always alarms me, as I never know what data has been lost on my file system to lead to the errors in question.  There's never a guarantee that cleaning up after the errors will resolve the problem in question.



Anyway, I rebooted, I went through a normal boot and - tada, the login screen.



Conclusion



So why did I write all that?  In part to motivate me to learn more about the current state of my system to regain the glory of self-sufficiency.  And in part so the few error messages I encountered that another in my case might encounter will be documented, and searchable. :D