[Technology] Accessing files.web.uoguelph.ca from Linux when off campus

Until a few weeks ago, I was able to login remotely to my University's web hosting server.  Now, I need to connect via VPN first.



On Fedora Linux, I made sure I had vpnc installed and NetworkManager-vpnc.   Then I went to Network Manager's settings/preferences, added a new connection (+ in the bottom left), chose VNC, and chose the vpnc client.  Then I used the settings from this page:



https://www.uoguelph.ca/ccs/internet-phones/virtual-private-network-vpn/using-vpn



It beats using the creepy script from vpn.uoguelph.ca at least.

[Technology] Automating my life away

A decade ago, part of a job was to help automate creation
of a report that normally involved a human scanning through dozens of
spreadsheets each year, copying and pasting numbers and generating
graphs and tables, manually.

The place where I worked
predominantly used Perl.  (It was a science laboratory.)   My solution
was to use OpenOffice's GUI to bulk-convert the Microsoft Excel
spreadsheets (binary XLS files) to whichever file format OpenOffice used
at the time that was actually a collection of XML in a zipped
directory. (Was that .sdc?)

I then used Perl's libxml
wrapper (probably wrapping libxml2) to parse the XML into an SQLite
database.  (Or was it PostgeSQL like everything else there?)  I had
tried to use some other Perl XML library, but it was Super Slow compared
to the snappy libxml2 (written in C).  Like, a difference in many
minutes to run versus one second.

From there, I wrote
some Perl CGI scripts to automatically generate most of the tables and
graphics into HTML form, in some fashion that looked nice when imported
into OpenOffice. (Really?!)  The person who was responsible for the
report, who normally took a few weeks to do, said that it would save him
so much of the time, that it would take him now a couple days to look
at the data and write conclusions and maybe do some cleaning.  He
marveled at automation and how quickly he could be replaced.  The
automation revolution in action.

Today I used LibreOffice to convert about a hundred spreadsheets into CSV files with this one command:

find . -name "*.xls" | while read line; do
  (cd "$(dirname "$line")" &&
   libreoffice --headless --convert-to csv "$(basename "$line")");
done

I
am not sure that OpenOffice could do headless conversion back then.  I
was probably overwhelmed by all the potential everywhere to discover the
functionality if it exists.  This would have helped free a few hours of
my life.

Automation, a revolution.



UPDATE:

Converting to CSV only does the 1st sheet tab.  You apparently can't specify a different sheet to be the active sheet.  I also had to convert some other old spreadsheets for easy parsing (mostly just grep'ing through them to find data).   In that case, I used --convert-to html which was beautiful, even if one of the resultant HTML files was 138MB large.

[Technology] Starting Firefox Faster

summary: if you use pinned tabs, and they're all loading at start, despite setting tabs to not load until selected, this post is for you.

I have two hard disks, a 24GB SSD (fast!) and a spinner.  I noticed that to fully load Firefox and wait for everything to settle, it was taking about a minute and a half, back when my root directory was on the HDD.  Recently (see earlier posts), I moved my root onto the SSD.  That improved things marginally.  Firefox was now settling in around 1m12s.  I tried moving my profile onto the SSD.  Now I was down around 1m00s.  (These are averaged from multiple warm runs.)  That is still abysmal.



One thing I notice is that a private web app I wrote (look up My Daily in past posts maybe) is guarded by an .htaccess file, and I have it as a pinned tab.  In that full minute, before things finish settling, I get prompted to authenticate myself.  I then recalled that I had enabled the Firefox option to load tabs when clicked (Preferences > General > "Don't load tabs until selected"), so why is it loading?



All the pinned tabs actually seem to be loading at startup.  Grr.  Perhaps that is tunable?



I went to about:config, and sure enough, typing in "pinned" filtered down to this setting:

browser.sessionstore.restore_pinned_tabs_on_demand

defaulting to "false".  I set it to "true", and now Firefox takes <3s to open and settle.  Sure, clicking on a pinned tab now takes a normal time to load the tab, but it's better than loading GMail, Google Calendar, Blogger, Google Drive, Google Keep and my web app ALL AT ONCE.

[Technology] Resizing my encrypted /home partition

I just wrote a post about why I reinstalled Fedora 22: because I wanted root to be on the SSD, and I needed to purge a lot of old installed software anyway. :P  And how that led to trying out the open source b43 wireless driver.



This is more interesting than my wireless problem. 
Because I no longer needed a / partition on my HDD, I wanted to extend
my home partition.  It isn't as dead-simple as back a decade ago.  Back
then, I would use fdisk to delete and recreate partitions and resize2fs
to resize the file system.



Now I have layers upon
layers of useful abstraction.  I'm calling each something potentially
inaccurate, and the tool I needed in brackets. (Those with wildcards
match multiple tools with the same beginning.)



Top

• ext4 file system (resize2fs)


• Logical volume (lv*)


• Volume group (vg*)


• Physical volume (pv*)


• partitions (fdisk)

Bottom



At first I was almost a little annoyed
at the apparent complexity, but it's actually straight forward.  My
drive is still divided into multiple partitions in a partition table. 
(/dev/sda1, /dev/sda2, /dev/sda3, etc; I have 8 for various reasons).



Physical
volumes seem to correspond to those partitions, but mostly just the
useful storage ones (e.g. I don't have them for EFI or swap
partitions).  pvcreate lets me define them.



The
physical volumes are collected into volume groups.  For example,
/dev/sdb1 (my intended root) might be a part of "pantaloons-fedora-ssd"
(if I named my machine pantaloons), and /dev/sda2 and /dev/sda4 might
both be part of "pantaloons-fedora-hdd" (two large areas for my /home
that I'd like to combine later, but which are currently separated by some other, annoying partition).



The
volume group kind of looks like a single contiguous (!) entity (despite
being a collection of disparate physical partitions).  Then, I can
define logical volumes atop the volume groups.



I can
have both / and swap as separate logical volumes atop
"pantaloons-fedora-ssd" (talk about cramped - maybe I don't want to
hibernate after all), and have a single logical volume for /home atop
the "pantaloons-fedora-hdd" volume group.



This is where things get a little trippy though.  Where does the encryption happen?



For my root file system, it looks like it happens like:

partition /dev/sdb1 > LUKS > physical volume > EXT4 file system



For my home file system, it looks like this:

partitions
/dev/sda2,/dev/sda4 > two physical volumes > 1 volume group > 1
logical volume > LUKS > EXT4 file system

FS	Ext4		Ext4
(LUKS)			/dev/mapper/luks-BAR
LVs	root	swap	/home
VGs	pantaloons-fedora-ssd		pantaloons-fedora-hdd
PVs	/dev/mapper/luks-FOO		/dev/sda2	/dev/sda4
(LUKS)	/dev/mapper/luks-FOO		/dev/sda2	/dev/sda4
partitions	/dev/sdb3
disks	/dev/sdb (SSD)		/dev/sda (HDD)

The home partition and its LUKS (using
cryptsetup/dm-crypt) was originally configured through whichever tool
Fedora provided years ago (preupgrade? fedup? anaconda?), while the
configuration on my SSD was what was recommended in F22.



Basically, I had to (with steps that actually increase space emboldened):

• define a new partition in the partition table, 


• define a new Physical Volume (PV) over top it, 


• extend my Volume Group (VG) to include the new Physical Volume


• unmount the file system I was going to extend, 


• close its LUKS setup


• extend the file system's Logical Volume (LV) overtop the new space in the Volume Group


• reopen the LUKS setup


• resize the LUKS setup (are these two out of order? hope not!), and


• resize the Ext4 partition

A bit more detail, here are the steps I used on my system (from
my bash_history, probably missing some steps; I think I also used
lvremove
or pvremove to remove LVs and PVs related to the old home partition, oh
well!):

• # cfdisk, to delete the old root on the /dev/sda4 partition and recreate it


• # pvcreate /dev/sda4 (creates a physical volume)


• # vgextend pantaloons-fedora-hdd /dev/sda4 (add the PV into the VG increasing its overall size)


• make sure /home isn't mounted


• # fsck.ext4 -C 0 -f /dev/mapper/luks-BAR (let's make sure the inodes are fine on our file system before messing with it)


• # cryptsetup luksClose luks-BAR (close LUKS while we change the underlying LV)


• # lvextend -L +61G /dev/pantaloons-fedora-hdd/home (increase the size of our LV within the VG)


• # lvextend -L +242M /dev/pantaloons-fedora-hdd/home (catch a little more space)


• there is probably a way to just extend it to the full space available, maybe read the man page


• # cryptsetup luksOpen /dev/pantaloons-fedora-home luks-BAR


• # cryptsetup resize luks-BAR (resizing the LUKS setup)


• # fsck.ext4 -C 0 -f /dev/mapper/luks-BAR (for paranoia maybe)


• # resize2fs /dev/mapper/luks-BAR (finally, let's resize our Ext4 file system)

Sadly, I can't cite the sources I pieced some parts together from, because I did this form a live USB key which is supposed to keep history as persistent storage but the browser history was empty when I tried checking it later :S  However, man pages were also useful.

[Technology] How to Mount the Overlay Filesystem of a Live USB

So, I
wanted to demonstrate the commands in the previous post used for
resizing my file system, but of course I did that from a live USB (since
I wanted my file systems unmounted). However, I did not want to reboot
my computer just now to copy root's bash history, so I looked up how to
mount the overlay using these instructions:



https://fedoraproject.org/wiki/LiveOS_image



The simplified version would be:

1. insert USB key into computer


2. identify it
   $ journalctl --system
   mine was created as /dev/sdc1


3. mount it, creating target directories as you go:
   # mount /dev/sdc1 /tmp/sdc1


4. mount the read-only squashed file system:
   # mount /tmp/sdc1/LiveOS/squashfs.img /tmp/squashfs


5. create a block device for its ext3fs image:
   # losetup /dev/loop1 /tmp/squashfs/LiveOS/ext3fs.img -r


6. create a block device for the overlay:
   # losetup /dev/loop2 /tmp/sdc1/LiveOS/overlay-LIVE-foo


7. calculate size of loop1
   # blockdev -q --getsz /dev/loop1mine was 12582912
   


8. define a device combining the two
   # dmsetup create myliveos --table "0 12582912 snapshot 7:1 7:2 P 8"
   myliveos: my own name for the device, call it what you will
   12582912: the # of blocks I acquired from step 7, put in yours
   7:1 references loop1
   7:2 references loop2


9. mount the combined device
   # mount /dev/mapper/myliveos /tmp/myliveos

Now I can cd into /tmp/myliveos, and get my .bash_history!

[Technology] Installing Fedora 22, again: wireless Internet and resizing file systems

I have a 24GB SSD and a 500GB HDD in my computer.



A few months ago, every time my computer hibernated or slept, my file system on my SSD got corrupted, and I had to fsck it to repair it.  I ended up just creating a new partition on my HDD and installing my OS there.



Recently, I've grown reckless, and decided I wanted to have my OS on the SSD after all.  I did some SMART tests and my SSD seems to be fine, so pretending to forget about the earlier drama, I reformatted the SSD and re-installed Fedora 22.

Wireless Internet

Of course, the Internet doesn't work out of the box.  That's because of Broadcom's proprietary drivers and firmware.  Previously, I used the wl driver, using akmod-wl and kmod-wl.  My device:

03:00.0 Network controller: Broadcom Corporation BCM43228 802.11a/b/g/n

I looked at the open source driver, b43, though, and it claimed to now support it(!).  I apparently needed the firmware from the Windows driver, though.  Sigh.  Indeed, journalctl --system (instead of dmesg) reported this:

[  162.191310] b43-phy0: Broadcom 43228 WLAN found (core revision 30)

[  162.191764] b43-phy0: Found PHY: Analog 9, Type 4 (N), Revision 16

[  162.191776] b43-phy0: Found Radio: Manuf 0x17F, ID 0x2057, Revision 9, Version 1

[  162.192544] Broadcom 43xx driver loaded [ Features: PMNLS ]

[  162.192570] b43 bcma0:1: Direct firmware load for b43/ucode30_mimo.fw failed with error -2

[  162.192596] b43 bcma0:1: Direct firmware load for b43/ucode30_mimo.fw failed with error -2

[  162.192995] b43 bcma0:1: Direct firmware load for b43-open/ucode30_mimo.fw failed with error -2

[  162.193015] b43 bcma0:1: Direct firmware load for b43-open/ucode30_mimo.fw failed with error -2

[  162.193019] b43-phy0 ERROR: Firmware file "b43/ucode30_mimo.fw" not found

[  162.193022] b43-phy0 ERROR: Firmware file "b43-open/ucode30_mimo.fw" not found

[  162.193024] b43-phy0 ERROR: You must go to http://wireless.kernel.org/en/users/Drivers/b43#devicefirmware and download the correct firmware for this driver version. Please carefully read all instructions on this website.

This lead me to http://linuxwireless.sipsolutions.net/en/users/Drivers/b43/#Fedora, which has a broken link to http://linuxwireless.sipsolutions.net/en/users/Drivers/b43#other_distros further down the page.





In the end, I had to do these things

$ dnf install b43-fwcutter wget     # dnf instead of yum

$ export FIRMWARE_INSTALL_DIR="/lib/firmware"

$ wget http://www.lwfinger.com/b43-firmware/broadcom-wl-5.100.138.tar.bz2

$ tar xjf broadcom-wl-5.100.138.tar.bz2

$ sudo b43-fwcutter -w "$FIRMWARE_INSTALL_DIR" broadcom-wl-5.100.138/linux/wl_apsta.o

$ modprobe b43

And it worked.

Mounting encrypted /home partition

I cheated and used gnome-disks to make my encrypted /home partition auto-mount.  Sorry.