Automating Backup on Raspberry Pi

I had been thinking of automating my Raspberry Pi backup for a long time but never really got around in doing so. Until, one day the SD card failed and my only backup was from 1 year ago! It was terrible but I rebuilt it this time I rebuilt it onto an old external HDD. You would think that this incident would make me start having a strict backup regime but sadly it didn’t. 6 months later, that external HDD failed! But fortunately, I manage to painfully salvage the data.

Side note: If your pi is constantly booting into emergency mode, you should really consider doing a full backup and ready for it to fail! My pi was booting into emergency mode for a good 3 months before it finally gave up.

My original backup script was a very rudimentary script. As I am booting it off a USB, I was able to boot back into the original SD card and perform a dd and gzip the img of the USB. This works as the USB is small (say 16GB drive) and gzip compresses it down to a few GB of data (depending on how much data there is). But as I move towards external HDD (which gives you way better speeds), dd just doesn’t cut it anymore. That and the original script was very manual process. One had to manually reboot into SD card and run the backup script, come back again in an hour or so and reboot back into USB drive. Hence I started researching and finally came up with the following script.

What this script does is that it perform the following backup to the following locations:

  1. BackupLocal – This is a large partition in the external HDD (same drive as the rootfs). I have the following in BackupLocal. What it gives me is that now I have daily backups for the past week, then weekly backup for the past 4 weeks followed by 6 months worth of monthly backup. I don’t think I need anything more refined than this at the moment. Although I can possibly have daily backups for the past 6 months without incurring too much data storage space (via hard linking. More on that later).
    1. backup.daily.1 – backup.daily.6
    2. backup.weekly.1 – backup.weekly.4
    3. backup.monthly.1 – backup.monthly.6
  2. BackupHot – This is another USB flash drive. This drive contains most up to date data and is designed to allow me to quickly swap the boot point to this drive while I source myself a new HDD.
  3. BackupRemote – This is where I store .tar.gz files from BackupLocal to. This is a remote drive (a NAS unit with RAID 1 configuration) so that in the event the BackupLocal drive fails, I have some data stored safely somewhere else.

This is the backup script. It performs daily backup at 3am everyday. If it is not 3am, it performs a hot backup. If a parameter is added when calling the script (can be anything), it will also perform weekly/monthly rotation if it is Saturday or first day of the month. Daily and hot backups are always done if you call the script manually

#!/bin/bash
# script to synchronise Pi files to backup
# Parameter $1 allows for rotating weekly and monthly records. Used to just in case invoking manual backup

backupLocalDirAwk=backupLocal
backupHotDirAwk=backupHot
backupRemoteDirAwk=backupRemote
backupExcludeList=/home/pi/backup-rsync-exclude.txt

backupLocalDir=/mnt/$backupLocalDirAwk/
backupHotDir=/mnt/$backupHotDirAwk/
backupRemoteDir=/mnt/$backupRemoteDirAwk/

backupDaily="$backupLocalDir"backup.daily.
backupWeekly="$backupLocalDir"backup.weekly.
backupMonthly="$backupLocalDir"backup.monthly.
hourToCreateDailyBackup=03    # needs to be in 24 hour format (00-23)



# ======= CODE STARTS HERE =======
# make sure we're running as root
if (( `id -u` != 0 )); then { echo "Sorry, must be root.  Exiting..."; exit; } fi

DOW=$(date +%u)
DOM=$(date +%d)
Hour=$(date +%H)    # Hour in 24 hour format. (00-23)

dailyRotated=0
weeklyRotated=0
monthlyRotated=0

invokedByCron=$1

echo "Backup Started " $(date)


# Parameters:
# 1: Directory to tar (without the number suffix)
# 2: Number suffix for the directory to backup
function createTar {
    backupTar="$1""$2".tar.gz
    if [ -f "$backupTar" ]; then
        rm "$backupTar"
    fi

    tarArgs="-zcp"
    if [ -z "$invokedByCron" ]; then
        tarArgs="$tarArgs""v"
    fi
    echo "Creating TAR: " $backupTar
    tar "$tarArgs" --file="$backupTar" --directory="$1""$2" .
    
    # Move TAR file to remote location
    BACKUPRemote_MOUNTED=$(mount | awk "/$backupRemoteDirAwk/"'{print $6}' | grep "rw")
    if [ "$BACKUPRemote_MOUNTED" ]; then
        echo "Moving $backupTar to $backupRemoteDir"
        mv -f "$backupTar" "$backupRemoteDir"
    else
        echo "$backupRemoteDir not found or not in rw mode"
    fi
    # Untar command: tar -zxvpf /filename.tar.gz
}

# Parameters:
# 1: Destination directory
# 2: Linked Directory (Optional. If not specified, no hard linking is done and no timestamp to be noted in backup)
function syncRoot {
    rsyncArgs="-axH"
    if [ -z "$invokedByCron" ]; then
        rsyncArgs="$rsyncArgs""v"
    fi
    echo "Commencing Backup to " $1
    if [ -z $2 ]; then
        rsync "$rsyncArgs" --delete-during --delete-excluded --exclude-from="$backupExcludeList" / "$1"
    else
        rsync "$rsyncArgs" --delete-during --delete-excluded --link-dest="$2" --exclude-from="$backupExcludeList" / "$1"
        echo $(date) > "$1"/backupDate.txt
    fi
}

BACKUPLocal_MOUNTED=$(mount | awk "/$backupLocalDirAwk/"'{print $6}' | grep "rw")
BACKUPHot_MOUNTED=$(mount | awk "/$backupHotDirAwk/"'{print $6}' | grep "rw")
if [ "$BACKUPLocal_MOUNTED" ]; then
    echo "Found $backupLocalDir in rw mode"
    
    if [ ! -z "$invokedByCron" ] && [ "$Hour" = "$hourToCreateDailyBackup" ]; then
        # Rotate Monthly backups
        if [ "$DOM" = "01" ]; then
            echo "Rotating Monthly"
            rm -rf "$backupMonthly"6
            mv "$backupMonthly"5 "$backupMonthly"6
            mv "$backupMonthly"4 "$backupMonthly"5
            mv "$backupMonthly"3 "$backupMonthly"4
            mv "$backupMonthly"2 "$backupMonthly"3
            mv "$backupMonthly"1 "$backupMonthly"2
            mv "$backupWeekly"4 "$backupMonthly"1
            monthlyRotated=1
        fi

        # Rotate Weekly backups
        if [ "$DOW" = "6" ]; then
            echo "Rotating Weekly"
            if [ "$monthlyRotated" = 0 ]; then
                rm -rf "$backupWeekly"4
            fi
            mv "$backupWeekly"3 "$backupWeekly"4
            mv "$backupWeekly"2 "$backupWeekly"3
            mv "$backupWeekly"1 "$backupWeekly"2
            mv "$backupDaily"6 "$backupWeekly"1
            weeklyRotated=1
        fi
    fi

    # Rotate Daily backups
    if [ "$Hour" = "$hourToCreateDailyBackup" ] || [ -z "$invokedByCron" ]; then
        echo "Rotating Daily"
        if [ "$weeklyRotated" = 0 ]; then
            rm -rf "$backupDaily"6
        fi
        mv "$backupDaily"5 "$backupDaily"6
        mv "$backupDaily"4 "$backupDaily"5
        mv "$backupDaily"3 "$backupDaily"4
        mv "$backupDaily"2 "$backupDaily"3
        mv "$backupDaily"1 "$backupDaily"2
        mkdir "$backupDaily"1
        dailyRotated=1
    fi

    # Perform backup
    if [ "$dailyRotated" = 1 ]; then
        echo "Commencing Daily Backup"
        syncRoot "$backupDaily"1 "$backupDaily"2
    fi

    if [ "$BACKUPHot_MOUNTED" ]; then
        echo "Found $backupHotDir in rw mode"
        echo "Commencing Hot Backup"
        syncRoot "$backupHotDir"
    fi

    # Create TARs
    # Create Daily TARs
    if [ "$dailyRotated" = 1 ]; then
        echo "Creating Daily TARs"
        createTar "$backupDaily" 1
    fi

    # Create Weekly TARs
    if [ "$weeklyRotated" = 1 ]; then
        echo "Creating Weekly TARs"
        createTar "$backupWeekly" 1
        createTar "$backupWeekly" 3
    fi

    # Create Monthly TARs
    if [ "$monthlyRotated" = 1 ]; then
        echo "Creating Monthly TARs"
        createTar "$backupMonthly" 1
        createTar "$backupMonthly" 2
        createTar "$backupMonthly" 4
        createTar "$backupMonthly" 6
    fi

    echo "Backup Completed " $(date)
    echo ""
else
    echo "Backup drive not available or not writable" $(date)
fi


And this is the exclude list


#this is the backup-rsync-exclude.txt file

/proc/*
/sys/*
/dev/*
/boot/*
/tmp/*
/run/*
/media/*

.Trashes
._.Trashes
.fseventsd
.Spotlight-V100
.DS_Store
.AppleDesktop
.AppleDB
Network Trash Folder
Temporary Items

.bash_history
/etc/fake-hwclock.data
/var/lib/rpimonitor/stat/

At the end of all these, you got to create a cron task so that it gets run hourly


# Executes at 3am, 9am, 12pm, 3pm, 6pm everyday to backup system
# m h dom mon dow user command
0 3,9,12,15,18 * * * root bash /home/pi/backup.sh 1 >> /home/pi/backup_log.txt

Posted in Raspberry Pi | Tagged , , , , , | Leave a comment

Disable IPv6 on Raspberry Pi

With the new Raspbian Jessie, IPv6 is enabled by default. With my router partially supporting IPv6 and most my devices are still on IPv4, there wasn’t much use for having IPv6 enabled (that and I had to constantly remember which IP to use when connecting to my pi since DNS naming doesn’t get allocated to IPv4 for the pi for some reason when IPv6 is enabled on it).

So I set out a way to get IPv6 disabled and this is what I had to do to fully disable it along with the driver

(Optional. I didn’t need it) Add “ipv6.disable=1” (without the quotes) to /boot/cmdline.txt


cat /etc/modprobe.d/ipv6.conf
# Don't load ipv6 by default
alias net-pf-10 off
#alias ipv6 off
# added to disable ipv6
options ipv6 disable_ipv6=1
# added to prevent ipv6 driver from loading
blacklist ipv6

Then followed by a reboot of the system and that’s it. IPv6 no longer loads on Raspberry Pi.

Also, if you have nginx on your pi as well, make sure the following line is commented off the default config if not nginx will fail to load as it is trying to listen on IPv6 as well (and of course fail horribly).


cat /etc/nginx/sites-available/default
...
#listen [::]:80 default_server;
...

Source

Posted in Raspberry Pi | Tagged , , | Leave a comment

AirPlay (Shairport) on Raspberry Pi 2

I’ve got a decent speakers that is connected to my PC via a USB sound card and had always used my PC to play music. Now that I’ve got a Raspberry Pi which is always on, I thought, why waste power by turning on my PC all the time!

I’ve got a mix of iOS devices and Windows Phone devices at home and iOS will work well on AirPlay and WP devices will need to be on Bluetooth (which I will try to set up later).

Now Shairport seems to be the go-to app for Airplay on rpi and there are a couple of different versions/forks on Github. The one I used is:

Shairport By Abrasive

I’ve found some help here as well.First, we change alsa.conf to remove some warnings that comes up when running the script.


sudo nano /usr/share/alsa/alsa.conf

And change


pcm.front cards.pcm.front

to


pcm.front cards.pcm.default

If we are using USB sound cards, you will need to comment out a line in alsa-base.conf


sudo nano /etc/modprobe.d/alsa-base.conf

Find this and comment it out with a “#” like this


#options snd-usb-audio index=-2

This will allow alsa to load up USB audio instead of suppressing it to lower priority

Next, we install Shairport

#Installing relevant packages
sudo apt-get install libao-dev libssl-dev git avahi-utils libwww-perl
sudo apt-get install libcrypt-openssl-rsa-perl libio-socket-inet6-perl libmodule-build-perl

#Getting perl-net-sdp
git clone https://github.com/njh/perl-net-sdp.git
cd perl-net-sdp
perl Build.PL
./Build
./Build test
sudo ./Build install
cd ..

#Abrasive's Shairport
git clone https://github.com/abrasive/shairport.git
cd shairport
make

#Change "AirPi" to something else if you like as long as there's no space in between
./shairport.pl -a AirPi

Now, you should see “AirPi” as an audio device from iOS AirPlay section. If that didn’t work, then we may have missed a step.

Next, we want to make it auto load when rpi starts up.


cd shairport
make install
cp scripts/debian/init.d/shairport /etc/init.d/shairport
cd /etc/init.d
chmod a+x shairport
update-rc.d shairport defaults

Next, we have to modify the contents of /etc/init.d/shairport to add our Airplay name and we can also change the buffer size of Shairport.


sudo nano /etc/init.d/shairport

#change the following parameter to your suiting
AP_NAME="AirPi"
BUFFER_FILL=30

The default size is 220 frames which comes up to be around 2s. From my experimenting, I find 30 frames is acceptable with minimal to no packet drops and with a fast response.

You can find out if it is dropping packets by looking at /var/log/shairport.err. If it is dropping packets, you will see lots of “Warning: Buffer underrun”. In which case, modify BUFFER_FILL to your liking.

Lastly, for streaming from PC to Shairport, we will need a program called TuneBlade. It is free if streaming to shairport but there’s some restrictions if you were to stream it to Airplay devices unless you get the full version.

Posted in Audio Streaming, Raspberry Pi | Tagged , , , | Leave a comment

Getting Fitbit to Sync on Raspberry Pi

Galileo is the program that you want to use for synchronizing all your trackers. It is written in python so first, you will need to install python

sudo apt-get install python-usb python-pip

 

Then, install Galileo

sudo pip install galileo

Next, edit permissions on the USB dongle so that you don’t need to run it as root all the time

sudo nano /etc/udev/rules.d/99-fitbit.rules

Add this into that file

SUBSYSTEM=="usb", ATTR{idVendor}=="2687", ATTR{idProduct}=="fb01", SYMLINK+="fitbit", MODE="0666"

Next, configure crontab to run it every 30 mins

sudo crontab -e
0,30 * * * * /usr/local/bin/galileo --no-dump

Side note:

Apparently, the latest pyusb (1.0.0b2) depreciated an API that is used in Galileo. The temporary fix would be to downgrade pyusb to 1.0.0b1


sudo pip uninstall -y pyusb
sudo pip install pyusb==1.0.0b1

Posted in Raspberry Pi | Tagged , , , , , | 1 Comment

Setting up VPN Server on Raspberry Pi 2

I have been running Softether for sometime now on my Win7 PC. Now that I have a Raspberry Pi, I thought I would just install that onto it and no longer need to have my PC on 24/7

Another benefit of Softether is that it supports so many different protocols. From OpenVPN to L2TP!

My current Raspberry Pi is running Raspbian Wheezy. If you were to refer to the Softether’s Linux setup page, it shows that you will need a plethora of libs and utils to get Softether to install on a Linux machine. Raspbian comes with it all.

First, get the ARM EABI 32bit version from Softether’s download page

Decompress it with


tar zxvf softether_file.tar.gz

It should then create a new folder called “vpnserver”. Go into that folder and compile it using


sudo make

Then go through the licenses etc and it should start compiling. At the end of the compilation, it will say that “All checks passed…”

Now, move the folder to somewhere else and change the permissions.


sudo mv vpnserver /usr/local

cd /usr/local/vpnserver

sudo chmod 600 *

sudo chmod 700 vpncmd vpnserver

Now, create a script (at /etc/init.d/vpnserver) to auto start vpnserver when you first boot up rpi.


sudo nano /etc/init.d/vpnserver

In it, paste the following script.

#!/bin/sh
### BEGIN INIT INFO
# Provides: softether-vpnserver
# Required-Start: $network $remote_fs $syslog
# Required-Stop: $network $remote_fs $syslog
# Should-Start: network-manager
# Should-Stop: network-manager
# X-Start-Before: $x-display-manager gdm kdm xdm wdm ldm sdm nodm
# X-Interactive: true
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
# Short-Description: SoftEther VPN service
### END INIT INFO

# chkconfig: 2345 99 01
# description: SoftEther VPN Server
DAEMON=/usr/local/vpnserver/vpnserver
LOCK=/var/lock/vpnserver
test -x $DAEMON || exit 0
case "$1" in
start)
$DAEMON start
touch $LOCK
;;
stop)
$DAEMON stop
rm $LOCK
;;
restart)
$DAEMON stop
sleep 3
$DAEMON start
;;
*)
echo "Usage: $0 {start|stop|restart}"
exit 1
esac
exit 0

Now, we will need to change the permissions so that the script will run on start up.

sudo chmod 755 /etc/init.d/vpnserver
sudo update-rc.d vpnserver defaults

At this point, rpi may complain about  missing LSB tags and overrides. Don’t worry, it is just insserv not happy that the top of that script does not have the optional init info. It will still run.

Updated with LSB tags so everything should be happy now 🙂

Now, to start, stop or restart vpnserver manually, you can do this:

sudo /etc/init.d/vpnserver start       #to start vpn
sudo /etc/init.d/vpnserver stop        #to stop vpn
sudo /etc/init.d/vpnserver restart   #to restart vpn

But before we can continue on setting up the nitty gritty of the vpn server, we will need to set a password for the admin account.

sudo /usr/local/vpnserver/vpncmd

Choose 1 and keep hitting enter for default settings till you get a prompt where you type in

ServerPasswordSet

Type in the password and hit enter.

Now, you can proceed with setting up Softether on your Windows machine via the use of “SoftEther VPN Server Manager for Windows”. In fact Softether recommends that you do your configuration there.

As usual, remember to configure relevant port forwarding so that you can access your vpn over internet 🙂

Side note: Remember to enable Secure NAT. If not, you wouldn’t be able to tunnel your traffic through it!

Posted in Raspberry Pi | Tagged , , , , | 9 Comments

RPi2 – Wifi Dropping Out

Found that wifi (Adafruit 814 running on RTL8192CU chip) on rpi2 drops out when on idle and I had to constantly ping it to maintain ssh connection.

A quick google got me this:

Create and edit a new file in /etc/modprobe.d/8192cu.conf


sudo nano /etc/modprobe.d/8192cu.conf

and paste the following in


# Disable power saving
options 8192cu rtw_power_mgnt=0 rtw_enusbss=1 rtw_ips_mode=1

Then reboot with sudo reboot

Essentially, it disables power saving on the wifi chip so that it maintains connection.

Posted in Raspberry Pi | Tagged , , , , | Leave a comment

Setting Up Raspberry Pi 2 (Including Moving rootfs to USB Drive)

Finally got myself a Raspberry Pi! And can’t wait to set it up 🙂

RPi had a reputation of corrupting SD cards whether it is due to running a desktop OS on an SD card wearing it down or improper shutdowns (considering most of the time it runs headless) by yanking the power cord, I have decided to reduce my chances by moving the rootfs to an external USB stick.

Parts:

  • Raspberry Pi 2
  • 8GB Class 10 SD card. Though, anything above 4GB Class 4 should do as we are only using it for boot
  • 2A power supply (with up to 1.2A available for USB hub, its best to get 2A. Apparently anything above 2A is a waste as the polyfuse is rated at 2A anyway)
  • USB Wi-Fi Adapter (RTL8188CUS or RTL8192CU chipsets. Mine is an Adafruit 814  on RTL8192CU)
  • (optional) Enclosure
  • (optional) HDMI cable

Setup:

  • Download NOOBS (New Out Of Box Software) or Raspbian (RPi version of Debian)
  • Use SDFormatter to format your SD card (use option: Quick Erase + Size Adjustment On)
  • Unzip and copy the files across if you are using NOOBS
  • or use Win32DiskImager to copy the .img file on a bit level. (Win32DiskImager is also useful for backing up your SD cards as it does not care what is on your card when you need to restore it, it will just overwrite everything and restore back to its former glory)
  • Pop that card in the Pi and boot it up!
  • Once it is all booted up, find out the IP address and use PuTTY to login remotely.

Moving rootfs to USB Stick (ref, ref2):

Pre-start Checks:

Connect to internet via ethernet (as we have not yet configured our Wi-Fi networks)

Check that the system detects your USB flash drive by using this command and you should something like this with a mention of the flash drive


pi@raspberrypi ~ $ lsusb
Bus 001 Device 002: ID 0424:9514 Standard Microsystems Corp.
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 001 Device 003: ID 0424:ec00 Standard Microsystems Corp.
Bus 001 Device 004: ID 0930:6544 Toshiba Corp. Kingston DataTraveler 2.0 Stick (2GB)
Bus 001 Device 005: ID 0bda:8176 Realtek Semiconductor Corp. RTL8188CUS 802.11n WLAN Adapter

If this is the only USB flash drive you have plugged in, it would be called “sda”. But we can check by doing this


pi@raspberrypi ~ $ tail /var/log/messages

...

Jun  6 21:19:35 raspberrypi kernel: [   23.974941]  sda: sda1

....

with that, the USB can be called


/dev/sda

with the first partition known as


/dev/sda1

Partitioning USB Stick:

First, we gotta get gdisk since we will be formatting this partition to the GPT format


sudo apt-get install gdisk

Then partition the relevant drive (in this case, /dev/sda)


sudo gdisk /dev/sda

  • First, delete all partitions on the drive by typing in “d” (for delete)
  • Then, “n” for new
  • Finally “w” for write
  • And “i” for info when it is all done
  • At anytime, you can quit by doing “q”

After that’s all been done, we should see something like this


pi@raspberrypi ~ $ sudo gdisk /dev/sda
GPT fdisk (gdisk) version 0.8.5

Partition table scan:
MBR: protective
BSD: not present
APM: not present
GPT: present

Found valid GPT with protective MBR; using GPT.

Command (? for help): i
Using 1
Partition GUID code: xxxxxxx (Linux filesystem)
Partition unique GUID: xxxxSome-GUIDxxxx
First sector: 2048 (at 1024.0 KiB)
Last sector: 15155166 (at 7.2 GiB)
Partition size: 15153119 sectors (7.2 GiB)
Attribute flags: 0000000000000000
Partition name: 'Linux filesystem'

Keep note of the “Partition unique GUID” so that you can add that into /boot/cmdline.txt


sudo nano /boot/cmdline.txt

Change the contents to look like this


pi@raspberrypi ~ $ cat /boot/cmdline.txt
dwc_otg.lpm_enable=0 console=ttyAMA0,115200 console=tty1 root=PARTUUID=xxxxSome-GUIDxxxx rootfstype=ext4 elevator=deadline rootwait rootdelay=5

Note the change of “root=” and the addition of “rootdelay=5”. This rootdelay is to tell the system to wait 5s on boot time to allow time for the USB to be mounted properly and ready for action.

Now, we format and rsync the drives


sudo mke2fs -t ext4 -L rootfs /dev/sda1
sudo mount /dev/sda1 /mnt
sudo rsync -axv / /mnt

While waiting for it to complete rsync, we can create a new SSH session and start working on other things such as editing the fstab


pi@raspberrypi ~ $ sudo tune2fs -l /dev/sda1
tune2fs 1.42.5 (29-Jul-2012)
Filesystem volume name: rootfs
Last mounted on: /
Filesystem UUID: xxxx-Some-FS-UUID-xxxx
Filesystem magic number: 0xEF53
...

Keep note of the Filesystem UUID and modify /etc/fstab to look something like this


/dev/disk/by-uuid/xxxx-Some-FS-UUID-xxxx / ext4 defaults,noatime 0 1

#/dev/mmcblk0p2 / ext4 defaults,noatime 0 1

We commented out the original root mount point and add our new mounting point.

Once that is all done (make sure the rsync is done too!) then you can finally reboot to get all the changes in place.


sudo reboot

Update/Upgrade/Update:

Once that is all done, you can finally update the entire system


sudo apt-get update

sudo apt-get upgrade

sudo rpi-update

Wi-Fi Setup:

Easiest way to do this is to boot into GUI using this command


startx

and add your Wi-Fi password manually.

Backup, Backup, Backup:

Ah, that all take some effort, lets back them up using Win32DiskImager!

But first, remember to shutdown properly using


sudo halt

Enjoy 🙂

Posted in Raspberry Pi | Tagged , | 1 Comment

iPad mini

I was never a big fan of Apple but last year’s Christmas (2012), my partner gifted me with an an iPad mini.

I had been thinking about getting a tablet and had considered the Nexus 7 for its price and its rich features but it was a little too small for my liking and was waiting for Google to release a 10″ version of Nexus tablet (which in my opinion isn’t that all attractive when compared to Nexus 7). With this 2 sizes, I thought that the 10″ one is too big and the 7″ one is way too small (in terms of aspect ratio). In this aspect, iPad mini really addressed my “requirements”.

With iOS6, it comes with AirPrint and I had been trying to get it to print stuff to PDF via AirPrint to my ReadyNAS as PDF and save it to DropBox but till now, I still have not found anything quite helpful. Although I found quite a bit on having a PC configured as a “print server” for the AirPrint protocol. But the whole point of having a tablet for me is to rely less on running the PC and more on tablet. Hence this was not the solution that I’m looking for (ie: why use tablet to print when the PC is already turned on).

If anyone knows how to get AirPrint to print to PDF via ReadyNAS, please let me know. Like wise, I’ll post my know how of I ever figure out a way to get it working on my ReadyNAS.

Posted in Uncategorized | Tagged , , | Leave a comment

Real Time HTML Charting from CSV Data Source

My friend is working on monitoring UAV’s health in real time and needed some programming help. So far, we have connected a bunch of vibration sensors to an Arduino. The Arduino in turns saves the vibration data onto a on board SD card and until now, it is only a CSV sitting there waiting to be transferred to a computer for charting.

Since we were got a WiFi module. I figured why not have the Arduino host a simple html page that charts the data. That way, we will be able to see what’s going on when a computer is in range.

A quick Google landed me with Google charts. Google charts is an impressive charting solution. However, it only accepts JSON objects as input.

For Google charts to work, I’ll need to cover the CSV files into JSON objects that complies with its API. I didn’t want to spend too much time in converting CSV to JSON so I moved on.

Next, I found Highcharts. Highcharts have an extensive API and examples to help me quickly get this up and running. Best of all, it supports CSV files as an input!

I went though the documentations and gotten it to chart. As a bonus, I manage to have the chart automatically update after a time interval without redrawing the entire chart or refresh the page. The only thing I wasn’t able to figure out is how to make the Javascript more efficient in getting new values from the file without loading the entire file, split the lines and jump to the line where it last read. Also didn’t want the Arduino to deal too much with charting since it is busy enough trying to grab as much data as possible as fast as possible.

I also managed to code it in such a way where the same html file can be used to get the CSV file path from its URL. This would allow the same script to be able to chart different does without any modification to the script.

The only downside to this is that the larger the CSV file is, the less responsive the chart can be (considering you will need to load the entire file to grab the last few data entry). But all things (think about the restrictions) considered, I think this is a good win.

Something to improve on next time is to get Arduino to purge the old data files so that the new data gets stored in a leaner and smaller file.

Codes

First, since the Arduino is not connected to the internet, it would need to have all the dependencies downloaded to its SD Card. The code below shows the .js files that needs to be included.

<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<title>Chart</title>

<script type="text/javascript" src="jquery-1.8.2.min.js"></script>
<script src="highcharts.js"></script>
<script src="exporting.js"></script>

Next, I needed some variables to make sure that we don’t overcrowd the screen with data. and will be used to shift the graph left and remove older values off the graph (the older values will still exist in the CSV file)


/*************** Variables ****************/
var refreshIntervalMS = 1000;    // how frequent do you want to update the chart.
var csvSamplingRateMS = 1000;    // whats the sampling rate of the data written into the CSV
var chartMins = 15;    // how many minutes of data do you want to display
var maxPointsInGraph = chartMins*60*1000/csvSamplingRateMS; //last 15 mins data (collected every 1000ms)
/************ End of Variables ************/

Then we get to some window formatting codes. These codes modify the title and resize or fit the graph according to the screen size.

var c = [];
var d = [];
var docTitle = 'Chart for ' + getURLParameter("data")
var lastLine = 0;
var chart;

var winW = 630, winH = 460;
if (document.body && document.body.offsetWidth) {
  winW = document.body.offsetWidth;
  winH = document.body.offsetHeight;
}
if (document.compatMode=='CSS1Compat' && document.documentElement && document.documentElement.offsetWidth ) {
  winW = document.documentElement.offsetWidth;
  winH = document.documentElement.offsetHeight;
}
if (window.innerWidth && window.innerHeight) {
  winW = window.innerWidth;
  winH = window.innerHeight;
}
winW = winW - 20;
winH = winH - 20;

document.title = docTitle;

Then we have the request new data function that grabs the CSV file and look for updated data. This is a simple function that remembers the last line it read previously and continue from there and assuming all data from that point is new. If there are no new data, the last data point will be re-added so that a long flat line will be shown on the graph after prolonged period without new data.

This function is essentially an infinite loop. Once it’s called, it will execute, sleep for a predetermined time (from the variables set above) then executes again.

function requestNewData() {
  $.get(getURLParameter("data"), function(data) {
    var lines = data.split('\n');
    for(i = lastLine; i < lines.length; i++) {
      if(i > 0) {
        var items = lines[i].split(',');
        chart.series[0].addPoint(parseInt(items[1]), false, chart.series[0].data.length > maxPointsInGraph);
      }
      lastLine = i;
    }
    chart.redraw();
    setTimeout(requestNewData, refreshIntervalMS);
  });
}

Then the Highcharts options which includes a call to requestNewData()

var options = {
  chart: {
    renderTo: 'chart',
    defaultSeriesType: 'spline',
    zoomType: 'x',
    spacingRight: 20,
    width: winW,
    height: winH,
    events: {
      load: requestNewData
    }
  },
  title: {
    text: docTitle
  },
  subtitle: {
    text: document.ontouchstart === undefined ? 'Click and drag in the plot area to zoom in' : 'Drag your finger over the plot to zoom in'
  },
  xAxis: {
    title: {
      text: 'Time Measurement'
    },
    categories: c
  },
  yAxis: {
    title: {
      text: 'Sensor Readings'
    }
  },
  tooltip: {
    shared: true
  },
  legend: {
    enabled: false
  },
  plotOptions: {
    lineWidth: 1,
    marker: {
      enabled: false,
      states: {
        hover: {
          enabled: true,
          radius: 5
        }
      }
    },
    series: {
      marker: {
        enabled: false,
        states: {
          hover: {
            enabled: true
          }
        }
      }
    },
    shadow: false,
    states: {
      hover: {
        lineWidth: 1
      }
    }
  },
  series: [{
    data: d
  }]
};

This functions gets called once the page is loaded up and serve as the initial reading of the CSV file and putting into a chart.

$.get(getURLParameter("data"), function(data) {
  var lines = data.split('\n');
  $.each(lines, function(lineNo, line) {
    if(lineNo > 0) {
      var items = line.split(',');
      c.push(items[0]);
      d.push(parseInt(items[1]));
    }
    lastLine = lineNo;
  });
  chart = new Highcharts.Chart(options);
});

This tiny one-liner grabs the CSV filename from the URL

function getURLParameter(name) {
  return decodeURIComponent((new RegExp('[?|&]' + name + '=' + '([^&;]+?)(&|#|;|$)').exec(location.search)||[,""])[1].replace(/\+/g, '%20'))||null;
}

Finally, this reloads the page when window size is changed, and a container for where the graph resides

<div id="chart" style="min-width: 400px; height: 400px; margin: 0 auto"></div>
<body onresize="window.location=window.location;">

When all things comes together, this is what it looks like.

<!DOCTYPE HTML>
<html>
    <head>
        <meta http-equiv="Content-Type" content="text/html; charset=utf-8">
        <title>Chart</title>

        <script type="text/javascript" src="jquery-1.8.2.min.js"></script>
        <script type="text/javascript">

    /*************** Variables ****************/
        var refreshIntervalMS = 1000;    // how frequent do you want to update the chart.
        var csvSamplingRateMS = 1000;    // whats the sampling rate of the data written into the CSV
        var chartMins = 15;    // how many minutes of data do you want to display
        var maxPointsInGraph = chartMins*60*1000/csvSamplingRateMS; //last 15 mins data (collected every 1000ms)
    /************ End of Variables ************/
    /****** Do not modify past this line ******/
        var c = [];
        var d = [];
        var docTitle = 'Chart for ' + getURLParameter("data")
        var lastLine = 0;
        var chart;

        var winW = 630, winH = 460;
        if (document.body && document.body.offsetWidth) {
         winW = document.body.offsetWidth;
         winH = document.body.offsetHeight;
        }
        if (document.compatMode=='CSS1Compat' &&
            document.documentElement &&
            document.documentElement.offsetWidth ) {
         winW = document.documentElement.offsetWidth;
         winH = document.documentElement.offsetHeight;
        }
        if (window.innerWidth && window.innerHeight) {
         winW = window.innerWidth;
         winH = window.innerHeight;
        }
        winW = winW - 20;
        winH = winH - 20;

        document.title = docTitle;

        function requestNewData() {
            $.get(getURLParameter("data"), function(data) {
                var lines = data.split('\n');
                for(i = lastLine; i < lines.length; i++) {
                    if(i > 0) {
                        var items = lines[i].split(',');
                        chart.series[0].addPoint(parseInt(items[1]), false, chart.series[0].data.length > maxPointsInGraph);
                    }
                    lastLine = i;
                }
                chart.redraw();
                setTimeout(requestNewData, refreshIntervalMS);
            });
        }

        $(document).ready(function() {

            $.get(getURLParameter("data"), function(data) {
                var lines = data.split('\n');
                $.each(lines, function(lineNo, line) {
                    if(lineNo > 0) {
                        var items = line.split(',');
                        c.push(items[0]);
                        d.push(parseInt(items[1]));
                    }
                    lastLine = lineNo;
                });
                chart = new Highcharts.Chart(options);
            });

            var options = {
                    chart: {
                        renderTo: 'chart',
                        defaultSeriesType: 'spline',
                        zoomType: 'x',
                        spacingRight: 20,
                        width: winW,
                        height: winH,
                        events: {
                            load: requestNewData
                        }
                    },
                    title: {
                        text: docTitle
                    },
                    subtitle: {
                        text: document.ontouchstart === undefined ?
                            'Click and drag in the plot area to zoom in' :
                            'Drag your finger over the plot to zoom in'
                    },
                    xAxis: {
                        title: {
                            text: 'Time Measurement'
                        },
                        categories: c
                    },
                    yAxis: {
                        title: {
                            text: 'Sensor Readings'
                        }
                    },
                    tooltip: {
                        shared: true
                    },
                    legend: {
                        enabled: false
                    },
                    plotOptions: {
                        lineWidth: 1,
                        marker: {
                            enabled: false,
                            states: {
                                hover: {
                                    enabled: true,
                                    radius: 5
                                }
                            }
                        },
                        series: {
                            marker: {
                                enabled: false,
                                states: {
                                    hover: {
                                        enabled: true
                                    }
                                }
                            }
                        },
                        shadow: false,
                        states: {
                            hover: {
                                lineWidth: 1
                            }
                        }
                    },
                    series: [{
                        data: d
                    }]
            };

            //var chart = new Highcharts.Chart(options);

        });

        function getURLParameter(name) {
            return decodeURIComponent((new RegExp('[?|&]' + name + '=' + '([^&;]+?)(&|#|;|$)').exec(location.search)||[,""])[1].replace(/\+/g, '%20'))||null;
        }
    </script>

    </head>
    <body>
<script src="highcharts.js"></script>
<script src="exporting.js"></script>

<div id="chart" style="min-width: 400px; height: 400px; margin: 0 auto"></div>
<body onresize="window.location=window.location;">
    </body>
</html>

References: Highcharts API

Posted in Webdev | Tagged , , , , | 25 Comments

Ad-Hoc WiFi for Motorola Xoom MZ601 3G (Honeycomb 3.2)

First mod for my rooted Xoom is to enable Ad-Hoc WiFi support so that the Xoom can connect to the phone’s ad-hoc tether.

1. Prerequisites

2. Instructions

  1. Copy zip file to the root of your MicroSD card.
  2. Reboot into CWM recovery (Volume Down during boot up and Volume Up to select Recovery).
  3. Select install zip from sdcard
  4. Select choose zip from sdcard
  5. Select
    Ad-hoc_network_flash_(alternate).zip
  6. Confirm that yes you want to install it.
  7. Let it run it’s commands.
  8. Reboot system and enjoy Ad-hoc networks!

Ad-Hoc networks will have a

(adhoc)

at the end of the network name to indicate that it is an adhoc network.

It is as simple as that.

Posted in Uncategorized | Tagged , , , , , | 1 Comment