Software engineering blog of Clément Bouillier: playground
Showing posts with label playground. Show all posts
Showing posts with label playground. Show all posts

Monday, January 3, 2011

Data backup solution based on RSync with a NAS

I have once experienced in the past some limited data loss due to a hard disk crash, and lastly, my first external hard drive starts to have some issues…I can just reiterate popular recommendations to think seriously to backup previous data as soon as you got more and more alerts, like repeated hard drive scans at startup (or when you plug it for external drives), suspicious behavior when reading data on drive…that’s what I have done lastly and I avoid lost of plenty of personal photos and videos…
From that moment forward, I decided to set a permanent backup solution. After having a look at web hosted solutions (not convinced completely convinced), I finally went for my own NAS, a DLink DNS-323, which is really easy to configure and extend (Linux embedded). It was also a chance to get hands dirty with Linux toys Sourire (long time…), but don’t be afraid to try! (except if you are just able to write documents and mails with a computer…else it could take you several long nights to get it running)

Rsync over SSH as the main toys

Rsync is an incremental files synchronization software for Unix systems. It is command-line based, but could be really powerful along with scripts. I let you search over the web for details on this tools, I will only show how I use it for backup. Note that there is several shared solutions around RSync. I was particularly inspired by wiki.dns323.info and BackupNetClone. I created my own scripts since the first one is too minimalist (based on BAT scripts…outch) and I found the second one too intrusive on clients computer (need SSH deamon and RSync server on each).

SSH will be used to secure RSync file synchronization.

To use it with Windows clients, the first thing is to install Cygwin (or other Linux emulator), really simple, you just have to click Next until package selection, then you select RSync and OpenSSH packages (just the main, dependencies will be grabbed automatically), and then you click Next until the end.

I will come back to client set up (don’t be afraid, it is just script that will have to be scheduled…) after a quick view on the server side, i.e. the NAS.

Set up NAS

My NAS, a DLink DNS-323, is Linux based. You have to use a fun_plug script that will be loaded at NAS startup. You can use ffp that includes some applications, particularly SSH and RSync daemons. Follow instructions in the following link to install it: wiki.dns323.info/howto:ffp.

Typically, you will set up a backup account on DNS-323 through admin interface (http://[NAS IP]), add a "backup" account in the Advanced tab. Next, you can change home and shell in /etc/passwd.

Set up clients

First, you have to configure the client once, then you would probably change configuration of which folders to backup.

First time set up

I explain here what you have to do once for each client computer (i.e. one to backup):

1. generate SSH keys that will be used next:

ssh-kengen -t dsa –b1024






You can let the default key path. Do not provide any passphrase if you want to automate your backups (it would ask it each time you want to backup).




2. copy SSH public key of client to the NAS with:




ssh-copy-id -i ~/.ssh/id_dsa.pub backup@[dns-323 IP]


I have packaged this in a script along with some simple configuration (IP, backup user name…).





What to backup?


My scripts (explained below) will search for configuration files, each giving one path to backup with its destination path on the NAS:



# Local path to backup, use /cygdrive/[drive letter]/... syntax
LOCAL_PATH_TO_BACKUP="/cygdrive/c/testbackup"

# [Optional] Target Rsync module -> override global settings
#TARGET_MODULE="backup"

# Target path in module
TARGET_PATH="test"



Launch a backup


I have a launchBackup.bat script that launches the backup.sh script through Cygwin. In this script, I load configuration from setup, I set up a SSH tunnel, then start rsync and finally close SSH tunnel.



RSync command is:



rsync -aivx --port ${SSH_TUNNEL_LOCAL_PORT} --chmod +rwx ${LOCAL_PATH_TO_BACKUP} 127.0.0.1::${TARGET_MODULE}/${TARGET_PATH}



Parameters name talk by themselves, –aivx are some common options of rsync. I don’t have yet set up incremental backup with --link-dest (hard linking option) and I am wondering about using –-delete that removes on server also what have been removed from your client folders (then you have to make sure that one server path is only used by one client to avoid massive deletions…).



Don’t forget to check your Firewall settings if you get some “Connection refused”-like errors.



Scheduling


You can simply rely on Windows Tasks scheduler. And you are done!



Assessment…



Not so pricy, I got the NAS for 100€ + 70€ for 1,5To hard disk drive. It is quite easy to set up, open as you are the only master of your backup, and then easily configurable/extendable and with unlimited possibilities.



About security, it removes hardware failure but do not protect from other more serious domestic risks like burglary, fire…but for that I got an idea, it is to build a small network of NAS like that (two to start…) with some parents for example, providing us a backup solution by the way Sourire



And a final word about environment impacts, I have bough an energy meter and it consumes only 10 Watts when idle (most of the time), quite good finally.

Saturday, February 13, 2010

My technological playground...

Well, it has been a long time since my last published post...in fact, I got several post waiting to be finalized...about application architecture, conception and communication.

Waiting for these post to be published, I would like to blog on the last evolution of my "technological playground".

My technological playground
For a little more than 5 years, I have done technological survey through web review, beta software test, community frameworks discovery...and I always consider this activity as a necessity to leverage my competencies.

For 2 years, I also maintain my own personal web site, using last Microsoft technology. It is a good way to dig some technologies (even if sometimes my wife complains of instability of our site...).

And now, for 1 year, I got a MSDN licence with Visual Studio Team Developer. With it, I can make tests environment. Now, I plan to build several virtual tests environments.

So today my technological playground is made of:
I am not on Twitter yet...but wondering, I see Google Buzz coming out or just Comment Sharing on Google Reader.

My virtual tests environments
I used to have a robust computer but a little undersized to be used as an efficient playground...but it was enough for the minimum. Now I reach a point where I need a little more powerful environment.

So I buy a new laptop, briefly Asus UL80vt boosted with an Intel Postville SSD drive, using hard drive in an external USB box for VMs as well as two hot USB keys (16Go Corsair Flash Voyager GT). Work well, great autonomy and powerfull...just a deception : I haven't an eSATA port, I have to rely only on USB...what a shame. But it is just what I need to support my plans.

So with this configuration, I can run several virtual machines given the 4Go RAM and 64bits host OS. For now, I am using VMWare Player as virtual hosting environment (mainly because Virtual PC does not support 64bits guests).

VMs allow me to separate tests. For now, for instance, I got the following VMs:
  • Windows Server 2008 with TFS2010 Beta2 (I need to upgrade to just released RC I know...)
  • Windows 7 with VS2010 Beta2
  • Windows 7 with VS2008
To build these VMs, I always keep a clean copy of each OS VM with base software I am used to. It is a little bit as if I do snapshot with VMWare Workstation.
Moreover, I track what I installed on each VMs (software version) in an Excel sheet.

My Google Reader & Delicious accounts
I use it to track RSS feeds on the web. There are mainly of three types:
I use my Share Items to share news. I prefer Delicious to bookmark and share my favorite web pages. On Delicious, I can better and simpler organize bookmarks through tags.

Participating in community
That the part I discovered last year with Alt.Net Paris community starting. It is the best place to learn and share with maximum interactivity. You learn a lot from others experiences, you meet some great people with rich experience.
Definitively a great experience!! Come to join us.

And finally this blog
I started this blog one year ago, and I experience difficulty to post regularly...I try my best and hope I will improve.