The online home of Toby Wilkinson

Improving on backing up with rsync

Recently my backups have been in rather a state of flux. Following the restoration of a site that I’d had cron religiously back up daily with rsync, I realised that I was accumulating rather a lot of cruft in my backups – old versions of CMS systems were present, temporary versions of images uploaded that were subsequently resized and pages put up briefly for special offers, maintainance or whatever all required pruning when I restored the site. I moved over to taking backups using rsync –delete which removed all files in the backup which are not in the live site. This was a good improvement, giving me a true snapshot of how the site was when the backup was taken. However it wasn’t long until there was an awkward conversation along the lines of “I deleted some images from our blog a few days ago, can you get them back?”

Why don’t you just check out from your own source control?

I was asked this a few times, and while it’s true that I’d love full control of what was on the sites in question, people these days expect more than static sites – they want to be able to update the content of their sites, run a blog and have external agencies work on the content of their sites. At the same time, they expect them to be backed up.

I wanted something that would provide backups with historical changes, so that I could restore sites to the way they were at a particular point in the past.

Research proved suprisingly fruitless. Many people documented how they backup with some of the more esoteric options of rsync and/or tar, giving them incremental backups, then using scripts to rebuild historical states of the files. Other people were using version control systems, such as SVN, written for maintaining source code with lengthy scripts to undo much of the work done by these systems (which generally need things like explicit instructions on the files that have come and gone since the last time the files were committed).

By chance, I finally found a comparison of backup tools on an Arch Linux page. This had several very hopeful looking bits of software. I settled on rdiff-backup as it seems mature, well documented, available in the standard Debian repositories, and best of all its familiar syntax meant that it was almost a drop-in replacement for rsync, but with the added backup of historical versions being saved. There are several examples on the project site which should get anyone interested up and running pretty quickly. However in my backup scripts it was a simple excersise to translate

rsync -avz --del user@webserver.tld:/srv/websites/project1/* /srv/site_backup/project1


rdiff-backup user@webserver.tld::/src/websites/project1/* /srv/site_backup/project1

(Note the double colon in the latter)

I can now copy from the backup directory in the normal manner, but I also have the option to delve back in history using something like this:

rdiff-backup -r 10D /srv/site_backup/project1/index.php ~/site_restore/

to recover the index file to the way it was ten days ago.

All well and good, this was a good step forward, however I noticed that there is a Windows version available. What a coup, being able to backup my servers and my desktop with the same tool! I downloaded and unpacked the Windows version, added the directory in which I’d put it into the path. (To modify the path on a Windows 7 computer, go to Start -> Computer -> System Properties -> Advanced System Settings -> Environment Variables -> Select “Path” and then click the “Edit…” button.) I fired up a command line and entered a rdiff-backup command and was met with an error saying :

Couldn't start up the remote connection by executing

ssh -C user@server.local rdiff-backup --server

Remember that, under the default settings, rdiff-backup must be
installed in the PATH on the remote system. See the man page for more
information on this. This message may also be displayed if the remote
version of rdiff-backup is quite different from the local version (1.2.8).

Oh dear, all did not seem well. I knew I had ssh facilities on the computer, as I use PuTTY many times a day, however from the command line there was indeed no way to simply type ssh to open a remote conection. Some searching brought me to this blog which fairly quickly got me up and running. I added the PuTTY directory to my Path, and created a keypair so that I was able to log straight into the server I was using for backup. It was then a fairly simple matter to supply the correct arguments to rdiff-backup to ensure it used plink, the PuTTY command line utility. What I have running now is as follows:

rdiff-backup --remote-schema "plink.exe -i C:\path\to\key.ppk %s rdiff-backup --server" "C:/Users/Me/Documents" user@server.local::/srv/desktop-backup/my-desktop

Be warned, subjectively this seems noticably slower than rsync the first time it is run, however subsequent backups seem to be reasonably speedy.

I hope this helps someone else out there to improve the depth of information held in their backups. Just remember, always check your logs to make sure your backups actually run ;-).

Solving a VirtualBox server rebooting with Debian Wheezy

Debian Wheezy recently made it to being released as the current Stable version. With a new version of Debian, I thought it time to update my Intel Atom based server. This runs two Virtual servers, allowing me the advantages of a good network setup without the associated power consumption, noise and space requirements of using physically separate hardware. After updating the host system, I turned my attention to one of the guest systems. After updating, it didn’t come up cleanly after a reboot. Checking the system with a Remote Desktop Connection showed a kernel error that the new -686 kernel no longer ran on non-PAE systems. Not a problem, quickly checking the options for VirtualBox meant that I could configure the guest system to include PAE extentions.

However whenever the guest system was stressed, the host would reboot. Using rsync over SSH seemed to be a particularly reliable way of causing it to reboot, though that could just have been due to the load on the guest system.

After quite a lot of head scratching, I finally noticed the following in my syslog.

 bebop kernel: [    0.000000] Atom PSE erratum detected, BIOS microcode update recommended

Sadly there seem to be no BIOS updates from Jetway to update this on my motherboard. Some research led me to the intel-microcode package, which updates the CPU’s microcode on the fly. A simple case of apt-get installing it and then a reboot showed that the microcode was being updated early in the boot process. Better still, my system no longer rebooted when the guest was stressed.

Another problem solved :).

T962 / T962A Reflow Oven Controller

Recently I’ve been working on a collaborative project with my friend Ed. His electronic work has been held back by a range of disappointing soldering results from his reflow oven. Having been supplied with a T962 oven for a project, he immediately found it frustrating to work with – Boards were inconsistently soldered and a lot of manual work was needed to create reliable circuits. The results were also badly affected by the ambient conditions. While an irritation, it was not the end of the world for building prototypes and one-off circuits.

Read more »

HOWTO – High current USB charging ports.

Having been playing with a Raspberry Pi recently I’ve written some logging software and left it powered to collect a decent amount of data. However it’s become quite a nuisance having a slightly-too-short network cable strung across to my desk, plus, let’s face it, my desk has enough clutter on it! As I have a small server on 24/7, it seemed an obvious source of power, however I was reluctant to simply plug it into the (Mini-ITX) motherboard’s USB ports as I was concerned that they may well not be man enough for the required 700mA to power the Pi.

After a little bit of groveling in the spares boxes lurking in the cupboard, I came up with a cunning plan…

Raw materials gatheredThe raw materials consisted of an old motherboard’s extra USB ports mounted on a PCI blanking plate, a cheap and nasty Molex -> Fan connector adapter and a few header pins left over from a recent project.






Connections preparedI decided to use a dedicated molex plug as there are a couple spare inside my small server, so I removed the extra cables from the fan adapter and simply had connections for 5V and ground. I didn’t have the correct pin removal tool, but the pins came out quite easily with a little poking from a narrow knife. I then stripped and tinned the cables appropriately.

I made sure that the cables would connect with a pair of pins by bending the ends into a J-shape and applying plenty of solder to make a good contact with both pins.



All connected.Here you can see I’ve inserted the header pins into the appropriate connections on the motherboard USB connector for ground and 5V. I’ve melted the plastic together with the tip of the iron to make sure that they don’t come apart inside the case. You could use a small blob of superglue if you prefer to avoid the hot plastic smell! A bit of insulating tape to make sure that it doesn’t touch anything it shouldn’t and we’re ready to go.




In place and connected.

Here you can see the finished adapter in place in my server connected to a spare molex drive connector. The only remaining job was to squeeze the spare cables out of the way of the fans and get the lid back on the case!






In action, RasPi powered straight from the PSU.

And here’s the final result – my Raspberry Pi running happily from the new adapter. Also useful for charging phones and other devices that use USB chargers. The strange nest of cables on top of the Pi is a Sparkfun temperature sensor breakout board connected to the Pi’s GPIO pins via a recycled floppy drive cable.

So there we have it, a few spare parts lying around and a few minutes soldering meant that I could happily power my Raspberry Pi without risking damage to my motherboard’s USB ports. I’ve one less cable to trip over and a little bit less clutter on my desk.

Have we reached a plateau in computers?

Yesterday I went to London, a regrettably rare occurrence these days but all the more special as I took the High Speed train for the first time. I was enjoying wafting through the countryside at hitherto unknown speeds. Out of curiosity I took out my phone and checked the speed on my GPS. We seemed to be travelling around 130-135mph most of the way. While I realize that there are trains elsewhere in the world that can travel considerably faster, the primary selling point of this service is its speed. It struck me that it’s not actually that much faster than we were capable of 70 years ago. OK, so this is an nice new express service. It’s clean, quiet, reasonably punctual and convenient – but in terms of ground breaking speed, it’s not that big a leap forward compared to how we traveled three generations ago. Not like cars… Well maybe – OK modern cars are much more safe, more comfortable, more reliable even, than they were in the past, but are they actually much faster? Given a deep enough bank balance I can walk into a showroom and drive out in a car capable of 200mph. Mind you, I could have bought a car in 1955 that would do 192mph (with a bit of preparation). Despite what vehicle makers would have you believe, fuel consumption hasn’t been revolutionized. A typical family car from the 1960s – The Ford Anglia 105E does about the same as a typical modern family car – a Ford Focus 2.0 Ghia – per gallon of fuel. At the end of the day, while cars have got “better” in a number of ways, some of the headline numbers haven’t really advanced that much.

And that got me thinking about parallels in the computer world…

Read more »

Turbo Training – A necessary evil, or a great time saver.

OK, so I have a Turbo-Trainer. (A device to attach your bike to holding it up and providing variable resistance, so that you can get on and pedal for a workout if you’ve not come across the term before.) At times I think it’s the most boring way to spend time. Others, I’m quite happy to get on the bike for a spin and feel better without the faff of going for a “proper” ride.

For some people it’s through necessary – injury, lack of time, or simply not having the opportunity to leave the house because you’re on-call or minding children – that the turbo trainer is the only way to develop / keep cycling fitness. For others it’s just a way to get some pedal strokes in – sitting pedaling while you watch TV or revise so as not to “waste” time. Then there’s those for whom the dedicated workout is a way to get fitter midweek without having to risk icy roads and idiots in cars or having to spend as long cleaning your kit and bike after a ride as you spent on the ride makes it an attractive option.

Read more »

Replacing the fan in a Zalman cooler

I recently upgraded the CPU cooler in my main desktop from the standard one included by Intel to a Zalman CNPS9700 LED. While I was impressed with the build quality and cooling capacity, I was disapointed that the included fan controller was manual, not only adding cables and controls inside my computer, but leaving it up to me to remember to turn the fan up when carrying out some intensive computing task. This struck me as silly when the motherboard comes with a PWM controller to adjust the speed of fans. After reading the documentation, I concluded that the 110mm fan it was supplied with was probably the dimensions of a standard 120mm case fan with the square frame. Unfortunately this was incorrect, having ordered a variable speed 120mm fan, I found it to be too big to fit in the cooler.

CPU cooler and replacement fan seen side by side.

CPU cooler and replacement fan seen side by side.

Read more »

HOWTO: Using Intel’s RST with Linux.

I recently updated my main system to a new Intel Sandybridge based setup which included a Asus P8Z68-V motherboard, as you may have seen. One of the features of the Intel Z68 chipset upon which this motherboard is based is Rapid Storage Technology (RST) which aims to use a Solid State Drive to cache the files your system uses frequently to improve system performance. While many people choose to install their operating system on a SSD and use a cheaper mechanical drive for applications and data, RST seems a better system as the most frequently used files, whatever they are, are available for fast access. Working on some code? Visual Studio and my project files are cached, shift to some 3D work? Cool, when it’s used, Blender and its files are put in the cache, all the while parts of windows that aren’t used are still available but not taking up expensive, fast SSD space (has anyone actually looked at the help file for MS Paint?). The system’s not perfect – things will only be cached the second (or more, depending on how the driver assess the “frequently used” nature of files) time they are accessed. Also as this is (at least partially) done in the software of the driver, there’s obviously an overhead on the running of the system, however overall it struck me as a good idea and pushed me into purchasing a SSD (a 128GB Crucial M4) to see if they really did transform the performance of my system as much as I’d been led to believe.

Read more »

Build Blog – Two high-end cases go head to head.

I recently had a bit of a shuffle round of computer parts, which meant that among other things, I put a brand new Sandy-Bridge processor in my computer used for video editing and PCB layout work, and shuffled my older, Core2 based, system into my general office computer. Of interest, though was the fact that I had recently aquired a Cooler Master CM 960 case, which being built for keen overclockers and gamers, had a quite different approach to making a high-end case than the Antec 3700BQE which I use at home, which is built for quiet operation. I thought it would be interesting to jot down some notes on how the process went so people can see a few more features of these cases.

Read more »

Are big changes coming?

I’ve recently been reading more and more analysts clamoring to find new hyperbole about the way new and coming technology will change the face of the Internet and computing. Web enabled TVs, smartphones and tablet computers are all being heralded as being on the cusp of revolutionizing the way we interact with computers and information. I’ve heard claims that SEO will be dead in two years and that the traditional ecommerce site is a thing of the past. If you listen to certain people the face of computing is going to be unrecognizable in a few short years….

Except it isn’t.

I’m sorry to be pessimistic, but this really isn’t an industry that likes change. Sure new ideas and refinements are brought on board, however big changers are viewed with a sense of irrelevant amusement. We have the technology to have immersive interfaces, to whoosh through virtual metaphores in 3D, controlled by how we move, where we look or even what we’re thinking, but we don’t. Instead we control a pointer with a movement sensor on a wire, first developed by Xerox 40 years ago, and we click on a button in the bottom left of the screen to bring up a menu. Heck, we’re still hitting keys laid out in a pattern designed to deliberately slow typists down so that Victorian typewriters wouldn’t jam! We’re not dealing with a market that likes anything too unfamiliar, just look at how Microsoft’s ribbon interface in Office was received – people are still complaining about how it’s different it is, how they don’t like it, not because of how it does work, but because it doesn’t work like what came before. IT departments are refusing to upgrade to a now two year old version of Office not for any technological reason, but because the costs of training users on something new is so high.

Faced with a new piece of technology, people’s reaction is almost universally to use it like an old piece of technology. Hand someone an iPad for the first time and all they will do is fire up a browser and go to Google. Sure there are refinements – people can use speech to search because it’s so awkward to type on a phone screen that’s only twice the width of your thumb and people tend to fill their iDevice up with apps that are thinly veiled RSS readers because mobile browsers and mobile sites tend to be tiresome to check regularly – but on the whole what they do and how they do it changes very little.

Technology will always have it’s early adopters who drive progress, and while those people may increasingly be buying the technology as a lifestyle accessory, they’re still pretty rare and the changes take time to trickle down to everyone else. The facial recognition software that tags their friends in photos of a night out is still working on the same concepts that awkward teenagers with head up displays from MIT were using to help recognise people and remind them of names, birthdays and what they last spoke about 15 years ago. The fact that you can use a browser on your Internet connected TV might be a lot more useable now that the typical home has a connection you can watch a film in acceptable quality without waiting a week for it to download, but I don’t think it’s going to revolutionise peoples habits and usage patterns any more than the ill fated WebTV browser did the best part of a decade ago.

I’m convinced that people’s interaction with technology is evolving, I think it’s a very exciting time watching many things that were once exclusive to enthusiasts and the very busy / rich trickle down to become common (if you’d shown me an Android phone back in the late ninties when I was lusting after the first of Nokia’s communicators, I’d not have belived such wonders were possible in my lifetime. Tell me however that people still used a qwerty keyboard on it to send the same emails and text messages we were then with such a magical item and I’d probably have cried.) We’re tending to buy a greater quantity of more specilized items and use them in new ways, but what we’re in the middle of I think is just that – evolution, not a revolution. Don’t throw away your search engine and web browser, I think you’ll be using them for a while yet…