file-mirror

1. Introduction
2. Requirements
3. Installation
4. Modus operandi

Edit Link Top

Introduction

File-mirror is a chicken egg used to update a remote copy of a local directory. It has certain advantages and (certain distadvantages) which might make it very useful (entirely useless) for you.

Advantanges are:

• It keeps a hash of all local files at the time of the last update, which means it requires very little bandwidth (usually). Basically, it remembers the state of the directory when it last updated the remote copy.
• The only assumption it makes on the server is that it is possible to transfer files to it with any of the following methods:
  • scp
  • tar | ssh tar
  • kermit
• It is implemented in Scheme with a programming interface that makes it very easy to call from your Scheme programs.

Disadvantages might be:

• If a tiny portion of a big file changes, it needs to resend the entire file.
• It doesn't (currently) erase remote files.
• It can only update one remote mirror of the local information. In the future we might remove this limitation by allowing it to keep information about the state of multiple remote copies.

I use it file-mirror very frequently. I like to make as little assumptions on the server as possible, to make it easy to move to different hosting providers. Here are a few examples of situations where file-mirror might be useful:

• The Open List is a web application. I run the application in my machine, where it downloads a lot of files and does its magic (resulting in the generation of a lot of HTML and some PNG files). It uses scpmirror to update the remote (published) version every hour. Without scpmirror I would have to run the whole TOL application on the server, which would require a more expensive contract with my hosting provider (probably requiring a dedicated machine). Thanks to scpmirror, only files with changes (which tend to be few, specially with regards to their size) are uploaded.
• Colibrí's web site is generated locally in my machine. This involves quite a few programs: TeTeX (LaTeX and friends) to generate PDFs from the TEX files in the Subversion repository, my mailfolder2html program to update the archives of the mailing lists, Subversion (to checkout the last version of the files) and maybe some others. A script (actually a target in a Makefile) calls all the programs to generate the files which are then published, using scpmirror.
• You generate your weblog in your desktop machine but then upload it to some server. You don't want to upload all the files everytime, just those that have changed or are new.

Edit Link Top

Requirements

file-mirror needs Chicken.

Edit Link Top

Installation

Download the contents of:

• http://anonymous:@freaks-unidos.net/azul-home/src/chicken-eggs-original/file-mirror/

Run:

cd file-mirror
chicken-setup file-mirror
for module in sha1 md5 format-modular srfi-40 stream-ext; do
  chicken-setup $module;
done
make

Edit Link Top

Modus operandi

From Chicken's CSI:

(update-mirror-tar-ssh HASH SRC SERVER DST)

Here HASH is a hashing function (use mtime-hash-calc if you want to use files' timestamps to detect changes or md5-hash-calc if you want to use their MD5 hash to detect changes), SRC is the source directory in the local machine that you want to upload somewhere, SERVER is the server you want to upload things to (it could include a username, as in azul@freaks-unidos.net) and DST is the directory on the remote server where you are uploading things.

You can do this from the command line with:

ssh-tar-mirror SRC SERVER DST

This will assume you want to use the MD5 hashes (right now there is no way to specify the hash function you want to use from the command line).

file-mirror will use SRC/.file_mirror_hash to store/load the hashes (according to whatever hash function you specify) for all the files in SRC. If this is the first time it is run for SRC (which means it won't find the .file_mirror_hash file), it will simply upload all files. If, on the other hand, it manages to read it, it will compare the hashes of all the files with the values it calculated during the last update. All modified files get uploaded to the remote directory. Ah the end of the run, it will store the hashes of all files.

An obvious consequence is that if you want file-mirror to upload all files in a given directory (regardless of the state of the remote copy), it suffices to erase the SRC/.file_mirror_hash file.

file-mirror will never upload files ending with tilde (~) or beginning with a dot (.). You could change this behaviour by modifying its ignore-file? function.

You can set two environment variables:

• FILE_MIRROR_HASH=no: won't modify the SRC/.file_mirror_hash file.
• FILE_MIRROR_RUN=no: won't execute any commands (just print them).