Watching Our Backs
Currently, we are storing archival content (described here: formats on a Linux server, in the directory structure described here: Organization_of_completed_content_for_long-term_storage. Content which we want included in LOCKSS (ADPNET) is linked into Manifests (as described here: File_Naming_and_Linking_for_LOCKSS.
Once this archival content is placed into storage, anything linked into the Manifest.html pages should NOT be changed. However, since our new delivery platform derives from this stored content, we need to be able to update the metadata as needed. Hence, the primary metadata file is copied as a versioned file, and the versioned file is what is linked into the Manifests for LOCKSS pickup. The metadata file which is NOT versioned is the most recent, and over-writeable, copy.
For the content which is not allowed to change, we have scripts running weekly to verify that the md5 checksum has not changed, prior to the full tape backup. If there's an error, we are notified in time to retrieve a good copy from a previous backup, before the corrupted item can be written to tape.
On libcontent1, in /srv/scripts/md5/cya/, the script which calculates and checks sums is called "dirs". First it looks for new content; if there is any, it calculates the sums for that content. Second, it checks the checksums for existing directories. If they have changed, it outputs an error. Third, it checks the checksums for existing files. If they have changed, it outputs an error. If there are no errors, it copies the sums /srv/scripts/md5/cya/backups in a tar.gz compressed form. Backups older than a year are deleted. Errors are sent to email@example.com and firstname.lastname@example.org. The script then logs in with the checkscripts database on content.lib.ua.edu to verify that it ran, the timestamp, and any errors.
After this script completes, another one in the same directory, on the same server runs. This one is called "storesums" and that's just what it does. Existing sums are checked against those stored in a MySQL database (md5sums) on another server (content.lib.ua.edu); if any have changed, or new sums were created, these are added to that database; no existing sums are overwritten (and of course, a datestamp is kept for each checksum). This script also logs in with the checkscripts database on content.lib.ua.edu to verify that it ran, the timestamp, and any errors.
Automated Script Verifications
Once a week, a script called "checkscripts" on content.lib.ua.edu in /home/jlderidder/scripts/cya looks through the entries in the checkscripts database for the past week, and compares them with the list of entries of existing cron scripts and when they are due to run.
The checkscripts database consists of 2 tables: "ran" and "scripts." Ran contains an auto-incrementing primary key (num), scriptid (identifier for the script, again a number), datestamp (for when this script ran) and errors. Scripts contains a numerical id (which corresponds to the scriptid in "ran"), scriptname, server, directory where the script resides, the cron job that calls it, and a textual description of when it runs ("runswhen") and what it does ("doeswhat") as well as names of any scripts that it precedes (preceeds) or succeeds (succeeds) in order to work properly.
If any scripts did NOT run, which were scheduled, or any of them logged errors, this script sends emails to notify us of problems. Of course, this script also logs in with the checkscripts database to verify that it ran, and when.
Then, since any server can go down, and multitudes of error possibilities exist, a third script on a third server ("checkscriptcheck" in /home/jlderidder/cya on libapp.ua-net.ua.edu) runs to verify that the checkscripts script ran as it should. Again, this one sends us any errors, and it logs in with the checkscripts database.
Since the checksums are backed up in a MySQL database, and a second MySQL database manages the scripts, there's a cron script (called "backups") on both content.lib.ua.edu (in /home/jlderidder/scripts/cya; backups go into /contentdbs/backups/databases/) and libcontent1.lib.ua.edu (in /srv/scripts/cya/; backups go into /srv/backups/ ) which backs up selected databases monthly. Both scripts delete backups over a year old. Currently the list of databases backed up on content.lib.ua.edu is this:
And the list of databases currently backed up on libcontent1.lib.ua.edu is this:
- InfoTrack ( see Tracking_for_the_long_term)
- metaview (the test database for Acumen)
These lists are subject to change.
And since CONTENTdm seems to be so prone to corruption, we also have a script called "indexes" in /home/jlderidder/scripts/metadata/backups/ on content.lib.ua.edu which backs up both the indexes and the binary index monitor files, for all content currently in the CONTENTdm database. These are .tar.gz compressed and stored in /contentdbs/backups/indexes, and this script also checks in with the checkscripts database, and deletes backups over a year old.