Tracking automated scripts
The "checkscripts" MySQL database currently resides on the libcontent server. This database is composed (currently) of two tables: "scripts" and "ran".
The scripts table contains the following information for each cron script that we depend upon for support of our infrastructure:
- id -- an identifying number, used by each script to log in when it runs
- scriptname (name of the script)
- server on which it resides
- directory in which it exists
- cron -- the crontab specification for when it runs
- runswhen -- a textual explanation of when it runs
- doeswhat -- a textual explanation of what the script does
- succeeds -- the name of scripts which must precede this one for it to do its job properly (dependencies)
- precedes -- the name of scripts which this script must precede in order for them to do their job properly (dependencies)
The "ran" table contains the entries made by each script after it completes its task, and before it exits. It contains:
- an auto-incrementing number for the database entry
- scriptid, the number of the script, which corresponds to the id number in the scripts table
- datestamp -- added automatically at the time of the entry
- errors -- a textual description of any errors encountered by the script during its run
As described in Watching Our Backs, Once a week, a script called "checkscripts" in /srv/scripts/cya looks through the entries in the checkscripts database for the past week, and compares them with the list of entries of existing cron scripts and when they are due to run. If any scripts did NOT run, which were scheduled, or any of them logged errors, this script sends emails to notify us of problems. Of course, this script also logs in with the checkscripts database to verify that it ran, and when.
Then, a third script ("checkscriptcheck" in /srv/scripts/cya) runs to verify that the checkscripts script ran as it should. Again, this one sends us any errors, and it logs in with the checkscripts database.
If there's problems with servers going down frequently, these three scripts can reside on three different servers. This helps track down what has/hasn't been taken care of, as well as notification of servers not functioning correctly.