Mass Content - Revision history

Jlderidder at 19:50, 18 October 2010

2010-10-18T19:50:42Z

←Older revision		Revision as of 19:50, 18 October 2010
Line 44:		Line 44:
	16) Check the output file from the moveMassContent script, and also check the directories on the share drive, to see if anything did NOT get moved successfully to the storage drive. If anything failed, rerun the script; if it still does not move the content, flag the directory with "_NotInStorage" added to its header, and report it to Jody.		16) Check the output file from the moveMassContent script, and also check the directories on the share drive, to see if anything did NOT get moved successfully to the storage drive. If anything failed, rerun the script; if it still does not move the content, flag the directory with "_NotInStorage" added to its header, and report it to Jody.

-	17) Check the website and make sure everything is hunky dory. :-)	+	17) Check the website and make sure everything is hunky dory. :-)

	18) exit out of secure shell. Good work!!		18) exit out of secure shell. Good work!!

Jlderidder: /* Here's the workflow for getting a Mass Content collection live */

2010-10-18T19:33:40Z

Here's the workflow for getting a Mass Content collection live

←Older revision		Revision as of 19:33, 18 October 2010
Line 25:		Line 25:
	7) check in the output directory for the latest Linking_README file (they are timestamped). Look for "ERROR" and note any warnings. If necessary, make repairs to files and rerun. If the EAD was trashed, the previous copy was placed in the backups directory with a timestamp on it. You can replace the trashed one with the previous version and rerun if you need to.		7) check in the output directory for the latest Linking_README file (they are timestamped). Look for "ERROR" and note any warnings. If necessary, make repairs to files and rerun. If the EAD was trashed, the previous copy was placed in the backups directory with a timestamp on it. You can replace the trashed one with the previous version and rerun if you need to.

-	8) run generateMODS in MassContent/scripts directory: type in: `generateMods` . This will again ask for the collection number; then pulls the new PURL links from the database, creates minimal mods for each item and places them in the MassContent/MODS directory -- it also copies them to the deposits directory for archiving with the tiffs. [[Image:GenerateMods.txt]]	+	8) run generateMODS in MassContent/scripts directory: type in: `generateMods` . This will again ask for the collection number; then pulls the new PURL links from the database, creates minimal mods for each item and places them in the MassContent/MODS directory -- it also copies them to the deposits directory for archiving with the tiffs. [[Image:GenerateMods.txt]] This script depends upon a template specifically developed for your collection; without it, this will fail. An example, one for Joshua Foster collection, can be viewed here: [[Example_MODS_template]]

	9) check in the output directory for the latest MODS_README file (they are timestamped). Look for "ERROR" and note any warnings. If necessary, make repairs to files and rerun. It will overwrite existing MODS only if they have NOT been distributed live.		9) check in the output directory for the latest MODS_README file (they are timestamped). Look for "ERROR" and note any warnings. If necessary, make repairs to files and rerun. It will overwrite existing MODS only if they have NOT been distributed live.

Jlderidder: /* Here's the workflow for getting a Mass Content collection live */

2010-10-18T19:29:32Z

Here's the workflow for getting a Mass Content collection live

←Older revision		Revision as of 19:29, 18 October 2010
Line 9:		Line 9:
	== Here's the workflow for getting a Mass Content collection live ==		== Here's the workflow for getting a Mass Content collection live ==

-	(Software here: [[Mass_Content_Software]]; Diagram of this workflow here: [[~~Mass_Content_workflow~~]])	+	(Software here: [[Mass_Content_Software]]; Diagram of this workflow here: [[Image:MassContent.png]])

	1) starting on the Share drive, make sure all tiffs are in a Scans folder in S:\Digital Projects\Digital_Coll_Complete\{collection_number}_whatever where {collection_number} is the number of your collection, and "whatever" is the text you've tacked on to it so you can find your collection by name		1) starting on the Share drive, make sure all tiffs are in a Scans folder in S:\Digital Projects\Digital_Coll_Complete\{collection_number}_whatever where {collection_number} is the number of your collection, and "whatever" is the text you've tacked on to it so you can find your collection by name

Jlderidder at 19:25, 18 October 2010

2010-10-18T19:25:10Z

←Older revision		Revision as of 19:25, 18 October 2010
Line 15:		Line 15:
	2) use the script S:\Digital Projects\Administrative\scripts\qc\BoxFolderCheck.pl to check filenames; run it till it shows no errors. [[Image:BoxFolderCheck.txt]]		2) use the script S:\Digital Projects\Administrative\scripts\qc\BoxFolderCheck.pl to check filenames; run it till it shows no errors. [[Image:BoxFolderCheck.txt]]

-	3) open a command-line window in SSH. Type in: `cd MassContent/scripts`. This will change your working directory to the one where the scripts are. Screenshot of directory structure: [[Image:~~Cabaniss_directories~~.~~PNG~~]]	+	3) open a command-line window in SSH. Type in: `cd MassContent/scripts`. This will change your working directory to the one where the scripts are. Screenshot of directory structure: [[Image:massdirs.jpg]]

	4) run makeMassJpegs in MassContent/scripts directory ([[Image:makeMassJpegs.txt]]): type in: `makeMassJpegs` This script will ask you for the collection number, and will also ask which directories you want JPEGs created for (do NOT use for transcripts, or they will overwrite the content images). The script will create JPEGs for all the tiffs in the directories you've chosen, and will place them all in the MassContent/jpegs directory on the server. After creating all the JPEGs it will correct for the multiples created due to thumbnails in the TIFFs.		4) run makeMassJpegs in MassContent/scripts directory ([[Image:makeMassJpegs.txt]]): type in: `makeMassJpegs` This script will ask you for the collection number, and will also ask which directories you want JPEGs created for (do NOT use for transcripts, or they will overwrite the content images). The script will create JPEGs for all the tiffs in the directories you've chosen, and will place them all in the MassContent/jpegs directory on the server. After creating all the JPEGs it will correct for the multiples created due to thumbnails in the TIFFs.

Jlderidder at 19:22, 18 October 2010

2010-10-18T19:22:19Z

Jlderidder at 18:52, 18 October 2010

2010-10-18T18:52:54Z

←Older revision		Revision as of 18:52, 18 October 2010
Line 1:		Line 1:
		+	''Note: By "Mass Content" we mean a collection that is being digitized without manually-created item-level metadata, intended for delivery online via links from the finding aid.''
		+
	== Preliminary steps on the server ==		== Preliminary steps on the server ==

Line 5:		Line 7:


-	== Here's the workflow for getting ~~Cabaniss content~~ live ==	+	== Here's the workflow for getting a Mass Content collection live ==

-	(Software here: [[~~Cabaniss_Software~~]]; Diagram of this workflow here: [[~~Cabaniss_Papers~~]])	+	(Software here: [[Mass_Content_Software]]; Diagram of this workflow here: [[Mass_Content_workflow]])

-	1) starting on the Share drive, make sure all tiffs are in a Scans folder in S:\Digital Projects\Digital_Coll_Complete\~~u0003_0000252_Cabaniss~~	+	1) starting on the Share drive, make sure all tiffs are in a Scans folder in S:\Digital Projects\Digital_Coll_Complete\{collection_number}_whatever where {collection_number} is the number of your collection, and "whatever" is the text you've tacked on to it so you can find your collection by name

-	2) use the script S:\Digital Projects\Administrative\scripts\~~Cabaniss~~\~~CabanissCheck~~.pl to check filenames; run it till it shows no errors. [[Image:~~CabanissCheck~~.txt]]	+	2) use the script S:\Digital Projects\Administrative\scripts\qc\BoxFolderCheck.pl to check filenames; run it till it shows no errors. [[Image:BoxFolderCheck.txt]]

-	3) open a command-line window in SSH. Type in: `cd ~~CABANISS~~/scripts`. This will change your working directory to the one where the scripts are. Screenshot of ~~Cabaniss~~ directory structure: [[Image:Cabaniss_directories.PNG]]	+	3) open a command-line window in SSH. Type in: `cd MassContent/scripts`. This will change your working directory to the one where the scripts are. Screenshot of directory structure: [[Image:Cabaniss_directories.PNG]]

-	4) run ~~makeCabanissJpegs~~ in ~~CABANISS~~/scripts directory ([[Image:~~makeCabanissJpegs~~.txt]]): type in: `~~makeCabanissJpegs~~` This script will ask you which directories you want JPEGs created for (do NOT use for transcripts, or they will overwrite the content images) ~~-- this~~ will create JPEGs for all the tiffs in the directories you've chosen, and will place them all in the ~~CABANISS~~/jpegs directory on the server. After creating all the JPEGs it will correct for the multiples created due to thumbnails in the TIFFs.	+	4) run makeMassJpegs in MassContent/scripts directory ([[Image:makeMassJpegs.txt]]): type in: `makeMassJpegs` This script will ask you for the collection number, and will also ask which directories you want JPEGs created for (do NOT use for transcripts, or they will overwrite the content images). The script will create JPEGs for all the tiffs in the directories you've chosen, and will place them all in the MassContent/jpegs directory on the server. After creating all the JPEGs it will correct for the multiples created due to thumbnails in the TIFFs.

	5) Spot check the output. There should be 2 jpegs for every tiff. One ends in _128, and one in _2048. These correspond to the number of maximum pixels to a side. The one ending in "_128.jpg" is a thumbnail. The one ending in "_2048.jpg" is a large image used in display.		5) Spot check the output. There should be 2 jpegs for every tiff. One ends in _128, and one in _2048. These correspond to the number of maximum pixels to a side. The one ending in "_128.jpg" is a thumbnail. The one ending in "_2048.jpg" is a large image used in display.

-	6) run linkContent in ~~CABANISS~~/scripts directory: type in: `linkContent` . This ~~updates~~ the EAD in the EAD directory ~~and copies~~ the older version to the ~~CABANISS~~/backups directory. [[Image:LinkContent.txt]]	+	6) run linkContent in MassContent/scripts directory: type in: `linkContent` . This script asks for the collection number, looks at all the "_2048" size jpegs in the jpeg directory that belong to this collection, and asks you a series of questions. Some finding aids have collection numbers preceding the box numbers; some have box numbers left-padded with zeros to a certain number of digits; and some have prefixes on the box numbers that were not used in the filename. For example, "252.012" is box 12 for collection MSS 252, but "415" is box 15 for collection MSS 535. The script needs this information in order to match up filenames with their box/folder location in the finding aid. Once the item is located in the EAD, this script will obtain a persistent identifier link for each item from the Mysql database, and update the EAD in the EAD directory, copying the older version to the MassContent/backups directory. [[Image:LinkContent.txt]]

	7) check in the output directory for the latest Linking_README file (they are timestamped). Look for "ERROR" and note any warnings. If necessary, make repairs to files and rerun. If the EAD was trashed, the previous copy was placed in the backups directory with a timestamp on it. You can replace the trashed one with the previous version and rerun if you need to.		7) check in the output directory for the latest Linking_README file (they are timestamped). Look for "ERROR" and note any warnings. If necessary, make repairs to files and rerun. If the EAD was trashed, the previous copy was placed in the backups directory with a timestamp on it. You can replace the trashed one with the previous version and rerun if you need to.

-	8) run ~~generateMODS~~ in ~~CABANISS~~/scripts directory: type in: `~~generateMods~~` . This creates minimal mods for each item and places them in the ~~CABANISS~~/MODS directory -- it also copies them to the deposits directory for archiving with the tiffs. [[Image:~~GenerateMods~~.txt]]	+	8) run generateMassMODS in MassContent/scripts directory: type in: `generateMassMods` . This will again ask for the collection number; then pulls the new PURL links from the database, creates minimal mods for each item and places them in the MassContent/MODS directory -- it also copies them to the deposits directory for archiving with the tiffs. [[Image:GenerateMassMods.txt]]

	9) check in the output directory for the latest MODS_README file (they are timestamped). Look for "ERROR" and note any warnings. If necessary, make repairs to files and rerun. It will overwrite existing MODS only if they have NOT been distributed live.		9) check in the output directory for the latest MODS_README file (they are timestamped). Look for "ERROR" and note any warnings. If necessary, make repairs to files and rerun. It will overwrite existing MODS only if they have NOT been distributed live.

Jlderidder: New page: == Preliminary steps on the server == # make sure that the Windows share is mounted: type into the ssh window on libcontent1: `ls /cifs-mount` -- if no listing appears, or the window h...

2010-10-18T15:13:01Z

New page: == Preliminary steps on the server == # make sure that the Windows share is mounted: type into the ssh window on libcontent1: `ls /cifs-mount` -- if no listing appears, or the window h...

New page

== Preliminary steps on the server ==

# make sure that the Windows share is mounted: type into the ssh window on libcontent1: `ls /cifs-mount` -- if no listing appears, or the window hangs up, you need to mount the share drive. Otherwise, proceed to getting Cabaniss content live...
# To mount the Windows drive, type this in on the commandline on libcontent1: `sudo mount -t cifs -o username=jjcolonnaromano,domain=lib //libfs1.lib.ua-net.ua.edu/share/Digital\ Projects/ /cifs-mount` and use the password for share for Jeremiah. If successful, the command in the last step will show you the directories within the Digital Projects folder on Share.

== Here's the workflow for getting Cabaniss content live ==

(Software here: [[Cabaniss_Software]]; Diagram of this workflow here: [[Cabaniss_Papers]])

1) starting on the Share drive, make sure all tiffs are in a Scans folder in S:\Digital Projects\Digital_Coll_Complete\u0003_0000252_Cabaniss

2) use the script S:\Digital Projects\Administrative\scripts\Cabaniss\CabanissCheck.pl to check filenames; run it till it shows no errors. [[Image:CabanissCheck.txt]]

3) open a command-line window in SSH. Type in: `cd CABANISS/scripts`. This will change your working directory to the one where the scripts are. Screenshot of Cabaniss directory structure: [[Image:Cabaniss_directories.PNG]]

4) run makeCabanissJpegs in CABANISS/scripts directory ([[Image:makeCabanissJpegs.txt]]): type in: `makeCabanissJpegs` This script will ask you which directories you want JPEGs created for (do NOT use for transcripts, or they will overwrite the content images) -- this will create JPEGs for all the tiffs in the directories you've chosen, and will place them all in the CABANISS/jpegs directory on the server. After creating all the JPEGs it will correct for the multiples created due to thumbnails in the TIFFs.

5) Spot check the output. There should be 2 jpegs for every tiff. One ends in _128, and one in _2048. These correspond to the number of maximum pixels to a side. The one ending in "_128.jpg" is a thumbnail. The one ending in "_2048.jpg" is a large image used in display.

6) run linkContent in CABANISS/scripts directory: type in: `linkContent` . This updates the EAD in the EAD directory and copies the older version to the CABANISS/backups directory. [[Image:LinkContent.txt]]

7) check in the output directory for the latest Linking_README file (they are timestamped). Look for "ERROR" and note any warnings. If necessary, make repairs to files and rerun. If the EAD was trashed, the previous copy was placed in the backups directory with a timestamp on it. You can replace the trashed one with the previous version and rerun if you need to.

8) run generateMODS in CABANISS/scripts directory: type in: `generateMods` . This creates minimal mods for each item and places them in the CABANISS/MODS directory -- it also copies them to the deposits directory for archiving with the tiffs. [[Image:GenerateMods.txt]]

9) check in the output directory for the latest MODS_README file (they are timestamped). Look for "ERROR" and note any warnings. If necessary, make repairs to files and rerun. It will overwrite existing MODS only if they have NOT been distributed live.

10) run makeItLive script in CABANISS/scripts directory ([[Image:makeItLive.txt]]): type in: `makeItLive` . This distributes the jpegs and
MODS in Acumen. It will delete the local copies of MODS and jpegs, so you can reuse the same directories over and over.Watch for errors on the command line.

11) move up to the home area, down into the scripts directory there, and run `findMissing` ([[Image:findMissing.txt]]); This script will hunt through Acumen to make sure there is a MODS file for every item, and at least one derivative for each MODS file. Any errors will be found in the output file written to the scripts/output directory. If errors are found, regenerate those MODS and/or JPEGs/MP3s and rerun relocate_all. Then run this script again to ensure all errors have been remedied.

12) run moveCabanissContent script in CABANISS/scripts directory ([[Image:moveCabanissContent.txt]]): type in: `moveCabanissContent` . This will check for a collection icon, for collection information (you can upload a new collection xml file if you need to make changes) and it will copy the content to the deposits directory on the storage drive. After checking the copy, it will delete the version on the share drive, including empty scans directories. Thus, anything left behind did NOT make it to the storage drive for some reason; don't delete it. This script also creates a timestamped output file in the CABANISS/outputs directory, and it will put errors there, as well as indicate when it has completed.

13) Wait for indexing to occur. This may be about 24 hours. If you want, you can watch the content being deposited on the storage server in the deposits/Cabaniss directory on libcontent1. :-)

14) Check online to locate one of the files you just uploaded. For example, if one of them was u0003_0000252_2010003, then look for http://acumen.lib.ua.edu/u0003_0000252_2010003 -- if it's there, indexing completed.

15) Once indexing has completed, run eadLive ([[Image:eadLive.txt]])to copy the EAD to the web directory. It's important to wait for indexing to be complete so the links in the EAD are valid.

16) Check the output file from the moveCabanissContent script, and also check the directories on the share drive, to see if anything did NOT get moved successfully to the storage drive. If anything failed, rerun the script; if it still does not move the content, flag the directory with "_NotInStorage" added to its header, and report it to Jody.

17) Check the website and make sure everything is hunky dory. :-)

18) exit out of secure shell. Good work!!

←Older revision		Revision as of 19:22, 18 October 2010
Line 25:		Line 25:
	7) check in the output directory for the latest Linking_README file (they are timestamped). Look for "ERROR" and note any warnings. If necessary, make repairs to files and rerun. If the EAD was trashed, the previous copy was placed in the backups directory with a timestamp on it. You can replace the trashed one with the previous version and rerun if you need to.		7) check in the output directory for the latest Linking_README file (they are timestamped). Look for "ERROR" and note any warnings. If necessary, make repairs to files and rerun. If the EAD was trashed, the previous copy was placed in the backups directory with a timestamp on it. You can replace the trashed one with the previous version and rerun if you need to.

-	8) run ~~generateMassMODS~~ in MassContent/scripts directory: type in: `~~generateMassMods~~` . This will again ask for the collection number; then pulls the new PURL links from the database, creates minimal mods for each item and places them in the MassContent/MODS directory -- it also copies them to the deposits directory for archiving with the tiffs. [[Image:~~GenerateMassMods~~.txt]]	+	8) run generateMODS in MassContent/scripts directory: type in: `generateMods` . This will again ask for the collection number; then pulls the new PURL links from the database, creates minimal mods for each item and places them in the MassContent/MODS directory -- it also copies them to the deposits directory for archiving with the tiffs. [[Image:GenerateMods.txt]]

	9) check in the output directory for the latest MODS_README file (they are timestamped). Look for "ERROR" and note any warnings. If necessary, make repairs to files and rerun. It will overwrite existing MODS only if they have NOT been distributed live.		9) check in the output directory for the latest MODS_README file (they are timestamped). Look for "ERROR" and note any warnings. If necessary, make repairs to files and rerun. It will overwrite existing MODS only if they have NOT been distributed live.

-	10) run ~~makeItLive~~ script in ~~CABANISS~~/scripts directory ([[Image:~~makeItLive~~.txt]]): type in: `~~makeItLive~~` . This distributes the jpegs and	+	10) run makeMassLive script in MassContent/scripts directory ([[Image:makeMassLive.txt]]): type in: `makeMassLive` . This distributes the jpegs and
-	MODS in Acumen. It will delete the local copies of MODS and jpegs, so you can reuse the same directories over and over.Watch for errors on the command line.	+	MODS in Acumen. It will delete the local copies of MODS and jpegs, so you can reuse the same directories over and over. Watch for errors on the command line.

-	11) move up to the home area, down into the scripts directory there, and run `findMissing` ([[Image:findMissing.txt]]); This script will hunt through Acumen to make sure there is a MODS file for every item, and at least one derivative for each MODS file. Any errors will be found in the output file written to the scripts/output directory. If errors are found, regenerate those MODS and/or JPEGs/MP3s and rerun ~~relocate_all~~. Then run this script again to ensure all errors have been remedied.	+	11) move up to the home area and down into the scripts directory there (cd ../../scripts), and run `findMissing` ([[Image:findMissing.txt]]); This script will hunt through Acumen to make sure there is a MODS file for every item, and at least one derivative for each MODS file. Any errors will be found in the output file written to the scripts/output directory. If errors are found, regenerate those MODS and/or JPEGs/MP3s and rerun makeMassLive. Then run this script again to ensure all errors have been remedied.

-	12) run ~~moveCabanissContent~~ script in ~~CABANISS~~/scripts directory ([[Image:~~moveCabanissContent~~.txt]]): type in: `~~moveCabanissContent~~` . This will check for a collection icon, for collection information (you can upload a new collection xml file if you need to make changes) and it will copy the content to the deposits directory on the storage drive. After checking the copy, it will delete the version on the share drive, including empty scans directories. Thus, anything left behind did NOT make it to the storage drive for some reason; don't delete it. This script also creates a timestamped output file in the ~~CABANISS~~/outputs directory, and it will put errors there, as well as indicate when it has completed.	+	12) run moveMassContent script in MassContent/scripts directory ( cd ../MassContent/scripts) ([[Image:moveMassContent.txt]]): type in: `moveMassContent` . This will check for a collection icon, for collection information (you can upload a new collection xml file if you need to make changes) and it will copy the content to the deposits directory on the storage drive. After checking the copy, it will delete the version on the share drive, including empty scans directories. Thus, anything left behind did NOT make it to the storage drive for some reason; don't delete it. This script also creates a timestamped output file in the MassContent/outputs directory, and it will put errors there, as well as indicate when it has completed.

-	13) Wait for indexing to occur. This may be about 24 hours. If you want, you can watch the content being deposited on the storage server in the deposits/~~Cabaniss~~ directory on libcontent1. :-)	+	13) Wait for indexing to occur. This may be about 24 hours. If you want, you can watch the content being deposited on the storage server in the deposits/MassContent directory on libcontent1. :-)

	14) Check online to locate one of the files you just uploaded. For example, if one of them was u0003_0000252_2010003, then look for http://acumen.lib.ua.edu/u0003_0000252_2010003 -- if it's there, indexing completed.		14) Check online to locate one of the files you just uploaded. For example, if one of them was u0003_0000252_2010003, then look for http://acumen.lib.ua.edu/u0003_0000252_2010003 -- if it's there, indexing completed.
Line 42:		Line 42:
	15) Once indexing has completed, run eadLive ([[Image:eadLive.txt]])to copy the EAD to the web directory. It's important to wait for indexing to be complete so the links in the EAD are valid.		15) Once indexing has completed, run eadLive ([[Image:eadLive.txt]])to copy the EAD to the web directory. It's important to wait for indexing to be complete so the links in the EAD are valid.

-	16) Check the output file from the ~~moveCabanissContent~~ script, and also check the directories on the share drive, to see if anything did NOT get moved successfully to the storage drive. If anything failed, rerun the script; if it still does not move the content, flag the directory with "_NotInStorage" added to its header, and report it to Jody.	+	16) Check the output file from the moveMassContent script, and also check the directories on the share drive, to see if anything did NOT get moved successfully to the storage drive. If anything failed, rerun the script; if it still does not move the content, flag the directory with "_NotInStorage" added to its header, and report it to Jody.

	17) Check the website and make sure everything is hunky dory. :-)		17) Check the website and make sure everything is hunky dory. :-)

	18) exit out of secure shell. Good work!!		18) exit out of secure shell. Good work!!