Most Content - Revision history

Kgmatheny at 16:38, 11 February 2013

Kgmatheny — Mon, 11 Feb 2013 16:38:00 GMT

←Older revision		Revision as of 16:38, 11 February 2013
Line 1:		Line 1:
	==Overview/Checklist==		==Overview/Checklist==
	* Make sure everything is properly prepared for upload (see [[Preparing_Collections_on_the_S_Drive_for_Online_Delivery_and_Storage \| Preparing Collections]])		* Make sure everything is properly prepared for upload (see [[Preparing_Collections_on_the_S_Drive_for_Online_Delivery_and_Storage \| Preparing Collections]])
		+	** Move the metadata spreadsheet to S:\Administrative\Pipeline\collectionInfo\forMDlib\needsRemediation
	* Run ''makeJpegs'', a script which		* Run ''makeJpegs'', a script which
	** makes a copy of all the MODS		** makes a copy of all the MODS

Kgmatheny at 15:59, 11 February 2013

Kgmatheny — Mon, 11 Feb 2013 15:59:36 GMT

←Older revision		Revision as of 15:59, 11 February 2013
Line 1:		Line 1:
-	The first sections of the attached diagram which pertain to Digital Services are delineated in [[Preparing_Collections_on_the_S_Drive_for_Online_Delivery_and_Storage]].
-
	==Overview/Checklist==		==Overview/Checklist==
-	* Make sure everything is ~~in order in the Completed folder on the Share drive~~	+	* Make sure everything is properly prepared for upload (see [[Preparing_Collections_on_the_S_Drive_for_Online_Delivery_and_Storage \| Preparing Collections]])
-	* Make MODS	+
	* Run ''makeJpegs'', a script which		* Run ''makeJpegs'', a script which
	** makes a copy of all the MODS		** makes a copy of all the MODS
Line 70:		Line 67:

	== Archiving content ==		== Archiving content ==
		+
		+	The first sections of the attached diagram which pertain to Digital Services are delineated in [[Preparing_Collections_on_the_S_Drive_for_Online_Delivery_and_Storage]].

	See lines 9-25 and 31-33 on this page: [[Moving_Content_To_Long-Term_Storage]]		See lines 9-25 and 31-33 on this page: [[Moving_Content_To_Long-Term_Storage]]

Kgmatheny at 14:56, 29 October 2012

Kgmatheny — Mon, 29 Oct 2012 14:56:05 GMT

←Older revision		Revision as of 14:56, 29 October 2012
Line 18:		Line 18:
	== Important things to remember about working with the server==		== Important things to remember about working with the server==

-	* '''~~Do not~~ run upload scripts on more than one collection at a time.''' Yes, it is possible to run the same scripts on multiple collections at a time. But just because it's possible doesn't mean you should do it. If something goes wrong, it can be very difficult to disentangle the separate collections in order to fix the problem.	+	* '''Think long and hard before you run upload scripts on more than one collection at a time.''' Yes, it is possible to run the same scripts on multiple collections at a time. But just because it's possible doesn't mean you should do it. If something goes wrong, it can be very difficult to disentangle the separate collections in order to fix the problem.
	* '''Do not run the same script more than once at a time on a particular collection.''' If a script encounters a problem such that it doesn't finish running, do not just run it again. We do not want two instances of the same script running simultaneously. This gets very confusing and multiplies the possibility of error exponentially!		* '''Do not run the same script more than once at a time on a particular collection.''' If a script encounters a problem such that it doesn't finish running, do not just run it again. We do not want two instances of the same script running simultaneously. This gets very confusing and multiplies the possibility of error exponentially!
	* '''If you need to kill a script, please see someone who manages the scripts (Jody or the Repository Manager).''' Closing the current Terminal Window (the command line interface) of SSH Secure File Transfer does not kill the scripts that are running. This is especially important to remember for makeJpegs, which splits into two processes as soon as you run it (whether you see results of this in the UploadArea or not).		* '''If you need to kill a script, please see someone who manages the scripts (Jody or the Repository Manager).''' Closing the current Terminal Window (the command line interface) of SSH Secure File Transfer does not kill the scripts that are running. This is especially important to remember for makeJpegs, which splits into two processes as soon as you run it (whether you see results of this in the UploadArea or not).

Kgmatheny at 15:13, 26 October 2012

Kgmatheny — Fri, 26 Oct 2012 15:13:53 GMT

←Older revision		Revision as of 15:13, 26 October 2012
Line 47:		Line 47:

	===Getting content to Acumen===		===Getting content to Acumen===
-	# In the UploadArea/scripts directory, type in 'relocate_all' ([[Image:relocate_all.txt]]. This script will move all the content just uploaded into the correct directories in Acumen. If anything remains in the above-mentioned directories after the script has completed, there's a problem which must be repaired. One thing to check, if there's no errors in the output, is [[Server Permissions]].	+	# In the UploadArea/scripts directory, type in 'relocate_all' ([[Image:relocate_all.txt]]). This script will move all the content just uploaded into the correct directories in Acumen. If anything remains in the above-mentioned directories after the script has completed, there's a problem which must be repaired. One thing to check, if there's no errors in the output, is [[Server Permissions]].
	# In the scripts directory (on the same level as the UploadArea, AUDIO and CABANISS directories, run 'findMissing'. This script will hunt through Acumen to make sure there is a MODS file for every item, and at least one derivative for each MODS file. Any errors will be found in the output file written to the scripts/output directory. If errors are found, regenerate those MODS and/or JPEGs/MP3s and rerun relocate_all. Then run this script again to ensure all errors have been remedied. (Script here: [[Image:findMissing.txt]])		# In the scripts directory (on the same level as the UploadArea, AUDIO and CABANISS directories, run 'findMissing'. This script will hunt through Acumen to make sure there is a MODS file for every item, and at least one derivative for each MODS file. Any errors will be found in the output file written to the scripts/output directory. If errors are found, regenerate those MODS and/or JPEGs/MP3s and rerun relocate_all. Then run this script again to ensure all errors have been remedied. (Script here: [[Image:findMissing.txt]])
	# Check the indexing of the uploaded content after about 24 - 30 hours to verify. Once the content appears online in Acumen, it is safe to do the next step for new collections; for ongoing collections that already have a listing on the collection page, you may proceed to the next step without waiting. The collection list can be viewed here: [http://www.lib.ua.edu/digital/browse Collection List Online].		# Check the indexing of the uploaded content after about 24 - 30 hours to verify. Once the content appears online in Acumen, it is safe to do the next step for new collections; for ongoing collections that already have a listing on the collection page, you may proceed to the next step without waiting. The collection list can be viewed here: [http://www.lib.ua.edu/digital/browse Collection List Online].
Line 57:		Line 57:

	===Getting content to storage===		===Getting content to storage===
-	# [[Image:moveContent.txt]] will move content to the deposits directory on the storage server, where it will be prepared for archiving at a later date.	+	MoveContent[[Image:moveContent.txt]] will move content to the deposits directory on the storage server, where it will be prepared for archiving at a later date.
-	## Additionally, it will check the collection-level xml file (in the Admin directory), then add it to the online database which feeds the collection list online. Thus, this script should NOT be run for new collections until the content is indexed (as described in the previous section); otherwise, the link from the collection listing will be dead.	+	# In the UploadArea/scripts directory, run 'moveContent.' This script will:
-	~~## After making the copy, it will verify that each file copied without alteration, and then delete the copy on the share drive.~~	+
-	~~# Running the moveContent script~~	+
-	## In the UploadArea/scripts directory, run 'moveContent.' ~~(Again, if this is a new collection, wait until the files have been indexed and content is viewable online, so there will not be a dead link in the collection list.)~~ This script will:	+
	## check the database for existing info about this collection, and provide you with whatever we already know, so you can correct it with your collection xml file,		## check the database for existing info about this collection, and provide you with whatever we already know, so you can correct it with your collection xml file,
-	## update our database and online collection xml file, if yours is new or changed -- adding the online link to the collection if new	+	## update our database and online collection xml file, if yours is new or changed -- adding the online link to the collection if new. ''This is why, if this is a new collection, you have to wait to run this script until the files have been indexed and content is viewable online, so there will not be a dead link in the collection list.''
	## update or add the icon image if you are providing it for a collection thumbnail		## update or add the icon image if you are providing it for a collection thumbnail
	## copy the archival content to the deposits/content directory on the server, for processing into the archive		## copy the archival content to the deposits/content directory on the server, for processing into the archive

Kgmatheny at 15:08, 26 October 2012

Kgmatheny — Fri, 26 Oct 2012 15:08:57 GMT

←Older revision		Revision as of 15:08, 26 October 2012
Line 60:		Line 60:
	## Additionally, it will check the collection-level xml file (in the Admin directory), then add it to the online database which feeds the collection list online. Thus, this script should NOT be run for new collections until the content is indexed (as described in the previous section); otherwise, the link from the collection listing will be dead.		## Additionally, it will check the collection-level xml file (in the Admin directory), then add it to the online database which feeds the collection list online. Thus, this script should NOT be run for new collections until the content is indexed (as described in the previous section); otherwise, the link from the collection listing will be dead.
	## After making the copy, it will verify that each file copied without alteration, and then delete the copy on the share drive.		## After making the copy, it will verify that each file copied without alteration, and then delete the copy on the share drive.
-	##'''If any files remain in the directories on the share drive, they did NOT copy to the server!!''' Run the script again, as there may have been a failure of the copy across the network. If this fails, the file will need to be moved manually, and the problem encountered by the script must be resolved.
	# Running the moveContent script		# Running the moveContent script
	## In the UploadArea/scripts directory, run 'moveContent.' (Again, if this is a new collection, wait until the files have been indexed and content is viewable online, so there will not be a dead link in the collection list.) This script will:		## In the UploadArea/scripts directory, run 'moveContent.' (Again, if this is a new collection, wait until the files have been indexed and content is viewable online, so there will not be a dead link in the collection list.) This script will:
Line 70:		Line 69:
	## output errors into a file in the output directory		## output errors into a file in the output directory
	# Check back after a few hours and look at the output file to verify that there were no problems and that the script completed. If you want, you can watch the files being uploaded in the deposits/content directory on libcontent1. :-)		# Check back after a few hours and look at the output file to verify that there were no problems and that the script completed. If you want, you can watch the files being uploaded in the deposits/content directory on libcontent1. :-)
-	# Check the share drive directories for any files that still remain. If any ~~archival~~ files ~~are still~~ there~~, rerun moveContent. There~~ may have been a failure in the network ~~connection between the servers~~. If the ~~files still remain~~, notify Jody.	+	# Check the share drive directories for any files that still remain. '''If any files remain in the directories on the share drive, they did NOT copy to the server!!''' Run the script again, as there may have been a failure of the copy across the network. If this fails, the file will need to be moved manually, and the problem encountered by the script must be resolved. Please notify Jody ASAP.
-	~~#If no files remain in the directory on the share drive, delete this directory.~~	+
	# Exit out of secure shell. Good work!!		# Exit out of secure shell. Good work!!

Kgmatheny: Major additions and rearranging

Kgmatheny — Fri, 26 Oct 2012 15:06:40 GMT

Major additions and rearranging

←Older revision		Revision as of 15:06, 26 October 2012
Line 1:		Line 1:
	The first sections of the attached diagram which pertain to Digital Services are delineated in [[Preparing_Collections_on_the_S_Drive_for_Online_Delivery_and_Storage]].		The first sections of the attached diagram which pertain to Digital Services are delineated in [[Preparing_Collections_on_the_S_Drive_for_Online_Delivery_and_Storage]].

-	~~Here's~~ the ~~steps for creating derivatives and moving content off~~ the ~~share~~ drive:	+	==Overview/Checklist==
-		+	* Make sure everything is in order in the Completed folder on the Share drive
-	~~== Preliminary steps on~~ the ~~server ==~~	+	* Make MODS
-		+	* Run ''makeJpegs'', a script which
-	~~# make sure that~~ the ~~Windows share~~ is ~~mounted: type~~ into the ~~ssh window~~ on libcontent1~~: `ls /cifs-mount` -- if no listing appears, or the window hangs up, you need~~ to ~~mount the share drive. Otherwise~~, ~~proceed to getting content from the share drive~~...	+	** makes a copy of all the MODS
-	~~# To mount the Windows drive~~, ~~type this~~ in on the ~~commandline on libcontent1: `sudo mount -t cifs -o username=jjcolonnaromano~~,~~domain=lib //libfs1.lib.ua~~-~~net.ua.edu/share/Digital\ Projects/ /cifs~~-~~mount` and use the password~~ for ~~share for Jeremiah. If successful~~, the ~~command in~~ the ~~last step will show you~~ the ~~directories within~~ the ~~Digital Projects folder on~~ Share.	+	** makes a copy of any .txt files in the Transripts folder (if there is one)
		+	** creates a derivative of all the TIFFs
		+	** puts them into the UploadArea on libcontent1
		+	* Wait for makeJpegs to finish, then...
		+	* Run ''relocate_all'', a script which puts that new content in the right places in Acumen
		+	* If this is a new collection, wait for 24 hours or so -- for Acumen to index it -- before moving forward
		+	* Run ''moveContent'', a script which
		+	** sends the collection XML to the Acumen database
		+	** move the original files from the Share drive to storage, in preparation for archiving

	== Important things to remember about working with the server==		== Important things to remember about working with the server==
Line 14:		Line 22:
	* '''If you need to kill a script, please see someone who manages the scripts (Jody or the Repository Manager).''' Closing the current Terminal Window (the command line interface) of SSH Secure File Transfer does not kill the scripts that are running. This is especially important to remember for makeJpegs, which splits into two processes as soon as you run it (whether you see results of this in the UploadArea or not).		* '''If you need to kill a script, please see someone who manages the scripts (Jody or the Repository Manager).''' Closing the current Terminal Window (the command line interface) of SSH Secure File Transfer does not kill the scripts that are running. This is especially important to remember for makeJpegs, which splits into two processes as soon as you run it (whether you see results of this in the UploadArea or not).
	* '''When checking a libcontent1 folder for ongoing upload status, hit Refresh.''' The SSH File Transfer Window will not refresh on its own.		* '''When checking a libcontent1 folder for ongoing upload status, hit Refresh.''' The SSH File Transfer Window will not refresh on its own.
		+
		+	<font color=grey>
		+	== Preliminary steps on the server ==
		+
		+	# make sure that the Windows share is mounted: type into the ssh window on libcontent1: `ls /cifs-mount` -- if no listing appears, or the window hangs up, you need to mount the share drive. Otherwise, proceed to getting content from the share drive...
		+	# To mount the Windows drive, type this in on the commandline on libcontent1: `sudo mount -t cifs -o username=jjcolonnaromano,domain=lib //libfs1.lib.ua-net.ua.edu/share/Digital\ Projects/ /cifs-mount` and use the password for share for Jeremiah. If successful, the command in the last step will show you the directories within the Digital Projects folder on Share.
		+	</font>

	== Getting content from the share drive to <u>''live''</u> in Acumen ==		== Getting content from the share drive to <u>''live''</u> in Acumen ==

		+	===Preparation===
	# Make sure content is in the Completed directory, and that all quality control scripts have been run, and all corrections made.		# Make sure content is in the Completed directory, and that all quality control scripts have been run, and all corrections made.
-	# Create the MODS from the exported text spreadsheet ([[Making_MODS]]), and place them in a MODS directory within the Metadata directory.	+	# Create the MODS from the exported text spreadsheet ([[Making_MODS]]), and place them in a MODS directory within the Metadata directory. If this is a batch, rather than a complete collection, see [[Batches]] for info on what parts of this process might be different.
	# Log onto libcontent1 and change into the UploadArea/scripts directory. (see [[Command-line_Work_on_Linux_Server]])		# Log onto libcontent1 and change into the UploadArea/scripts directory. (see [[Command-line_Work_on_Linux_Server]])
		+
		+	===Getting content to libcontent1 server===
	# Type in 'makeJpegs' and answer the questions that arise. If errors appear on the commandline or in the output file, repair them. The output file will be located in the UploadArea/output directory, and the script will tell you its name, which is timestamped. Script here: ([[Image:makeJpegs.txt]])		# Type in 'makeJpegs' and answer the questions that arise. If errors appear on the commandline or in the output file, repair them. The output file will be located in the UploadArea/output directory, and the script will tell you its name, which is timestamped. Script here: ([[Image:makeJpegs.txt]])
	## The makeJpegs script will do some minor QC, including checking to see if there is a MODS for each item-level object and an object for each item-level MODS.		## The makeJpegs script will do some minor QC, including checking to see if there is a MODS for each item-level object and an object for each item-level MODS.
	## If there are no problems, it will copy the MODS files to the UploadArea/MODS directory on the server, copy the transcripts to the UploadArea/transcripts directory on the server, and will generate JPEG derivatives (a thumb ending in _128.jpg and a large image ending in _2048.jpg) and place them in the UploadArea/jpegs directory on the server.		## If there are no problems, it will copy the MODS files to the UploadArea/MODS directory on the server, copy the transcripts to the UploadArea/transcripts directory on the server, and will generate JPEG derivatives (a thumb ending in _128.jpg and a large image ending in _2048.jpg) and place them in the UploadArea/jpegs directory on the server.
-	## If you provide an [[OCR List \| ocrList.txt]] file in the Admin directory then the selected items and their child files will be OCR'd. If any on that list are already online, the script will look for the tiffs in the archive, look for existing text files in Acumen - and if it finds the former and not the latter, it will OCR the tiff and place it in Acumen. For the content in the ocrList which is currently being uploaded, the OCR will be placed in the UploadArea/ocr directory. For content not found, there will be a list of the unlocated TIFFs in the output file.	+	## OCR
-	## If you select to OCR all content, it will OCR every TIFF you are uploading; by default it will OCR all transcript tiffs unless a corresponding text file exists in the upload directory (indicating a corrected transcript). You may also select to OCR all of this collection, including what is already online. (~~See previous entry~~ for ~~how it will handle this.~~)	+	### If you provide an [[OCR List \| ocrList.txt]] file in the Admin directory then the selected items and their child files will be OCR'd.
		+	### If you select to OCR all content, it will OCR every TIFF you are uploading; by default it will OCR all transcript tiffs unless a corresponding text file exists in the upload directory (indicating a corrected transcript). You may also select to OCR all of this collection, including what is already online. It will handle discrepancies the same way it does if you'd provided a list (go [[OCR List \| here]] for information on that).
	# If there were no errors on the command line, check the output file in a few hours to verify that the script completed these tasks; and check the results on the server in the above-named directories. If all is well, proceed to the next step.		# If there were no errors on the command line, check the output file in a few hours to verify that the script completed these tasks; and check the results on the server in the above-named directories. If all is well, proceed to the next step.
		+
		+	===Getting content to Acumen===
	# In the UploadArea/scripts directory, type in 'relocate_all' ([[Image:relocate_all.txt]]. This script will move all the content just uploaded into the correct directories in Acumen. If anything remains in the above-mentioned directories after the script has completed, there's a problem which must be repaired. One thing to check, if there's no errors in the output, is [[Server Permissions]].		# In the UploadArea/scripts directory, type in 'relocate_all' ([[Image:relocate_all.txt]]. This script will move all the content just uploaded into the correct directories in Acumen. If anything remains in the above-mentioned directories after the script has completed, there's a problem which must be repaired. One thing to check, if there's no errors in the output, is [[Server Permissions]].
	# In the scripts directory (on the same level as the UploadArea, AUDIO and CABANISS directories, run 'findMissing'. This script will hunt through Acumen to make sure there is a MODS file for every item, and at least one derivative for each MODS file. Any errors will be found in the output file written to the scripts/output directory. If errors are found, regenerate those MODS and/or JPEGs/MP3s and rerun relocate_all. Then run this script again to ensure all errors have been remedied. (Script here: [[Image:findMissing.txt]])		# In the scripts directory (on the same level as the UploadArea, AUDIO and CABANISS directories, run 'findMissing'. This script will hunt through Acumen to make sure there is a MODS file for every item, and at least one derivative for each MODS file. Any errors will be found in the output file written to the scripts/output directory. If errors are found, regenerate those MODS and/or JPEGs/MP3s and rerun relocate_all. Then run this script again to ensure all errors have been remedied. (Script here: [[Image:findMissing.txt]])
-	# Check the indexing of the uploaded content after about 24 - 30 hours to verify. Once the content appears online in Acumen, it is safe to do the next step for new collections; for ongoing collections that already have a listing on the collection page, you may proceed to the next step without waiting. ~~(We don't want to create dead links in the collection list, by adding listings before the content is findable in Acumen.)~~ The collection list can be viewed here: [http://www.lib.ua.edu/digital/browse Collection List Online].	+	# Check the indexing of the uploaded content after about 24 - 30 hours to verify. Once the content appears online in Acumen, it is safe to do the next step for new collections; for ongoing collections that already have a listing on the collection page, you may proceed to the next step without waiting. The collection list can be viewed here: [http://www.lib.ua.edu/digital/browse Collection List Online].

	== Getting content from the share drive to the storage server to prepare for archiving ==		== Getting content from the share drive to the storage server to prepare for archiving ==

-	~~First, check~~ the file NOTIFICATIONS.txt in S:\Digital Projects\Digital_Coll_Complete to verify that none of the content is part of collections that need to have special attention paid to them by Jody or the metadata unit.	+	===Preparation===
-		+	Check the file NOTIFICATIONS.txt in S:\Digital Projects\Digital_Coll_Complete to verify that none of the content is part of collections that need to have special attention paid to them by Jody or the metadata unit.
-	~~The moveContent script ([[Image:moveContent.txt]]) will move content to the deposits directory on the storage server, where it will be prepared for archiving at a later date.~~	+
-		+
-	Additionally, it will check the collection-level xml file (in the Admin directory), then add it to the online database which feeds the collection list online. Thus, this script should NOT be run for new collections until the content is indexed (as described in the previous section); otherwise, the link from the collection listing will be dead.	+
-		+
-	~~After making the copy, it will verify that each file copied without alteration, and then delete the copy on the share drive.~~	+
-		+
-	'''If any files remain in the directories on the share drive, they did NOT copy to the server!!''' Run the script again, as there may have been a failure of the copy across the network. If this fails, the file will need to be moved manually, and the problem encountered by the script must be resolved.	+

-	# In the UploadArea/scripts directory, run 'moveContent.' (Again, if this is a new collection, wait until the files have been indexed and content is viewable online, so there will not be a dead link in the collection list.) This script will:	+	===Getting content to storage===
		+	# [[Image:moveContent.txt]] will move content to the deposits directory on the storage server, where it will be prepared for archiving at a later date.
		+	## Additionally, it will check the collection-level xml file (in the Admin directory), then add it to the online database which feeds the collection list online. Thus, this script should NOT be run for new collections until the content is indexed (as described in the previous section); otherwise, the link from the collection listing will be dead.
		+	## After making the copy, it will verify that each file copied without alteration, and then delete the copy on the share drive.
		+	##'''If any files remain in the directories on the share drive, they did NOT copy to the server!!''' Run the script again, as there may have been a failure of the copy across the network. If this fails, the file will need to be moved manually, and the problem encountered by the script must be resolved.
		+	# Running the moveContent script
		+	## In the UploadArea/scripts directory, run 'moveContent.' (Again, if this is a new collection, wait until the files have been indexed and content is viewable online, so there will not be a dead link in the collection list.) This script will:
	## check the database for existing info about this collection, and provide you with whatever we already know, so you can correct it with your collection xml file,		## check the database for existing info about this collection, and provide you with whatever we already know, so you can correct it with your collection xml file,
	## update our database and online collection xml file, if yours is new or changed -- adding the online link to the collection if new		## update our database and online collection xml file, if yours is new or changed -- adding the online link to the collection if new
Line 57:		Line 77:

	See lines 9-25 and 31-33 on this page: [[Moving_Content_To_Long-Term_Storage]]		See lines 9-25 and 31-33 on this page: [[Moving_Content_To_Long-Term_Storage]]
-	----------------------------------------------------------------

	The movement of metadata is spelled out in [[Metadata_Movement]] and the process for uploading MODS is in [[Uploading_MODS]].		The movement of metadata is spelled out in [[Metadata_Movement]] and the process for uploading MODS is in [[Uploading_MODS]].

Kgmatheny: linked OCR List

Kgmatheny — Tue, 23 Oct 2012 15:16:20 GMT

linked OCR List

←Older revision		Revision as of 15:16, 23 October 2012
Line 23:		Line 23:
	## The makeJpegs script will do some minor QC, including checking to see if there is a MODS for each item-level object and an object for each item-level MODS.		## The makeJpegs script will do some minor QC, including checking to see if there is a MODS for each item-level object and an object for each item-level MODS.
	## If there are no problems, it will copy the MODS files to the UploadArea/MODS directory on the server, copy the transcripts to the UploadArea/transcripts directory on the server, and will generate JPEG derivatives (a thumb ending in _128.jpg and a large image ending in _2048.jpg) and place them in the UploadArea/jpegs directory on the server.		## If there are no problems, it will copy the MODS files to the UploadArea/MODS directory on the server, copy the transcripts to the UploadArea/transcripts directory on the server, and will generate JPEG derivatives (a thumb ending in _128.jpg and a large image ending in _2048.jpg) and place them in the UploadArea/jpegs directory on the server.
-	## If you provide an ocrList.txt file in the Admin directory then the selected items and their child files will be OCR'd. If any on that list are already online, the script will look for the tiffs in the archive, look for existing text files in Acumen - and if it finds the former and not the latter, it will OCR the tiff and place it in Acumen. For the content in the ocrList which is currently being uploaded, the OCR will be placed in the UploadArea/ocr directory. For content not found, there will be a list of the unlocated TIFFs in the output file.	+	## If you provide an [[OCR List \| ocrList.txt]] file in the Admin directory then the selected items and their child files will be OCR'd. If any on that list are already online, the script will look for the tiffs in the archive, look for existing text files in Acumen - and if it finds the former and not the latter, it will OCR the tiff and place it in Acumen. For the content in the ocrList which is currently being uploaded, the OCR will be placed in the UploadArea/ocr directory. For content not found, there will be a list of the unlocated TIFFs in the output file.
	## If you select to OCR all content, it will OCR every TIFF you are uploading; by default it will OCR all transcript tiffs unless a corresponding text file exists in the upload directory (indicating a corrected transcript). You may also select to OCR all of this collection, including what is already online. (See previous entry for how it will handle this.)		## If you select to OCR all content, it will OCR every TIFF you are uploading; by default it will OCR all transcript tiffs unless a corresponding text file exists in the upload directory (indicating a corrected transcript). You may also select to OCR all of this collection, including what is already online. (See previous entry for how it will handle this.)
	# If there were no errors on the command line, check the output file in a few hours to verify that the script completed these tasks; and check the results on the server in the above-named directories. If all is well, proceed to the next step.		# If there were no errors on the command line, check the output file in a few hours to verify that the script completed these tasks; and check the results on the server in the above-named directories. If all is well, proceed to the next step.

Kgmatheny at 15:10, 23 October 2012

Kgmatheny — Tue, 23 Oct 2012 15:10:44 GMT

←Older revision		Revision as of 15:10, 23 October 2012
Line 10:		Line 10:
	== Important things to remember about working with the server==		== Important things to remember about working with the server==

-	# '''Do not run upload scripts on more than one collection at a time.''' Yes, it is possible to run the same scripts on multiple collections at a time. But just because it's possible doesn't mean you should do it. If something goes wrong, it can be very difficult to disentangle the separate collections in order to fix the problem.	+	* '''Do not run upload scripts on more than one collection at a time.''' Yes, it is possible to run the same scripts on multiple collections at a time. But just because it's possible doesn't mean you should do it. If something goes wrong, it can be very difficult to disentangle the separate collections in order to fix the problem.
-	# '''~~Please~~ see someone who manages the scripts (Jody or the Repository Manager) ~~if you need to kill a script~~.''' Closing the current Terminal Window (the command line interface) of SSH Secure File Transfer does not kill the scripts that are running. This is especially important to remember for makeJpegs, which splits into two processes as soon as you run it (whether you see results of this in the UploadArea or not).	+	* '''Do not run the same script more than once at a time on a particular collection.''' If a script encounters a problem such that it doesn't finish running, do not just run it again. We do not want two instances of the same script running simultaneously. This gets very confusing and multiplies the possibility of error exponentially!
-	~~# Remember to~~ '''~~hit Refresh when~~ checking libcontent1 ~~folders'''~~ for upload status ~~through the~~ SSH File Transfer Window ~~(the interface that looks more like Windows explorer). It does~~ not refresh on its own.	+	* '''If you need to kill a script, please see someone who manages the scripts (Jody or the Repository Manager).''' Closing the current Terminal Window (the command line interface) of SSH Secure File Transfer does not kill the scripts that are running. This is especially important to remember for makeJpegs, which splits into two processes as soon as you run it (whether you see results of this in the UploadArea or not).
		+	* '''When checking a libcontent1 folder for ongoing upload status, hit Refresh.''' The SSH File Transfer Window will not refresh on its own.

	== Getting content from the share drive to <u>''live''</u> in Acumen ==		== Getting content from the share drive to <u>''live''</u> in Acumen ==

Kgmatheny: Added things to remember, removed OCR info for another page

Kgmatheny — Tue, 23 Oct 2012 14:30:40 GMT

Added things to remember, removed OCR info for another page

←Older revision		Revision as of 14:30, 23 October 2012
Line 7:		Line 7:
	# make sure that the Windows share is mounted: type into the ssh window on libcontent1: `ls /cifs-mount` -- if no listing appears, or the window hangs up, you need to mount the share drive. Otherwise, proceed to getting content from the share drive...		# make sure that the Windows share is mounted: type into the ssh window on libcontent1: `ls /cifs-mount` -- if no listing appears, or the window hangs up, you need to mount the share drive. Otherwise, proceed to getting content from the share drive...
	# To mount the Windows drive, type this in on the commandline on libcontent1: `sudo mount -t cifs -o username=jjcolonnaromano,domain=lib //libfs1.lib.ua-net.ua.edu/share/Digital\ Projects/ /cifs-mount` and use the password for share for Jeremiah. If successful, the command in the last step will show you the directories within the Digital Projects folder on Share.		# To mount the Windows drive, type this in on the commandline on libcontent1: `sudo mount -t cifs -o username=jjcolonnaromano,domain=lib //libfs1.lib.ua-net.ua.edu/share/Digital\ Projects/ /cifs-mount` and use the password for share for Jeremiah. If successful, the command in the last step will show you the directories within the Digital Projects folder on Share.
		+
		+	== Important things to remember about working with the server==
		+
		+	# '''Do not run upload scripts on more than one collection at a time.''' Yes, it is possible to run the same scripts on multiple collections at a time. But just because it's possible doesn't mean you should do it. If something goes wrong, it can be very difficult to disentangle the separate collections in order to fix the problem.
		+	# '''Please see someone who manages the scripts (Jody or the Repository Manager) if you need to kill a script.''' Closing the current Terminal Window (the command line interface) of SSH Secure File Transfer does not kill the scripts that are running. This is especially important to remember for makeJpegs, which splits into two processes as soon as you run it (whether you see results of this in the UploadArea or not).
		+	# Remember to '''hit Refresh when checking libcontent1 folders''' for upload status through the SSH File Transfer Window (the interface that looks more like Windows explorer). It does not refresh on its own.

	== Getting content from the share drive to <u>''live''</u> in Acumen ==		== Getting content from the share drive to <u>''live''</u> in Acumen ==
Line 16:		Line 22:
	## The makeJpegs script will do some minor QC, including checking to see if there is a MODS for each item-level object and an object for each item-level MODS.		## The makeJpegs script will do some minor QC, including checking to see if there is a MODS for each item-level object and an object for each item-level MODS.
	## If there are no problems, it will copy the MODS files to the UploadArea/MODS directory on the server, copy the transcripts to the UploadArea/transcripts directory on the server, and will generate JPEG derivatives (a thumb ending in _128.jpg and a large image ending in _2048.jpg) and place them in the UploadArea/jpegs directory on the server.		## If there are no problems, it will copy the MODS files to the UploadArea/MODS directory on the server, copy the transcripts to the UploadArea/transcripts directory on the server, and will generate JPEG derivatives (a thumb ending in _128.jpg and a large image ending in _2048.jpg) and place them in the UploadArea/jpegs directory on the server.
-	## If you provide an ocrList.txt file in the Admin directory ~~which contains a list of item numbers to be OCR'd (each to be OCR'd is followed by a tab and then a one ("1"), with one item number per line (those followed with zeros will be ignored) --~~ then the selected items and their child files will be OCR'd. If any on that list are already online, the script will look for the tiffs in the archive, look for existing text files in Acumen - and if it finds the former and not the latter, it will OCR the tiff and place it in Acumen. For the content in the ocrList which is currently being uploaded, the OCR will be placed in the UploadArea/ocr directory. For content not found, there will be a list of the unlocated TIFFs in the output file.	+	## If you provide an ocrList.txt file in the Admin directory then the selected items and their child files will be OCR'd. If any on that list are already online, the script will look for the tiffs in the archive, look for existing text files in Acumen - and if it finds the former and not the latter, it will OCR the tiff and place it in Acumen. For the content in the ocrList which is currently being uploaded, the OCR will be placed in the UploadArea/ocr directory. For content not found, there will be a list of the unlocated TIFFs in the output file.
-	~~###Note: ocrList.txt files must be prefaced with the collection number, followed by a dot.~~	+
-	~~###ex: u0003_0001577.ocrList.txt~~	+
	## If you select to OCR all content, it will OCR every TIFF you are uploading; by default it will OCR all transcript tiffs unless a corresponding text file exists in the upload directory (indicating a corrected transcript). You may also select to OCR all of this collection, including what is already online. (See previous entry for how it will handle this.)		## If you select to OCR all content, it will OCR every TIFF you are uploading; by default it will OCR all transcript tiffs unless a corresponding text file exists in the upload directory (indicating a corrected transcript). You may also select to OCR all of this collection, including what is already online. (See previous entry for how it will handle this.)
	# If there were no errors on the command line, check the output file in a few hours to verify that the script completed these tasks; and check the results on the server in the above-named directories. If all is well, proceed to the next step.		# If there were no errors on the command line, check the output file in a few hours to verify that the script completed these tasks; and check the results on the server in the above-named directories. If all is well, proceed to the next step.

Kgmatheny at 16:58, 17 September 2012

Kgmatheny — Mon, 17 Sep 2012 16:58:18 GMT

←Older revision		Revision as of 16:58, 17 September 2012
Line 26:		Line 26:

	== Getting content from the share drive to the storage server to prepare for archiving ==		== Getting content from the share drive to the storage server to prepare for archiving ==
		+
		+	First, check the file NOTIFICATIONS.txt in S:\Digital Projects\Digital_Coll_Complete to verify that none of the content is part of collections that need to have special attention paid to them by Jody or the metadata unit.

	The moveContent script ([[Image:moveContent.txt]]) will move content to the deposits directory on the storage server, where it will be prepared for archiving at a later date.		The moveContent script ([[Image:moveContent.txt]]) will move content to the deposits directory on the storage server, where it will be prepared for archiving at a later date.