Moving Content To Long-Term Storage - Revision history

Jlderidder at 21:15, 18 December 2012

Jlderidder — Tue, 18 Dec 2012 21:15:48 GMT

←Older revision		Revision as of 21:15, 18 December 2012
Line 1:		Line 1:
	Currently, we are developing collections on a shared Windows drive called "Share" in the Digital Projects folder.		Currently, we are developing collections on a shared Windows drive called "Share" in the Digital Projects folder.

-	We are storing content long-term on a Linux server.	+	We are storing content long-term on a SUSE Linux server.

-	We are delivering content on ~~another~~ Linux server via ~~CONTENTdm~~, ~~and Tonio Loewald (of the Office of Library Technology) has developed an alternative (Linux-based) delivery solution, Acumen~~ [http://acumen.lib.ua./edu http://acumen.lib.ua./edu] which runs off a web-enabled version of our storage area.	+	We are delivering content on the same Linux server via Acumen, [http://acumen.lib.ua./edu http://acumen.lib.ua./edu] which runs off a web-enabled version of our storage area.

	[[If moving from share]]		[[If moving from share]]

	[[If already in deposits]] (uploaded by staff)		[[If already in deposits]] (uploaded by staff)

Jlderidder at 14:09, 13 October 2010

Jlderidder — Wed, 13 Oct 2010 14:09:36 GMT

(Difference between revisions)

Jlderidder at 22:57, 25 February 2010

Jlderidder — Thu, 25 Feb 2010 22:57:22 GMT

←Older revision		Revision as of 22:57, 25 February 2010
Line 73:		Line 73:
	and notify Mary		and notify Mary

-	11) RUN testIncoming (in /srv/scripts/qc; script available here: [[Image:TestIncoming.txt]]) and look at the output for problems.	+	11) RUN testIncoming (in /srv/scripts/qc; script available here: [[Image:TestIncoming.txt]]) and look at the output
		+	for problems.

-	12) RUN cleanCollInfo (in /srv/scripts/qc) to clean up the collection xml record in the Admin directory of each collection.	+	12) RUN cleanCollInfo (in /srv/scripts/qc) to clean up the collection xml record in the Admin directory of each collection.
-	This removes Word encodings and checks for valid content. This also puts a copy of the collection xml record	+	More about this xml record can be found here: [[Collection Information]]
		+	This script removes Word encodings and checks for valid content. This also puts a copy of the collection xml record
	in the /srv/deposits/hold directory to be used later to feed the database that feeds our online list of collections		in the /srv/deposits/hold directory to be used later to feed the database that feeds our online list of collections
	with links, descriptions, and titles, seen here: [[http://www.lib.ua.edu/digital/browse Browse Digital Collections]]		with links, descriptions, and titles, seen here: [[http://www.lib.ua.edu/digital/browse Browse Digital Collections]]

Jlderidder at 22:55, 25 February 2010

Jlderidder — Thu, 25 Feb 2010 22:55:08 GMT

←Older revision		Revision as of 22:55, 25 February 2010
Line 159:		Line 159:
	if no content: rm -r /srv/deposits/content/*		if no content: rm -r /srv/deposits/content/*
	(this will recursively delete all content in the directory)		(this will recursively delete all content in the directory)
		+
		+	'''MAKE IT LIVE IN ACUMEN!! '''
		+	-----------------------------------

	26) go to /srv/scripts/surfacing and run copychange (script available here: [[Image:Copychange.txt]])		26) go to /srv/scripts/surfacing and run copychange (script available here: [[Image:Copychange.txt]])
Line 211:		Line 214:


-	''updated 22 February 2010'' [[User:Jlderidder\|Jlderidder]]	+	''updated 25 February 2010'' [[User:Jlderidder\|Jlderidder]]

Jlderidder at 22:53, 25 February 2010

Jlderidder — Thu, 25 Feb 2010 22:53:43 GMT

(Difference between revisions)

Jlderidder at 22:44, 23 February 2010

Jlderidder — Tue, 23 Feb 2010 22:44:57 GMT

(Difference between revisions)

Jlderidder at 22:43, 23 February 2010

Jlderidder — Tue, 23 Feb 2010 22:43:55 GMT

←Older revision		Revision as of 22:43, 23 February 2010
Line 73:		Line 73:

	11) RUN testIncoming (in /srv/scripts/qc; script available here: [[Image:TestIncoming.txt]]) and look at the output for problems.		11) RUN testIncoming (in /srv/scripts/qc; script available here: [[Image:TestIncoming.txt]]) and look at the output for problems.
-
-
-
-	'''UPDATE DATABASE WITH COLLECTION INFO'''
-	------------------------------------
-	12) If the collection has an icon: make sure it is in the Admin folder, named for the collection.
-	For example, u0003_0002121.icon.png for the collection above. (You will need to copy this
-	to the CONTENTdm server (content.lib.ua.edu) and place it in the /usr/local/Content4/docs/cdm4/images
-	directory, with 664 access rights "chmod 664 xyz.png"). List this icon in /srv/scripts/collstuff/icons,
-	so the following script will know not to apply a default icon link.
-
-	13) Then go to the collstuff directory /srv/scripts/collstuff and run: collToDbase
-	to put the collection info in the database. This looks through the Admin folders in the top level
-	directories placed in /srv/deposits, for collection-numbered XML files. It parses through the content,
-	repairing encoding errors, puts the content into mysql InfoTrack.allColls, adds the expected canned link,
-	and asks you for each collection and each delivery system whether or not the collection is online.
-	The software will give you the expected URL to test; if it fails, and you know the content is in the system,
-	locate the correct URL and provide it to the software, which will ask for a correction.
-
-	14) Check the database entries... mysql, InfoTrack.allColls. For example:
-	mysql -u jlderidder -p
-	(enter password)
-	use InfoTrack;
-	select * from digColls where id_2009 like "u0003_0002121" and AnalogOrDigital like "D";
-	(when done, enter "quit;")
-	If not okay, make repairs from command line.
-
-	15) Verify that links and display are correct. they will show up online here: [[http://www.lib.ua.edu/digital/browse\|http://www.lib.ua.edu/digital/browse]].


Line 108:		Line 80:
	----------------------------------		----------------------------------

-	16)	+	12) Copy all the collection XML files out for addition to the database: (from /srv/deposits/content/ directory, run
		+	'cp /Admin/xml ../hold/.' This puts them in the /srv/deposits/hold/ directory for addition to the online collection list.
		+	You don't want the collections showing up online when the links go to nothing, so database upload of this needs to wait.
		+
		+	13) If the collection has an icon: make sure it is in the Admin folder, named for the collection.
		+	For example, u0003_0002121.icon.png for the collection above. (You will need to copy this
		+	to the CONTENTdm server (content.lib.ua.edu) and place it in the /usr/local/Content4/docs/cdm4/images
		+	directory, with 664 access rights "chmod 664 xyz.png"). List this icon in /srv/scripts/collstuff/icons,
		+	so the collToDbase script will know not to apply a default icon link.
		+
		+	14)
	a) open /srv/scripts/storing/relocating (script available here: [[Image:Relocating.txt]]) and set $test = 1.		a) open /srv/scripts/storing/relocating (script available here: [[Image:Relocating.txt]]) and set $test = 1.
	b) run "relocating > look". It will:		b) run "relocating > look". It will:
Line 117:		Line 99:
	c) Look at the output in the "look" file, and the manifests concatenated in RelocateManifests and parentMans.		c) Look at the output in the "look" file, and the manifests concatenated in RelocateManifests and parentMans.

-	17) Make whatever repairs are necessary for script to work. If there are any new categories (z0004?),	+	15) Make whatever repairs are necessary for script to work. If there are any new categories (z0004?),
	then add them, and create top level manifests in those category directories (copy from like directories		then add them, and create top level manifests in those category directories (copy from like directories
	and modify to fit).		and modify to fit).

-	18) Open /srv/scripts/storing/relocating and edit out the line $test = 1;	+	16) Open /srv/scripts/storing/relocating and edit out the line $test = 1;

-	19) Empty the file /srv/scripts/storing/RelocateManifests, and copy moveme to another filename.	+	17) Empty the file /srv/scripts/storing/RelocateManifests, and copy moveme to another filename.

-	20) WARNING! Before running /srv/scripts/storing/relocating, do a "chmod -R 755 /srv/archive". This will enable you to	+	18) WARNING! Before running /srv/scripts/storing/relocating, do a "chmod -R 755 /srv/archive". This will enable you to
	write to the directories as root		write to the directories as root

-	21) run "relocating > look" again, this time for real. It will take hours. Occasionally check the output in "look" and the	+	19) run "relocating > look" again, this time for real. It will take hours. Occasionally check the output in "look" and the
	archive directory for errors.		archive directory for errors.

Line 137:		Line 119:
	-------------------------------------------------		-------------------------------------------------

-	22) If the output tells you to modify or create manifests, do that next (if not done already)	+	20) If the output tells you to modify or create manifests, do that next (if not done already)
	in top level of /srv/www/htdocs/lockss directory		in top level of /srv/www/htdocs/lockss directory
	The script does NOT create the top 2 levels of manifests, as it does not have sufficient		The script does NOT create the top 2 levels of manifests, as it does not have sufficient
	information about new categories to fill this in.		information about new categories to fill this in.

-	23) check links at http://libcontent1.lib.ua.edu/lockss/Manifest.html -- drill down.	+	21) check links at http://libcontent1.lib.ua.edu/lockss/Manifest.html -- drill down.
	If you cannot access it, modify the .htaccess file in /srv/www/htdocs/lockss to allow your IP,		If you cannot access it, modify the .htaccess file in /srv/www/htdocs/lockss to allow your IP,
	and restart the apache web server: '/usr/sbin/apache2ctl restart'		and restart the apache web server: '/usr/sbin/apache2ctl restart'
	(be sure to change it back, and then restart the web server)		(be sure to change it back, and then restart the web server)

-	24) then run 'checkem > look' in /srv/scripts/storing (script available here: [[Image:Checkem.txt]])	+	22) then run 'checkem > look' in /srv/scripts/storing (script available here: [[Image:Checkem.txt]])
	-- it will use the "moveme" file to		-- it will use the "moveme" file to
	verify md5 sums of content that was moved,		verify md5 sums of content that was moved,
Line 155:		Line 137:
	nohup.out file for the content that normally would go into "look".		nohup.out file for the content that normally would go into "look".

-	25) run checkArchive (script available here; [[Image:CheckArchive.txt]]) and look at ArchiveERRORS; make repairs.	+	23) run checkArchive (script available here; [[Image:CheckArchive.txt]]) and look at ArchiveERRORS; make repairs.
	If needed, alter and run doubleCheck to make sure things were copied over; checkMans to check that what's linked in		If needed, alter and run doubleCheck to make sure things were copied over; checkMans to check that what's linked in
	the Manifest actually copied; addToMans to add content that was moved but not written to the Manifest. Problems occur		the Manifest actually copied; addToMans to add content that was moved but not written to the Manifest. Problems occur
Line 161:		Line 143:
	when it went down, meaning multiple images are corrupted or damaged.		when it went down, meaning multiple images are corrupted or damaged.

-	26) do a "chmod -R 555 /srv/archive" to protect the archive from unauthorized alteration until next time.	+	24) do a "chmod -R 555 /srv/archive" to protect the archive from unauthorized alteration until next time.

-	27) then go look in the /srv/deposits directory, make sure folders are clean, and delete them.	+	25) then go look in the /srv/deposits directory, make sure folders are clean, and delete them.
	I do this: ls /srv/deposits/content/*		I do this: ls /srv/deposits/content/*
	ls /srv/deposits/content//		ls /srv/deposits/content//
Line 173:		Line 155:
	(this will recursively delete all content in the directory)		(this will recursively delete all content in the directory)

-	28) go to /srv/scripts/surfacing and run copychange (script available here: [[Image:Copychange.txt]])	+
		+
		+	'''MAKE IT LIVE ONLINE'''
		+	-------------------------------------------------
		+
		+	26) go to /srv/scripts/surfacing and run copychange (script available here: [[Image:Copychange.txt]])
	to create derivatives from tiffs, wave files and copy transcripts over to the web directory for access;		to create derivatives from tiffs, wave files and copy transcripts over to the web directory for access;

-	29) then run /srv/scripts/surfacing/repair to remediate the doubles created from tiffs which include both thumbs and large images.	+	27) then run /srv/scripts/surfacing/repair to remediate the doubles created from tiffs which include both thumbs and large images.
		+
		+	28) cd /home/Jeremiah and locate the MODS for what was just moved, in the MODS directory. Run "relocate.pl" to send them live.
		+
		+
		+
		+	'''UPDATE DATABASE WITH COLLECTION INFO'''
		+	------------------------------------
		+
		+
		+	29) Then go to the collstuff directory /srv/scripts/collstuff and run: collToDbase
		+	to put the collection info in the database. This looks through /srv/deposits/hold directory
		+	for collection-numbered XML files. It parses through the content,
		+	repairing encoding errors, puts the content into mysql InfoTrack.allColls, adds the expected canned link,
		+	and asks you for each collection and each delivery system whether or not the collection is online.
		+	The software will give you the expected URL to test; if it fails, and you know the content is in the system,
		+	locate the correct URL and provide it to the software, which will ask for a correction.
		+
		+	30) Check the database entries... mysql, InfoTrack.allColls. For example:
		+	mysql -u jlderidder -p
		+	(enter password)
		+	use InfoTrack;
		+	select * from digColls where id_2009 like "u0003_0002121" and AnalogOrDigital like "D";
		+	(when done, enter "quit;")
		+	If not okay, make repairs from command line.
		+
		+	15) Verify that links and display are correct. they will show up online here: [[http://www.lib.ua.edu/digital/browse\|http://www.lib.ua.edu/digital/browse]].
		+
		+

-	30) cd /home/Jeremiah and locate the MODS for what was just moved, in the MODS directory. Run "relocate.pl" to send them live.

	31) run /srv/scripts/metadata/findMissing to find out what doesn't have metadata, what doesn't have derivatives -- troubleshoot.		31) run /srv/scripts/metadata/findMissing to find out what doesn't have metadata, what doesn't have derivatives -- troubleshoot.

Jlderidder: /* TO MOVE CONTENT TO STORAGE */

Jlderidder — Mon, 22 Feb 2010 15:13:12 GMT

TO MOVE CONTENT TO STORAGE

←Older revision		Revision as of 15:13, 22 February 2010
Line 125:		Line 125:
	19) Empty the file /srv/scripts/storing/RelocateManifests, and copy moveme to another filename.		19) Empty the file /srv/scripts/storing/RelocateManifests, and copy moveme to another filename.

-	20) WARNING! Before running /srv/scripts/storing/relocating, do a "chmod -R 755 /srv/archive". This will enable you to \	+	20) WARNING! Before running /srv/scripts/storing/relocating, do a "chmod -R 755 /srv/archive". This will enable you to
	write to the directories as root		write to the directories as root

Jlderidder at 15:12, 22 February 2010

Jlderidder — Mon, 22 Feb 2010 15:12:43 GMT

←Older revision		Revision as of 15:12, 22 February 2010
Line 155:		Line 155:
	nohup.out file for the content that normally would go into "look".		nohup.out file for the content that normally would go into "look".

-	25) run checkArchive (script available here; [[Image:CheckArchive.txt]]) and look at ArchiveERRORS; make repairs. If needed, alter and run doubleCheck to make sure things were copied over; checkMans to check that what's linked in the Manifest actually copied; addToMans to add content that was moved but not written to the Manifest. Problems occur especially when the server goes down during a transfer. Especially be aware that dozens of tiffs may have been copying when it went down, meaning multiple images are corrupted or damaged.	+	25) run checkArchive (script available here; [[Image:CheckArchive.txt]]) and look at ArchiveERRORS; make repairs.
		+	If needed, alter and run doubleCheck to make sure things were copied over; checkMans to check that what's linked in
		+	the Manifest actually copied; addToMans to add content that was moved but not written to the Manifest. Problems occur
		+	especially when the server goes down during a transfer. Especially be aware that dozens of tiffs may have been copying
		+	when it went down, meaning multiple images are corrupted or damaged.

	26) do a "chmod -R 555 /srv/archive" to protect the archive from unauthorized alteration until next time.		26) do a "chmod -R 555 /srv/archive" to protect the archive from unauthorized alteration until next time.
Line 169:		Line 173:
	(this will recursively delete all content in the directory)		(this will recursively delete all content in the directory)

-	28) go to /srv/scripts/surfacing and run copychange (script available here: [[Image:Copychange.txt]]) to create derivatives from tiffs, wave files and copy transcripts over to the web directory for access;	+	28) go to /srv/scripts/surfacing and run copychange (script available here: [[Image:Copychange.txt]])
		+	to create derivatives from tiffs, wave files and copy transcripts over to the web directory for access;

	29) then run /srv/scripts/surfacing/repair to remediate the doubles created from tiffs which include both thumbs and large images.		29) then run /srv/scripts/surfacing/repair to remediate the doubles created from tiffs which include both thumbs and large images.

Jlderidder at 15:05, 22 February 2010

Jlderidder — Mon, 22 Feb 2010 15:05:10 GMT

←Older revision		Revision as of 15:05, 22 February 2010
Line 3:		Line 3:
	We are storing content long-term on a Linux server.		We are storing content long-term on a Linux server.

-	We are delivering content on another Linux server via CONTENTdm, and Tonio Loewald of OLT has developed an alternative (Linux-based) delivery solution, [http://acumen.lib.ua./edu http://acumen.lib.ua./edu] which runs off a web-enabled version of our storage area.	+	We are delivering content on another Linux server via CONTENTdm, and Tonio Loewald of OLT has developed an alternative (Linux-based) delivery solution, Acumen [http://acumen.lib.ua./edu http://acumen.lib.ua./edu] which runs off a web-enabled version of our storage area.

	This is the current set of procedures for moving content from Share to long term storage, and linking them into manifests for LOCKSS.		This is the current set of procedures for moving content from Share to long term storage, and linking them into manifests for LOCKSS.