For Archiving - Revision history

Jlderidder at 15:19, 14 December 2012

Jlderidder — Fri, 14 Dec 2012 15:19:06 GMT

Scpstu32 at 16:52, 29 October 2010

Scpstu32 — Fri, 29 Oct 2010 16:52:36 GMT

←Older revision		Revision as of 16:52, 29 October 2010
Line 1:		Line 1:
-	Currently, content being uploaded for archival storage is in a specific organization (specified here: [[~~Share_Drive_Organization~~]]).	+	Currently, content being uploaded for archival storage is in a specific organization (specified here: [[Share_Drive_Protocols]]).

	Once this content is placed into the /srv/deposits/content/ directory on libcontent1 (a Linux server), we :		Once this content is placed into the /srv/deposits/content/ directory on libcontent1 (a Linux server), we :

Jlderidder at 21:51, 30 July 2010

Jlderidder — Fri, 30 Jul 2010 21:51:36 GMT

←Older revision		Revision as of 21:51, 30 July 2010
Line 17:		Line 17:

	Another handy script is archiveCheck [[Image:CheckArchive.txt]] which verifies that everything in each manifest is in the archive, and everything I intended to link into the manifest is indeed linked there properly.		Another handy script is archiveCheck [[Image:CheckArchive.txt]] which verifies that everything in each manifest is in the archive, and everything I intended to link into the manifest is indeed linked there properly.
		+
		+	When we digitize multiple tiny collections, we may combine the spreadsheets, for simplicity. Then, however, they must be split out by collection for archiving: [[Image:splitExcel.txt]]

Jlderidder at 15:27, 19 November 2009

Jlderidder — Thu, 19 Nov 2009 15:27:44 GMT

←Older revision		Revision as of 15:27, 19 November 2009
Line 3:		Line 3:
	Once this content is placed into the /srv/deposits/content/ directory on libcontent1 (a Linux server), we :		Once this content is placed into the /srv/deposits/content/ directory on libcontent1 (a Linux server), we :
	# verify that it copied correctly across the network,		# verify that it copied correctly across the network,
-	# check the content with quality control verification scripts,	+	# check the content with quality control verification scripts (such as [[Image:testIncoming.txt]])
	# upload the collection information file content into the database to provide access to the online collection via a web-side php script, and		# upload the collection information file content into the database to provide access to the online collection via a web-side php script, and
	# then we archive it.		# then we archive it.

Jlderidder at 15:20, 19 November 2009

Jlderidder — Thu, 19 Nov 2009 15:20:00 GMT

←Older revision		Revision as of 15:20, 19 November 2009
Line 16:		Line 16:
	Here's the checkem script: [[Image:Checkem.txt]]		Here's the checkem script: [[Image:Checkem.txt]]

-	Another handy script is archiveCheck [[Image:~~ArchiveCheck~~.txt]] which verifies that everything in each manifest is in the archive, and everything I intended to link into the manifest is indeed linked there properly.	+	Another handy script is archiveCheck [[Image:CheckArchive.txt]] which verifies that everything in each manifest is in the archive, and everything I intended to link into the manifest is indeed linked there properly.

Jlderidder at 15:14, 19 November 2009

Jlderidder — Thu, 19 Nov 2009 15:14:17 GMT

←Older revision		Revision as of 15:14, 19 November 2009
Line 1:		Line 1:
-	Currently, content being uploaded for archival storage is in a specific organization, specified here [[Share_Drive_Organization]].	+	Currently, content being uploaded for archival storage is in a specific organization (specified here: [[Share_Drive_Organization]]).

	Once this content is placed into the /srv/deposits/content/ directory on libcontent1 (a Linux server), we :		Once this content is placed into the /srv/deposits/content/ directory on libcontent1 (a Linux server), we :
Line 7:		Line 7:
	# then we archive it.		# then we archive it.

-	Archiving it means that we weed out extraneous files, re-order content (via copy) according to our storage organization, specified here [[Organization_of_completed_content_for_long-term_storage]], version the metadata, xml, or text files (linking into the manifest only the version; the updated one overwrites the unversioned copy in the directory) and either create a LOCKSS manifest for this content or alter existing ones to include this content.	+	Archiving it means that we weed out extraneous files, re-order content (via copy) according to our storage organization (specified here: [[Organization_of_completed_content_for_long-term_storage]]), version the metadata, xml, or text files (linking into the manifest only the version; the updated one overwrites the unversioned copy in the directory) and either create a LOCKSS manifest for this content or alter existing ones to include this content.

	This script (still being modified and updated to handle new problems) is here: [[Image:Relocating.txt]]		This script (still being modified and updated to handle new problems) is here: [[Image:Relocating.txt]]

Jlderidder: New page: Currently, content being uploaded for archival storage is in a specific organization, specified here Share_Drive_Organization. Once this content is placed into the /srv/deposits/conte...

Jlderidder — Thu, 19 Nov 2009 15:13:23 GMT

New page: Currently, content being uploaded for archival storage is in a specific organization, specified here Share_Drive_Organization. Once this content is placed into the /srv/deposits/conte...

New page

Currently, content being uploaded for archival storage is in a specific organization, specified here [[Share_Drive_Organization]].

Once this content is placed into the /srv/deposits/content/ directory on libcontent1 (a Linux server), we :
# verify that it copied correctly across the network,
# check the content with quality control verification scripts,
# upload the collection information file content into the database to provide access to the online collection via a web-side php script, and
# then we archive it.

Archiving it means that we weed out extraneous files, re-order content (via copy) according to our storage organization, specified here [[Organization_of_completed_content_for_long-term_storage]], version the metadata, xml, or text files (linking into the manifest only the version; the updated one overwrites the unversioned copy in the directory) and either create a LOCKSS manifest for this content or alter existing ones to include this content.

This script (still being modified and updated to handle new problems) is here: [[Image:Relocating.txt]]
By uncommenting out the $test = 1; line, you can run this as a test, which will not change any existing manifests or copy content. Instead, it will write all the manifest changes and creations into one huge file called RelocateManfests, and it will still write a list of what files it will copy where to the "moveme" file.

After running this script for real, run "checkem" which goes through the moveme file, does md5 comparison on the old file and the new one -- if they're the same, it will delete the old on in the deposits directory. If they're not the same, it will output an error and leave the original untouched.

Here's the checkem script: [[Image:Checkem.txt]]

Another handy script is archiveCheck [[Image:ArchiveCheck.txt]] which verifies that everything in each manifest is in the archive, and everything I intended to link into the manifest is indeed linked there properly.

←Older revision		Revision as of 15:19, 14 December 2012
Line 1:		Line 1:
	Currently, content being uploaded for archival storage is in a specific organization (specified here: [[Share_Drive_Protocols]]).		Currently, content being uploaded for archival storage is in a specific organization (specified here: [[Share_Drive_Protocols]]).

-	Once this content is placed into the /srv/deposits/content/ directory on ~~libcontent1~~ (a Linux server), we :	+	Once this content is placed into the /srv/deposits/content/ directory on libcontent (a Linux server), we :
	# verify that it copied correctly across the network,		# verify that it copied correctly across the network,
	# check the content with quality control verification scripts (such as [[Image:testIncoming.txt]])		# check the content with quality control verification scripts (such as [[Image:testIncoming.txt]])
Line 9:		Line 9:
	Archiving it means that we weed out extraneous files, re-order content (via copy) according to our storage organization (specified here: [[Organization_of_completed_content_for_long-term_storage]]), version the metadata, xml, or text files (linking into the manifest only the version; the updated one overwrites the unversioned copy in the directory) and either create a LOCKSS manifest for this content or alter existing ones to include this content.		Archiving it means that we weed out extraneous files, re-order content (via copy) according to our storage organization (specified here: [[Organization_of_completed_content_for_long-term_storage]]), version the metadata, xml, or text files (linking into the manifest only the version; the updated one overwrites the unversioned copy in the directory) and either create a LOCKSS manifest for this content or alter existing ones to include this content.

-	~~This~~ script (~~still being modified and updated to handle new problems~~) is ~~here: [[Image:Relocating~~.~~txt]]~~	+	We have different versions of the archiving script (Relocating) for different materials (on libcontent). Most content is archived using the relocating script in /srv/storing/; there's another for ETDs in the bornDigital subdirectory, one for MODS in the MODS subdirectory, one for EADs in the eads subdirectory, one for tags and transcriptions in the crowdsource subdirectory, and so forth. In each,
-	By uncommenting out the $test = 1; line, you can run this as a test, which will not change any existing manifests or copy content. Instead, it will write all the manifest changes and creations into one huge file called ~~RelocateManfests~~, and it will still write a list of what files it will copy where to the "moveme" file.	+	by uncommenting out the $test = 1; line, you can run this as a test, which will not change any existing manifests or copy content. Instead, it will write all the manifest changes and creations into one huge file called RelocateManifests, and it will still write a list of what files it will copy where to the "moveme" file.

	After running this script for real, run "checkem" which goes through the moveme file, does md5 comparison on the old file and the new one -- if they're the same, it will delete the old on in the deposits directory. If they're not the same, it will output an error and leave the original untouched.		After running this script for real, run "checkem" which goes through the moveme file, does md5 comparison on the old file and the new one -- if they're the same, it will delete the old on in the deposits directory. If they're not the same, it will output an error and leave the original untouched.
Line 16:		Line 16:
	Here's the checkem script: [[Image:Checkem.txt]]		Here's the checkem script: [[Image:Checkem.txt]]

-	~~Another handy script is archiveCheck [[Image:CheckArchive.txt]]~~ which verifies that everything in each manifest is in the archive, and everything I intended to link into the manifest is indeed linked there properly.	+	We also have verification scripts, such as "checkArchive" which verifies that everything in each manifest is in the archive, and everything I intended to link into the manifest is indeed linked there properly.

	When we digitize multiple tiny collections, we may combine the spreadsheets, for simplicity. Then, however, they must be split out by collection for archiving: [[Image:splitExcel.txt]]		When we digitize multiple tiny collections, we may combine the spreadsheets, for simplicity. Then, however, they must be split out by collection for archiving: [[Image:splitExcel.txt]]