Batches - Revision history

Kgmatheny at 22:01, 23 April 2013

Kgmatheny — Tue, 23 Apr 2013 22:01:26 GMT

←Older revision		Revision as of 22:01, 23 April 2013
Line 14:		Line 14:

	* As you track your captures in the metadata spreadsheet, make sure to note in the Batch column that the item is part of the current batch.		* As you track your captures in the metadata spreadsheet, make sure to note in the Batch column that the item is part of the current batch.
-	* Note which items have been skipped by labeling them 'skipped' in the Batch column	+	* Note which items have been skipped by labeling them 'skipped' in the Batch column (see [[Skipped Items]])
	* When you are ready to upload a batch, sort the spreadsheet by the Batch column; this will bring all of the lines of metadata for that batch together		* When you are ready to upload a batch, sort the spreadsheet by the Batch column; this will bring all of the lines of metadata for that batch together

Kgmatheny at 21:50, 23 April 2013

Kgmatheny — Tue, 23 Apr 2013 21:50:22 GMT

←Older revision		Revision as of 21:50, 23 April 2013
Line 13:		Line 13:
	===Isolating a batch within the metadata spreadsheet===		===Isolating a batch within the metadata spreadsheet===

-	* As you ~~tracking~~ your captures in the metadata spreadsheet, make sure to note in the Batch column that the item is part of the current batch.	+	* As you track your captures in the metadata spreadsheet, make sure to note in the Batch column that the item is part of the current batch.
	* Note which items have been skipped by labeling them 'skipped' in the Batch column		* Note which items have been skipped by labeling them 'skipped' in the Batch column
	* When you are ready to upload a batch, sort the spreadsheet by the Batch column; this will bring all of the lines of metadata for that batch together		* When you are ready to upload a batch, sort the spreadsheet by the Batch column; this will bring all of the lines of metadata for that batch together

Kgmatheny at 16:32, 11 February 2013

Kgmatheny — Mon, 11 Feb 2013 16:32:35 GMT

←Older revision		Revision as of 16:32, 11 February 2013
Line 33:		Line 33:
	===Batch text export===		===Batch text export===
	* Used by: Digital Services		* Used by: Digital Services
-	* Used for: [[Making MODS \| MODS creation]])	+	* Used for: [[Making MODS \| MODS creation]]

	To create a batch text export...		To create a batch text export...
Line 44:		Line 44:

	===Batch log file===		===Batch log file===
-	* ~~Used by: Digital Services~~	+	* Created for: archiving tracking data collected during the capture process
-	* Used for: ~~recording~~ tracking data collected during the capture process	+

	To create a batch log file...		To create a batch log file...
Line 54:		Line 53:
	** Example: u0003_0000193.2.m01.log.txt		** Example: u0003_0000193.2.m01.log.txt
	* Save this file to the collection's Admin folder (S:\Digital Projects\Digital_Coll_Complete\[collectionnumber]\Admin)		* Save this file to the collection's Admin folder (S:\Digital Projects\Digital_Coll_Complete\[collectionnumber]\Admin)
		+
		+	===Batch OCR list===
		+	* Used by: Upload Script
		+	* Used for: telling MakeJpegs which files to process through Tesseract
		+
		+	To create a batch OCR list, see [[OCR List \| these instructions]]

	===Note on naming files made from older spreadsheets===		===Note on naming files made from older spreadsheets===

Kgmatheny at 16:24, 11 February 2013

Kgmatheny — Mon, 11 Feb 2013 16:24:52 GMT

←Older revision		Revision as of 16:24, 11 February 2013
Line 29:		Line 29:
	** Naming format: ''collectionnumber.batchnumber.m0number.xlsx''		** Naming format: ''collectionnumber.batchnumber.m0number.xlsx''
	** Example: u0003_0000193.2.m01.xlsx		** Example: u0003_0000193.2.m01.xlsx
-	* Save this file to Administrative\Pipeline\collectionInfo\forMDlib\needsRemediation	+	* Save this file to the Metadata folder temporarily; just prior to upload, it will be moved to Administrative\Pipeline\collectionInfo\forMDlib\needsRemediation

	===Batch text export===		===Batch text export===

Kgmatheny at 22:15, 8 January 2013

Kgmatheny — Tue, 08 Jan 2013 22:15:23 GMT

←Older revision		Revision as of 22:15, 8 January 2013
Line 1:		Line 1:
	A batch is simply a portion of a collection that goes through the upload process bundled together. This page contains or links to information related to the batching process and what differentiates it from a whole-collection upload.		A batch is simply a portion of a collection that goes through the upload process bundled together. This page contains or links to information related to the batching process and what differentiates it from a whole-collection upload.

-	==~~When to make~~ a ~~batch~~==	+	==Creating a Batch==

-	A new batch is created when you decide to upload content from an ongoing collection. This almost always happens when you've broken off and numbered a new [[Scans folder]], but it is possible to upload several Scans folders at once. Whatever the case, the content from a particular collection which is uploaded together is a batch~~, and this content should be ''no more than 300 items or 4000 captures.''~~	+	===What is a batch?===

-	~~==Documentation for~~ a batch==	+	A new batch is created when you decide to upload content from an ongoing collection. This almost always happens when you've broken off and numbered a new [[Scans folder]], but it is possible to upload several Scans folders at once. Whatever the case, the content from a particular collection which is uploaded together is a batch, and this content should be ''no more than 300 items or 4000 captures.''

	===Dual role of the metadata spreadsheet===		===Dual role of the metadata spreadsheet===

-	As of January 2013, tracking data is incorporated into the metadata spreadsheet during the capture process. For the purposes of this explanation, metadata refers to the data which goes online, while tracking data is the internal, administrative ~~data~~ we capture for our unit. Those columns are found at the end of the spreadsheet ~~(after the Staff Notes~~ and ~~Batch columns)~~.	+	As of January 2013, tracking data is incorporated into the metadata spreadsheet during the capture process. For the purposes of this explanation, ''metadata'' refers to the data which goes online, while ''tracking data'' is the internal, administrative information we capture for our unit. Those columns are found at the end of the spreadsheet, and they should be removed before handing off the spreadsheet to the metadata unit.

-	===Isolating ~~the~~ batch within the metadata spreadsheet===	+	===Isolating a batch within the metadata spreadsheet===

	* As you tracking your captures in the metadata spreadsheet, make sure to note in the Batch column that the item is part of the current batch.		* As you tracking your captures in the metadata spreadsheet, make sure to note in the Batch column that the item is part of the current batch.
Line 17:		Line 17:
	* When you are ready to upload a batch, sort the spreadsheet by the Batch column; this will bring all of the lines of metadata for that batch together		* When you are ready to upload a batch, sort the spreadsheet by the Batch column; this will bring all of the lines of metadata for that batch together

-	==~~=Creating batch documentation=~~==	+	==Batch Documentation==

-	* ''Batch metadata spreadsheet''	+	===Batch Spreadsheet===
-	** Copy the ~~batch metadata~~ to ~~another sheet~~, ~~and also~~ the ~~spreadsheet headers~~	+	* Used by: the Metadata Unit
-	*** Do not copy the tracking data	+	* Used for: remediating metadata that has already gone online
-	** Save ~~that sheet~~ as a new, separate file~~, so that you don't alter the original collection metadata~~	+
		+	To create a batch spreadsheet...
		+	* Select just the rows that belong to the batch, plus the header row
		+	* Deselect the tracking data, leaving the metadata
		+	* Save the selection as a new, separate file: a Microsoft Excel document (.xlsx)
	** Naming format: ''collectionnumber.batchnumber.m0number.xlsx''		** Naming format: ''collectionnumber.batchnumber.m0number.xlsx''
-	*** Example: u0003_0000193.2.m01.xlsx	+	** Example: u0003_0000193.2.m01.xlsx
		+	* Save this file to Administrative\Pipeline\collectionInfo\forMDlib\needsRemediation
		+
		+	===Batch text export===
		+	* Used by: Digital Services
		+	* Used for: [[Making MODS \| MODS creation]])

-	* ''Batch text export~~'' for [[Making MODS \| MODS creation]]~~	+	To create a batch text export...
-	** Export as a tab delimited file all the ~~columns of metadata~~, ~~but NOT~~ the tracking data	+	* Select just the rows that belong to the batch, plus the header row
-	** Save to the ~~collection's Metadata folder~~	+	* Deselect the tracking data, leaving the metadata
		+	* Save the selection as a new, separate file: a tab-delimited text document (.txt)
	** Naming format: ''collectionnumber.batchnumber.m0number.txt''		** Naming format: ''collectionnumber.batchnumber.m0number.txt''
-	*** Example: u0003_0000193.2.m01.txt	+	** Example: u0003_0000193.2.m01.txt
		+	* Save this file to the collection's Metadata folder (S:\Digital Projects\Digital_Coll_Complete\[collectionnumber]\Metadata)

-	* ''Batch log file''	+	===Batch log file===
-	** ~~Export as~~ a ~~tab delimited~~ file ~~all~~ the ~~columns of tracking data (but NOT~~ the ~~metadata)~~, plus the ~~Item Identifier column~~	+	* Used by: Digital Services
-	** Save to the ~~collection's Admin folder~~	+	* Used for: recording tracking data collected during the capture process
		+
		+	To create a batch log file...
		+	* Select just the rows that belong to the batch, plus the header row
		+	* Deselect the metadata, leaving the tracking data
		+	* Save the selection as a new, separate file: a tab-delimited text document (.txt)
	** Naming format: ''collectionnumber.batchnumber.m0number.log.txt''		** Naming format: ''collectionnumber.batchnumber.m0number.log.txt''
-	*** Example: u0003_0000193.2.m01.log.txt	+	** Example: u0003_0000193.2.m01.log.txt
		+	* Save this file to the collection's Admin folder (S:\Digital Projects\Digital_Coll_Complete\[collectionnumber]\Admin)

-	===~~Older spreadsheet?~~===	+	===Note on naming files made from older spreadsheets===

	* For older spreadsheets that don't have m0numbers (m01, m02, m03, etc) simply leave that out		* For older spreadsheets that don't have m0numbers (m01, m02, m03, etc) simply leave that out
Line 44:		Line 61:
	** u0003_0001577.30.xlsx		** u0003_0001577.30.xlsx
	** u0001_2007010.4.log.txt		** u0001_2007010.4.log.txt
-

	==Checklist: What's different about workflow for a batch==		==Checklist: What's different about workflow for a batch==
	* Preparation for Upload		* Preparation for Upload
-	** ~~Separate~~ batch spreadsheet in Administrative\Pipeline\collectionInfo\forMDlib\needsRemediation	+	** Create a collection folder in Digital_Coll_Complete, and
-	** ~~Text file~~ metadata in S:\Digital Projects\Digital_Coll_Complete\[collectionnumber]\Metadata, with MODS created from that	+	*** '''Move''' the Scans folder to it
-	** ~~Text file~~ log in S:\Digital Projects\Digital_Coll_Complete\[collectionnumber]\Admin	+	*** '''Create''' Admin and Metadata folders
		+	*** '''Copy''' the appropriate documents from in_progress to Admin and Metadata
		+	** Create batch spreadsheet in Administrative\Pipeline\collectionInfo\forMDlib\needsRemediation
		+	** Create batch metadata text file in S:\Digital Projects\Digital_Coll_Complete\[collectionnumber]\Metadata, with MODS created from that
		+	** Create batch log file in S:\Digital Projects\Digital_Coll_Complete\[collectionnumber]\Admin
	* Upload		* Upload
	** The script-running process should be identical		** The script-running process should be identical
-	** With a batch (unless it's the first), the collection info xml is already online, so you don't have to wait ~~24 hours~~ for Acumen to index the items before you can run moveContent	+	** With a batch (unless it's the first), the collection info xml is already online, so you don't have to wait for Acumen to index the items before you can run moveContent
	* Storage		* Storage
	** You will not need the collection info xml file when you move the collection from the share drive to storage. In fact, the moveContent script will tell you to delete it...unless it is a new, improved version		** You will not need the collection info xml file when you move the collection from the share drive to storage. In fact, the moveContent script will tell you to delete it...unless it is a new, improved version
	* Documentation		* Documentation
	** Unless you're working on the last batch of a collection, do not move the collection from in progress to completed on the Selection spreadsheet		** Unless you're working on the last batch of a collection, do not move the collection from in progress to completed on the Selection spreadsheet

Kgmatheny: reworked instructions based on new work flow, incorporating tracking into SS

Kgmatheny — Mon, 07 Jan 2013 23:14:37 GMT

reworked instructions based on new work flow, incorporating tracking into SS

←Older revision		Revision as of 23:14, 7 January 2013
Line 5:		Line 5:
	A new batch is created when you decide to upload content from an ongoing collection. This almost always happens when you've broken off and numbered a new [[Scans folder]], but it is possible to upload several Scans folders at once. Whatever the case, the content from a particular collection which is uploaded together is a batch, and this content should be ''no more than 300 items or 4000 captures.''		A new batch is created when you decide to upload content from an ongoing collection. This almost always happens when you've broken off and numbered a new [[Scans folder]], but it is possible to upload several Scans folders at once. Whatever the case, the content from a particular collection which is uploaded together is a batch, and this content should be ''no more than 300 items or 4000 captures.''

-	==~~Documenting and creating metadata~~ for a batch==	+	==Documentation for a batch==

-	===~~How do you determine which batch number it is?~~===	+	===Dual role of the metadata spreadsheet===

-	In the ~~collection's~~ metadata spreadsheet, the ~~last column contains the batch number~~.	+	As of January 2013, tracking data is incorporated into the metadata spreadsheet during the capture process. For the purposes of this explanation, metadata refers to the data which goes online, while tracking data is the internal, administrative data we capture for our unit. Those columns are found at the end of the spreadsheet (after the Staff Notes and Batch columns).
-	~~# Check that column for~~ the ~~largest number --~~ which ~~should be~~ the ~~last/previous batch~~	+
-	~~# Checking~~ for ~~the largest batch number on the storage sever may also help with secondary verification~~	+
-	~~# Assign your batch the next number~~.	+
-	~~# Fill in that number for all~~ the ~~lines~~ of ~~metadata in your batch~~.	+

-	===~~How do you create~~ batch metadata?===	+	===Isolating the batch within the metadata spreadsheet===

-	~~While~~ the ~~whole~~ metadata spreadsheet ~~should stay with the collection in progress~~, the ~~collection batch~~ that ~~has~~ been ~~moved~~ to ~~Completed needs its own~~ batch metadata.	+	* As you tracking your captures in the metadata spreadsheet, make sure to note in the Batch column that the item is part of the current batch.
		+	* Note which items have been skipped by labeling them 'skipped' in the Batch column
		+	* When you are ready to upload a batch, sort the spreadsheet by the Batch column; this will bring all of the lines of metadata for that batch together

-	~~Batch metadata includes~~	+	===Creating batch documentation===
-	* ''Batch spreadsheet''	+
-	** ~~This is created from~~ the ~~collection~~ metadata spreadsheet	+	* ''Batch metadata spreadsheet''
-	** ~~Copy~~ the ~~following to a new sheet: the header row and the rows of metadata that correspond to your batch~~	+	** Copy the batch metadata to another sheet, and also the spreadsheet headers
		+	*** Do not copy the tracking data
	** Save that sheet as a new, separate file, so that you don't alter the original collection metadata		** Save that sheet as a new, separate file, so that you don't alter the original collection metadata
	** Naming format: ''collectionnumber.batchnumber.m0number.xlsx''		** Naming format: ''collectionnumber.batchnumber.m0number.xlsx''
	*** Example: u0003_0000193.2.m01.xlsx		*** Example: u0003_0000193.2.m01.xlsx
-	*** For older spreadsheets that don't have m0numbers (m01, m02, m03, etc) simply leave that out: u0003_0000193.2.xlsx	+
-	* ''Batch text ~~file~~'' for MODS creation	+	* ''Batch text export'' for [[Making MODS \| MODS creation]]
-	** ~~Text~~ file ~~creation explained in~~ the ~~first part~~ of ~~[[Making MODS]] -- make sure to process your batch spreadsheet~~, ~~not~~ the ~~whole~~ collection ~~metadata spreadsheet~~	+	** Export as a tab delimited file all the columns of metadata, but NOT the tracking data
		+	** Save to the collection's Metadata folder
	** Naming format: ''collectionnumber.batchnumber.m0number.txt''		** Naming format: ''collectionnumber.batchnumber.m0number.txt''
	*** Example: u0003_0000193.2.m01.txt		*** Example: u0003_0000193.2.m01.txt
-	*** ~~Same rule applies for text files made from~~ metadata spreadsheets ~~without~~ m0numbers.	+
		+	* ''Batch log file''
		+	** Export as a tab delimited file all the columns of tracking data (but NOT the metadata), plus the Item Identifier column
		+	** Save to the collection's Admin folder
		+	** Naming format: ''collectionnumber.batchnumber.m0number.log.txt''
		+	*** Example: u0003_0000193.2.m01.log.txt
		+
		+	===Older spreadsheet?===
		+
		+	* For older spreadsheets that don't have m0numbers (m01, m02, m03, etc) simply leave that out
		+	* Examples
		+	** u0003_0001577.30.xlsx
		+	** u0001_2007010.4.log.txt
		+

	==Checklist: What's different about workflow for a batch==		==Checklist: What's different about workflow for a batch==
	* Preparation for Upload		* Preparation for Upload
	** Separate batch spreadsheet in Administrative\Pipeline\collectionInfo\forMDlib\needsRemediation		** Separate batch spreadsheet in Administrative\Pipeline\collectionInfo\forMDlib\needsRemediation
-	** Text file ~~version~~ in S:\Digital Projects\Digital_Coll_Complete\[collectionnumber]\Metadata, with MODS created from that	+	** Text file metadata in S:\Digital Projects\Digital_Coll_Complete\[collectionnumber]\Metadata, with MODS created from that
		+	** Text file log in S:\Digital Projects\Digital_Coll_Complete\[collectionnumber]\Admin
	* Upload		* Upload
	** The script-running process should be identical		** The script-running process should be identical

Jjcolonnaromano at 17:42, 29 October 2012

Jjcolonnaromano — Mon, 29 Oct 2012 17:42:14 GMT

←Older revision		Revision as of 17:42, 29 October 2012
Line 11:		Line 11:
	In the collection's metadata spreadsheet, the last column contains the batch number.		In the collection's metadata spreadsheet, the last column contains the batch number.
	# Check that column for the largest number -- which should be the last/previous batch		# Check that column for the largest number -- which should be the last/previous batch
-	# Checking for the largest batch number on the storage sever ~~in the collections metadata folder can be used for~~ secondary verification	+	# Checking for the largest batch number on the storage sever may also help with secondary verification
	# Assign your batch the next number.		# Assign your batch the next number.
	# Fill in that number for all the lines of metadata in your batch.		# Fill in that number for all the lines of metadata in your batch.

Jjcolonnaromano at 17:40, 29 October 2012

Jjcolonnaromano — Mon, 29 Oct 2012 17:40:45 GMT

Kgmatheny at 14:52, 29 October 2012

Kgmatheny — Mon, 29 Oct 2012 14:52:46 GMT

←Older revision		Revision as of 14:52, 29 October 2012
Line 16:		Line 16:
	===How do you create batch metadata?===		===How do you create batch metadata?===

-	While the whole metadata spreadsheet should stay with the collection in progress, the collection batch that has been moved to Completed needs its own batch metadata. ~~This~~ includes	+	While the whole metadata spreadsheet should stay with the collection in progress, the collection batch that has been moved to Completed needs its own batch metadata.
		+
		+	Batch metadata includes
	* ''Batch spreadsheet''		* ''Batch spreadsheet''
	** This is created from the collection metadata spreadsheet		** This is created from the collection metadata spreadsheet
Line 25:		Line 27:
	*** For older spreadsheets that don't have m0numbers (m01, m02, m03, etc) simply leave that out: u0003_0000193.2.xlsx		*** For older spreadsheets that don't have m0numbers (m01, m02, m03, etc) simply leave that out: u0003_0000193.2.xlsx
	* ''Batch text file'' for MODS creation		* ''Batch text file'' for MODS creation
-	** Text file creation explained in the first part of [[Making MODS]] -- use the batch spreadsheet	+	** Text file creation explained in the first part of [[Making MODS]] -- make sure to use the batch spreadsheet
	** Naming format: ''collectionnumber.batchnumber.m0number.txt''		** Naming format: ''collectionnumber.batchnumber.m0number.txt''
	*** Example: u0003_0000193.2.m01.txt		*** Example: u0003_0000193.2.m01.txt

Kgmatheny at 15:19, 26 October 2012

Kgmatheny — Fri, 26 Oct 2012 15:19:16 GMT

←Older revision		Revision as of 15:19, 26 October 2012
Line 31:		Line 31:

	==Checklist: What's different about workflow for a batch==		==Checklist: What's different about workflow for a batch==
-
-	* Capture, Optimization, and Quality Control
-	** No change
	* Preparation for Upload		* Preparation for Upload
-	** Separate batch spreadsheet in Administrative/Pipeline/forMDlib~~/needsremediation folder~~	+	** Separate batch spreadsheet in Administrative\Pipeline\collectionInfo\forMDlib\needsRemediation
-	** Text file version in ~~Completed/~~[~~collection number~~]/Metadata ~~folder~~, with MODS created from that	+	** Text file version in S:\Digital Projects\Digital_Coll_Complete\[collectionnumber]\Metadata, with MODS created from that
	* Upload		* Upload
	** The script-running process should be identical		** The script-running process should be identical

←Older revision		Revision as of 17:40, 29 October 2012
Line 1:		Line 1:
-	A batch is simply a portion of a collection that goes through the upload process together. This page contains or links to information related to the batching process and what differentiates it from a whole-collection upload.	+	A batch is simply a portion of a collection that goes through the upload process bundled together. This page contains or links to information related to the batching process and what differentiates it from a whole-collection upload.

	==When to make a batch==		==When to make a batch==
Line 11:		Line 11:
	In the collection's metadata spreadsheet, the last column contains the batch number.		In the collection's metadata spreadsheet, the last column contains the batch number.
	# Check that column for the largest number -- which should be the last/previous batch		# Check that column for the largest number -- which should be the last/previous batch
		+	# Checking for the largest batch number on the storage sever in the collections metadata folder can be used for secondary verification
	# Assign your batch the next number.		# Assign your batch the next number.
	# Fill in that number for all the lines of metadata in your batch.		# Fill in that number for all the lines of metadata in your batch.
Line 27:		Line 28:
	*** For older spreadsheets that don't have m0numbers (m01, m02, m03, etc) simply leave that out: u0003_0000193.2.xlsx		*** For older spreadsheets that don't have m0numbers (m01, m02, m03, etc) simply leave that out: u0003_0000193.2.xlsx
	* ''Batch text file'' for MODS creation		* ''Batch text file'' for MODS creation
-	** Text file creation explained in the first part of [[Making MODS]] -- make sure to ~~use the~~ batch spreadsheet	+	** Text file creation explained in the first part of [[Making MODS]] -- make sure to process your batch spreadsheet, not the whole collection metadata spreadsheet
	** Naming format: ''collectionnumber.batchnumber.m0number.txt''		** Naming format: ''collectionnumber.batchnumber.m0number.txt''
	*** Example: u0003_0000193.2.m01.txt		*** Example: u0003_0000193.2.m01.txt