Diacritics - Revision history

Kgmatheny at 18:03, 19 February 2013

Kgmatheny — Tue, 19 Feb 2013 18:03:56 GMT

←Older revision		Revision as of 18:03, 19 February 2013
Line 23:		Line 23:

	# Open the Excel spreadsheet for the collection (or collection batch)		# Open the Excel spreadsheet for the collection (or collection batch)
-
	# Use Excel's built in character map to replace all found problems in the Excel file		# Use Excel's built in character map to replace all found problems in the Excel file
	## To access the character map, follow this path: Insert tab - Symbols group - Symbol		## To access the character map, follow this path: Insert tab - Symbols group - Symbol
	## Select the character you need to replace and make sure "Unicode Hex" is selected		## Select the character you need to replace and make sure "Unicode Hex" is selected
	## Insert the character		## Insert the character
-
	# Close the text export and delete it (DO NOT CLOSE/DELETE THE SPREADSHEET)		# Close the text export and delete it (DO NOT CLOSE/DELETE THE SPREADSHEET)
-
	# Export the metadata as a new Unicode text file		# Export the metadata as a new Unicode text file
-		+	# Continue on (or go back) to [[Making MODS]]
-	# Continue on to [[Making MODS]]	+


	[http://intranet.lib.ua.edu/wiki/digcoll/images/e/e6/French_encoding_map.xlsx This Excel file] may be useful in getting one started in searching/replacing diacritics for French language metadata.		[http://intranet.lib.ua.edu/wiki/digcoll/images/e/e6/French_encoding_map.xlsx This Excel file] may be useful in getting one started in searching/replacing diacritics for French language metadata.

Kgmatheny at 18:01, 19 February 2013

Kgmatheny — Tue, 19 Feb 2013 18:01:52 GMT

←Older revision		Revision as of 18:01, 19 February 2013
Line 14:		Line 14:

	# Open the file in NotePad ++ and choose "Encode in UTF-8".		# Open the file in NotePad ++ and choose "Encode in UTF-8".
-
	# Use the TextFX plugin and choose: TextFX Characters > zap all non-printable characters to #		# Use the TextFX plugin and choose: TextFX Characters > zap all non-printable characters to #
-
	# Do a search for all instances of "#"		# Do a search for all instances of "#"
-		+	# If this character shows up, it represents a diacritic -- keep this text export open as a guide for the next step, Repairing Diacritics
-	# If this character shows up, it represents a diacritic -- ~~if you have diacritics,~~ keep this text export open as a guide for the next step	+	# If this character does not show up, continue on to [[Making MODS]]

Kgmatheny at 17:50, 19 February 2013

Kgmatheny — Tue, 19 Feb 2013 17:50:12 GMT

←Older revision		Revision as of 17:50, 19 February 2013
Line 1:		Line 1:
	As metadata spreadsheets exchange hands and are often even created from diverse sources, issues arise regarding diacritics. These characters often do no translate from encoding to encoding, creating poor results in the resultant [[MODS_Mockup\|MODS metadata files]].		As metadata spreadsheets exchange hands and are often even created from diverse sources, issues arise regarding diacritics. These characters often do no translate from encoding to encoding, creating poor results in the resultant [[MODS_Mockup\|MODS metadata files]].

		+	Encoding problems will appear as black rectangular three letter "blocks" in Notepad++ or diamond shaped question marks in Archivist Utility.

-	Based on our experience with French language diacritics, the following process allowed MODS to be created without the numerous encoding problems that we initially encountered with the ~~collection: u0002_0000006 (~~French Revolutionary Pamphlets).	+	Based on our experience with French language diacritics, the following process allowed MODS to be created without the numerous encoding problems that we initially encountered with the French Revolutionary Pamphlets (u0002_0000006).

	To do these steps, you will need a Windows computer with: Microsoft Excel and [http://notepad-plus.sourceforge.net/ NotePad ++].		To do these steps, you will need a Windows computer with: Microsoft Excel and [http://notepad-plus.sourceforge.net/ NotePad ++].
Line 8:		Line 9:
	To get plugins for NotePad++ refer to the application documentation with regard to the Plugin Manager.		To get plugins for NotePad++ refer to the application documentation with regard to the Plugin Manager.

-	~~1. Export the metadata from Excel as a tab delimited text file (use the Unicode option). If the Excel file is in the legacy .xls format, first convert it to .xlsx prior to export.~~	+	==Finding Diacritics==

		+	This method uses a tab delimited text export of the metadata spreadsheet. Note: If the Excel file is in the legacy .xls format, first convert it to .xlsx prior to export.

-	2. Open the file in NotePad ++ and choose "Encode in UTF-8".	+	# Open the file in NotePad ++ and choose "Encode in UTF-8".

		+	# Use the TextFX plugin and choose: TextFX Characters > zap all non-printable characters to #

-	~~3. If you dont know where the problem characters are, Use the TextFX plugin and choose: TextFX Characters>zap~~ all ~~non-printable characters to~~ #	+	# Do a search for all instances of "#"

		+	# If this character shows up, it represents a diacritic -- if you have diacritics, keep this text export open as a guide for the next step

-	4. Do a search for all instances of "#" and using the text file in notepad++ as a guide, replace all found problems in the Excel (.xlsx) file. Use Excel's built in character map. To access the character map window from within excel goto the Insert tab - Symbols group - Symbol, select the character you need to replace and choose "Unicode Hex" as the encoding while making changes.

-	~~[http://intranet.lib.ua.edu/wiki/digcoll/images/e/e6/French_encoding_map.xlsx This Excel file] may be useful in getting one started in searching/replacing diacritics for French language metadata.~~	+	==Repairing Diacritics==

		+	# Open the Excel spreadsheet for the collection (or collection batch)

-	~~5. From~~ Excel, ~~export~~ the ~~metadata as a~~ Unicode ~~text file.~~	+	# Use Excel's built in character map to replace all found problems in the Excel file
		+	## To access the character map, follow this path: Insert tab - Symbols group - Symbol
		+	## Select the character you need to replace and make sure "Unicode Hex" is selected
		+	## Insert the character

		+	# Close the text export and delete it (DO NOT CLOSE/DELETE THE SPREADSHEET)

-	~~6. Open~~ the Unicode ~~export in Notepad ++ and choose "Encode in UTF-8 without BOM". Repeat Step 3 and 4 until all problems are taken care of.~~	+	# Export the metadata as a new Unicode text file

		+	# Continue on to [[Making MODS]]

-	7. If all is well, save the text file with NotePad ++ (it will be a UTF-8 file).
-	This UTF-8 file is now the file from which to create MODS with [http://acumen.lib.ua.edu/project/?f=Archivist%20Utility.txt Archivist Utility]. This is also the text version of the metadata that will go into long term Storage.

-	8. ~~Encoding problems can~~ be ~~discovered along the way by looking for black rectangular three letter "blocks"~~ in ~~Notepad++ or diamond shaped question marks~~ in ~~Archivist Utility~~.	+	[http://intranet.lib.ua.edu/wiki/digcoll/images/e/e6/French_encoding_map.xlsx This Excel file] may be useful in getting one started in searching/replacing diacritics for French language metadata.

Jjcolonnaromano at 21:04, 25 January 2013

Jjcolonnaromano — Fri, 25 Jan 2013 21:04:40 GMT

←Older revision		Revision as of 21:04, 25 January 2013
Line 11:		Line 11:


-	2. Open in the file in NotePad ++ and choose "Encode in UTF-8".	+	2. Open the file in NotePad ++ and choose "Encode in UTF-8".


Line 17:		Line 17:


-	4. Do a search for all instances of "#" and replace all found problems in the Excel (.xlsx) file. Use Excel's built in character map. To access the character map window from within excel goto the Insert tab - Symbols group - Symbol, select the character you need to replace and choose "Unicode Hex" as the encoding while making changes.	+	4. Do a search for all instances of "#" and using the text file in notepad++ as a guide, replace all found problems in the Excel (.xlsx) file. Use Excel's built in character map. To access the character map window from within excel goto the Insert tab - Symbols group - Symbol, select the character you need to replace and choose "Unicode Hex" as the encoding while making changes.

	[http://intranet.lib.ua.edu/wiki/digcoll/images/e/e6/French_encoding_map.xlsx This Excel file] may be useful in getting one started in searching/replacing diacritics for French language metadata.		[http://intranet.lib.ua.edu/wiki/digcoll/images/e/e6/French_encoding_map.xlsx This Excel file] may be useful in getting one started in searching/replacing diacritics for French language metadata.

Jjcolonnaromano at 21:01, 25 January 2013

Jjcolonnaromano — Fri, 25 Jan 2013 21:01:41 GMT

←Older revision		Revision as of 21:01, 25 January 2013
Line 14:		Line 14:


-	3. Use the TextFX plugin and choose: TextFX Characters>zap all non-printable characters to #	+	3. If you dont know where the problem characters are, Use the TextFX plugin and choose: TextFX Characters>zap all non-printable characters to #


Line 22:		Line 22:


-	4. From Excel, export the metadata as a Unicode ~~tab-delimited~~ file.	+	5. From Excel, export the metadata as a Unicode text file.


-	5. Open the Unicode export in Notepad ++ and choose "Encode in UTF-8". Repeat Step 3 and 4 until all problems are taken care of.	+	6. Open the Unicode export in Notepad ++ and choose "Encode in UTF-8 without BOM". Repeat Step 3 and 4 until all problems are taken care of.


-	6. If all is well, save the text file with NotePad ++ (it will be a UTF-8 file).	+	7. If all is well, save the text file with NotePad ++ (it will be a UTF-8 file).
	This UTF-8 file is now the file from which to create MODS with [http://acumen.lib.ua.edu/project/?f=Archivist%20Utility.txt Archivist Utility]. This is also the text version of the metadata that will go into long term Storage.		This UTF-8 file is now the file from which to create MODS with [http://acumen.lib.ua.edu/project/?f=Archivist%20Utility.txt Archivist Utility]. This is also the text version of the metadata that will go into long term Storage.
		+
		+	8. Encoding problems can be discovered along the way by looking for black rectangular three letter "blocks" in Notepad++ or diamond shaped question marks in Archivist Utility.

Jjcolonnaromano at 15:35, 14 December 2012

Jjcolonnaromano — Fri, 14 Dec 2012 15:35:35 GMT

←Older revision		Revision as of 15:35, 14 December 2012
Line 17:		Line 17:


-	4. Do a search for all instances of "#" and replace all found problems in the Excel (.xlsx) file. Use Excel's built in ~~“character map”~~ and choose Unicode as the encoding while making changes.	+	4. Do a search for all instances of "#" and replace all found problems in the Excel (.xlsx) file. Use Excel's built in character map. To access the character map window from within excel goto the Insert tab - Symbols group - Symbol, select the character you need to replace and choose "Unicode Hex" as the encoding while making changes.

	[http://intranet.lib.ua.edu/wiki/digcoll/images/e/e6/French_encoding_map.xlsx This Excel file] may be useful in getting one started in searching/replacing diacritics for French language metadata.		[http://intranet.lib.ua.edu/wiki/digcoll/images/e/e6/French_encoding_map.xlsx This Excel file] may be useful in getting one started in searching/replacing diacritics for French language metadata.

Scpstu32 at 16:40, 27 January 2011

Scpstu32 — Thu, 27 Jan 2011 16:40:32 GMT

←Older revision		Revision as of 16:40, 27 January 2011
Line 29:		Line 29:

	6. If all is well, save the text file with NotePad ++ (it will be a UTF-8 file).		6. If all is well, save the text file with NotePad ++ (it will be a UTF-8 file).
-	This UTF-8 file is now the file from which to create MODS with Archivist Utility. This is also the text version of the metadata that will go into long term Storage.	+	This UTF-8 file is now the file from which to create MODS with [http://acumen.lib.ua.edu/project/?f=Archivist%20Utility.txt Archivist Utility]. This is also the text version of the metadata that will go into long term Storage.

Scpstu32 at 16:29, 27 January 2011

Scpstu32 — Thu, 27 Jan 2011 16:29:55 GMT

←Older revision		Revision as of 16:29, 27 January 2011
Line 28:		Line 28:


-	6. If all is well, save text file in NotePad ++ (it will be a UTF-8 file).	+	6. If all is well, save the text file with NotePad ++ (it will be a UTF-8 file).
	This UTF-8 file is now the file from which to create MODS with Archivist Utility. This is also the text version of the metadata that will go into long term Storage.		This UTF-8 file is now the file from which to create MODS with Archivist Utility. This is also the text version of the metadata that will go into long term Storage.

Scpstu32 at 16:29, 27 January 2011

Scpstu32 — Thu, 27 Jan 2011 16:29:20 GMT

←Older revision		Revision as of 16:29, 27 January 2011
Line 11:		Line 11:


-	2. Open in the file in and choose "Encode in UTF-8".	+	2. Open in the file in NotePad ++ and choose "Encode in UTF-8".

Scpstu32 at 16:28, 27 January 2011

Scpstu32 — Thu, 27 Jan 2011 16:28:38 GMT

←Older revision		Revision as of 16:28, 27 January 2011
Line 4:		Line 4:
	Based on our experience with French language diacritics, the following process allowed MODS to be created without the numerous encoding problems that we initially encountered with the collection: u0002_0000006 (French Revolutionary Pamphlets).		Based on our experience with French language diacritics, the following process allowed MODS to be created without the numerous encoding problems that we initially encountered with the collection: u0002_0000006 (French Revolutionary Pamphlets).

		+	To do these steps, you will need a Windows computer with: Microsoft Excel and [http://notepad-plus.sourceforge.net/ NotePad ++].
		+	You will also need the TextFX plusing for NotePad ++.
		+	To get plugins for NotePad++ refer to the application documentation with regard to the Plugin Manager.

-	1. Export the metadata from Excel as a tab delimited text file (~~Do Not~~ use the Unicode option). If the Excel file is in the legacy .xls format, first convert it to .xlsx prior to export.	+	1. Export the metadata from Excel as a tab delimited text file (use the Unicode option). If the Excel file is in the legacy .xls format, first convert it to .xlsx prior to export.


-	2. Open in the ~~text~~ file in ~~Encodinator, which will help identify problems. Also, you can open in the file in [http://notepad-plus.sourceforge.net/ NotePad ++]~~ and choose "Encode in UTF-8"~~, as this will also help show where the problems are. A third option is to open the file in your web browser and manually change the encoding to UTF-8~~.	+	2. Open in the file in and choose "Encode in UTF-8".


-	3. ~~Search and replace all found problems in~~ the ~~Excel (.xlsx) file. Use Excel's built in “character map”~~ and choose ~~Unicode as the encoding.~~	+	3. Use the TextFX plugin and choose: TextFX Characters>zap all non-printable characters to #

-	[http://intranet.lib.ua.edu/wiki/digcoll/images/e/e6/French_encoding_map.xlsx This Excel file] may be useful in getting one started in searching/replacing diacritics for French language metadata.

		+	4. Do a search for all instances of "#" and replace all found problems in the Excel (.xlsx) file. Use Excel's built in “character map” and choose Unicode as the encoding while making changes.

-	4. ~~From~~ Excel~~, export the metadata as a Unicode tab-delimited~~ file.	+	[http://intranet.lib.ua.edu/wiki/digcoll/images/e/e6/French_encoding_map.xlsx This Excel file] may be useful in getting one started in searching/replacing diacritics for French language metadata.


-	5. ~~Open~~ the Unicode ~~export in Notepad ++ and choose "Convert to ANSI"~~.	+	4. From Excel, export the metadata as a Unicode tab-delimited file.


-	6. ~~Load~~ the ~~ANSI text file into Encodinator~~ and ~~make sure all is well~~. ~~If there are still problems, go back to step~~ 3.	+	5. Open the Unicode export in Notepad ++ and choose "Encode in UTF-8". Repeat Step 3 and 4 until all problems are taken care of.


-	7. If all is well, ~~open the ANSI~~ text file in NotePad ++ ~~and choose "Convert to~~ UTF-8".	+	6. If all is well, save text file in NotePad ++ (it will be a UTF-8 file).
	This UTF-8 file is now the file from which to create MODS with Archivist Utility. This is also the text version of the metadata that will go into long term Storage.		This UTF-8 file is now the file from which to create MODS with Archivist Utility. This is also the text version of the metadata that will go into long term Storage.