Diacritics

From UA Libraries Digital Services Planning and Documentation
(Difference between revisions)
Jump to: navigation, search
(14 intermediate revisions by 2 users not shown)
Line 1: Line 1:
As metadata spreadsheets exchange hands and are often even created from diverse sources, issues arise regarding diacritics. These characters often do no translate from encoding to encoding, creating poor results in the resultant [MODS_Mockup MODS metadata files].  
+
As metadata spreadsheets exchange hands and are often even created from diverse sources, issues arise regarding diacritics. These characters often do no translate from encoding to encoding, creating poor results in the resultant [[MODS_Mockup|MODS metadata files]].  
  
  
Based on our experience with French language diacritics, the following process allowed MODS to be created without the numerous encoding problems that we initially encountered with the collection: u0002_0000006 (French Revolutionary Metadata).
+
Based on our experience with French language diacritics, the following process allowed MODS to be created without the numerous encoding problems that we initially encountered with the collection: u0002_0000006 (French Revolutionary Pamphlets).
  
 +
  To do these steps, you will need a Windows computer with: Microsoft Excel and [http://notepad-plus.sourceforge.net/ NotePad ++].
 +
  You will also need the TextFX plusing for NotePad ++.
 +
  To get plugins for NotePad++ refer to the application documentation with regard to the Plugin Manager.
  
1. Export the metadata from Excel as a tab delimited text file (Do Not use the Unicode option). If the Excel file is in the legacy .xls, first convert it to .xlsx prior to export.
+
1. Export the metadata from Excel as a tab delimited text file (use the Unicode option). If the Excel file is in the legacy .xls format, first convert it to .xlsx prior to export.
  
  
2. Open in the text file in [http://lb-416-003.lib.ua-net.ua.edu/notes/?f=Encodinator.txt Encodinator], which will help identify problems. Also, you can open in the file in [http://notepad-plus.sourceforge.net/ NotePad ++] and choose "Encode in UTF-8", as this will also help show where the problems are.
+
2. Open the file in NotePad ++ and choose "Encode in UTF-8".
  
  
3. Search and replace all found problems these in the Excel (.xlsx) file. Use Excel's built in “character map” and choose Unicode as the encoding.
+
3. If you dont know where the problem characters are, Use the TextFX plugin and choose: TextFX Characters>zap all non-printable characters to #
  
[http://intranet.lib.ua.edu/wiki/digcoll/images/e/e6/French_encoding_map.xlsx This Excel file] may be useful in getting one started in searching/replacing diacritics for French language metadata.
 
  
 +
4. Do a search for all instances of "#" and using the text file in notepad++ as a guide, replace all found problems in the *Excel* (.xlsx) file. Use Excel's built in character map. To access the character map window from within excel goto the Insert tab - Symbols group - Symbol, select the character you need to replace and choose "Unicode Hex" as the encoding while making changes.
 +
 +
[http://intranet.lib.ua.edu/wiki/digcoll/images/e/e6/French_encoding_map.xlsx This Excel file] may be useful in getting one started in searching/replacing diacritics for French language metadata.
  
4. From Excel, export the metadata as a Unicode tab-delimited file.
 
  
 +
5. From Excel, export the metadata as a Unicode text file.
  
5. Open the Unicode export in Notepad ++ choose "Convert to ANSI".
 
  
 +
6. Open the Unicode export in Notepad ++ and choose "Encode in UTF-8 without BOM". Repeat Step 3 and 4 until all problems are taken care of.
  
6. Load the ANSI text file into Encodinator and make sure all is well. If there are still problems, go back to step 3.
 
  
 +
7. If all is well, save the text file with NotePad ++ (it will be a UTF-8 file).
 +
  This UTF-8 file is now the file from which to create MODS with [http://acumen.lib.ua.edu/project/?f=Archivist%20Utility.txt Archivist Utility]. This is also the text version of the metadata that will go into long term Storage.
  
7. If all is well, open the ANSI text file in NotePad ++ and choose "Convert to UTF-8".
+
8. Encoding problems can be discovered along the way by looking for black rectangular three letter "blocks" in Notepad++ or diamond shaped question marks in Archivist Utility.
  This UTF-8 file is now the file from which to create MODS with [http://lb-416-003.lib.ua-net.ua.edu/notes/?f=MARC Utility.txt Archivist Utility]. This is also the text version of the metadata that will go into long term Storage.
+

Revision as of 15:04, 25 January 2013

As metadata spreadsheets exchange hands and are often even created from diverse sources, issues arise regarding diacritics. These characters often do no translate from encoding to encoding, creating poor results in the resultant MODS metadata files.


Based on our experience with French language diacritics, the following process allowed MODS to be created without the numerous encoding problems that we initially encountered with the collection: u0002_0000006 (French Revolutionary Pamphlets).

  To do these steps, you will need a Windows computer with: Microsoft Excel and NotePad ++.
  You will also need the TextFX plusing for NotePad ++.
  To get plugins for NotePad++ refer to the application documentation with regard to the Plugin Manager.

1. Export the metadata from Excel as a tab delimited text file (use the Unicode option). If the Excel file is in the legacy .xls format, first convert it to .xlsx prior to export.


2. Open the file in NotePad ++ and choose "Encode in UTF-8".


3. If you dont know where the problem characters are, Use the TextFX plugin and choose: TextFX Characters>zap all non-printable characters to #


4. Do a search for all instances of "#" and using the text file in notepad++ as a guide, replace all found problems in the *Excel* (.xlsx) file. Use Excel's built in character map. To access the character map window from within excel goto the Insert tab - Symbols group - Symbol, select the character you need to replace and choose "Unicode Hex" as the encoding while making changes.

This Excel file may be useful in getting one started in searching/replacing diacritics for French language metadata.


5. From Excel, export the metadata as a Unicode text file.


6. Open the Unicode export in Notepad ++ and choose "Encode in UTF-8 without BOM". Repeat Step 3 and 4 until all problems are taken care of.


7. If all is well, save the text file with NotePad ++ (it will be a UTF-8 file).

  This UTF-8 file is now the file from which to create MODS with Archivist Utility. This is also the text version of the metadata that will go into long term Storage.

8. Encoding problems can be discovered along the way by looking for black rectangular three letter "blocks" in Notepad++ or diamond shaped question marks in Archivist Utility.

Personal tools