Data processing

• Date, time, and Unix
  • Parse a date in string format
• Text normalisation
  • Using the tool
  • Example of normalised data in XML format generated with VARD
• PDF, Word, images
  • Extract text from PDF and other multimedia formats
    • Installing the tool
    • Using the tool
• Language detection
  • Identify a set of predefined languages in .txt files and write a summary report in spreadsheet format
• Emoticons and emojis
  • Install the required module
  • Function to transliterate emojis (using two different output formats)
• Hashtags (word segmentation)
  • Install the required module
  • Segment hashtags and transform them into XML tags in a XML corpus file
  • Example of hashtags transformed through [s5.17]
  • Word segmentation tools for languages other than English
• Other elements
  • Regular expression to capture usernames (e.g. @matteodic)
  • Regular expression to capture simple URLs (e.g. http://example.com and https://example.com)
  • Regular expression to capture complex URLs (e.g. simple URLs plus email addresses, mailto: links, URLs with optional parameters)
  • Regular expression to capture cashtags (e.g. $EUR)
• Annotations
  • Stanza
    • Installing the tool
    • Importing the module and installing a language model
    • Annotating textual data from .txt files into XML format
      • How to use script [s5.19]
    • Annotating textual data from .txt files into verticalised XML format
      • How to use script [s5.20]
    • Example of data extracted with script [s5.19]
• Verticalised (.vrt) format