This flashcard is just one of a free flashcard set. See all flashcards!

• Separate a text (String) to its tokens

     • Example — Input:
     • „Natural language processing makes fun.“

     • Result:
     • „Natural“, „language“, „processing“, „makes“, „fun“, “.“


• Best practice is to work without punctuations and
lowercased tokens (normalization of tokens).

     • Normalized result:
     • „natural“, „language“, „processing“, „makes“, „fun“