Knowledge Graph — Coursera Notes › Academic disciplines › Information Technology / Computer Science › Artificial Intelligence › Natural Language Processing (NLP)

Tokenization

concept · part of Natural Language Processing (NLP)

Tokenization converts text into a format that models can process, typically by splitting text into tokens (words or subwords) and mapping them to integer IDs. Hugging Face's AutoTokenizer provides tokenization with options like truncation and padding to a fixed length.

Tokenization splits text into smaller units called tokens (words, subwords, or characters). Example: 'The quick brown fox' becomes ['The', 'quick', 'brown', 'fox']. It allows NLP systems to process text by breaking language into manageable parts.

Tokenization should always be the first step in an NLP pipeline. The output must be compatible with downstream components like POS tagging and NER; for example, tokenization should not split proper names like 'New York City' if NER is used later.

This is the text view of an interactive 3D knowledge graph — open this page with JavaScript enabled to explore it visually.