Model Serving · Software component

Tokenizer

Software componentModel ServingModelsarc:Tokenizer

A model-specific component that segments text into the subword tokens a language model processes, determining the token counts against which context capacity is measured.

Responsibility. Converts text into the token sequence of a specific model.

Also known as: Tokenization, tiktoken

Direct neighbourhood (hover for relationship types)

Relationships

is invoked by dependency

Design guidance

MUST count tokens with the target model's tokenizer, not words or characters, when budgeting context.

Quantitative guidance

As stated by the sources; verify before use.

~1.3-1.5 tokens per English word; code tokenizes ~1.5-2x less efficiently than English; lower-resource non-English text can need 2-3x more tokens (Ch5.9).

Classification

Patterns: Statistical subword tokenization learned during training
Quality attributes: Compatibility (ISO/IEC 25010)

Sources

Ch5.9: T. Nguyen, "Working Memory," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 5.9. ISBN: 9798244538229.
Ch6.3A: T. Nguyen, "ETL Pipeline Fundamentals," in Mastering Agentic AI Systems: Guide for the NVIDIA NCP-AAI Exam, 1st ed. 2026, ch. 6.3A. ISBN: 9798244538229.