get_tokenizer - Tinker Documentation
tinker_cookbook.tokenizer_utils.get_tokenizer
tinker_cookbook.tokenizer_utils.get_tokenizer( model_name)
Get a tokenizer by name.
Checks the custom registry first (see register_tokenizer), then returns a tml-renderers tokenizer for Inkling, or falls back to HuggingFace AutoTokenizer. HuggingFace tokenizers are cached after first load.
Parameters:
- model_name ( str) – HuggingFace model identifier (e.g. "Qwen/Qwen3-8B") or a custom registered name.
Returns: Tokenizer – A tml-renderers tokenizer for Inkling, otherwise a HuggingFace PreTrainedTokenizer.
from tinker_cookbook.tokenizer_utils import get_tokenizer
tokenizer = get_tokenizer("Qwen/Qwen3-8B")
tokens = tokenizer.encode("Hello world")