tinker_cookbook.tokenizer_utils.register_tokenizer

tinker_cookbook.tokenizer_utils.register_tokenizer( name, factory)

Register a custom tokenizer factory.

Once registered, get_tokenizer will call factory() instead of loading from HuggingFace when the given name is requested.

Parameters:

• name ( str) – The tokenizer name (typically a HuggingFace model ID like "Foo/foo_tokenizer").
• factory ( Callable[[], Tokenizer]) – A callable that takes no arguments and returns a Tokenizer instance.

Returns: None

def my_tokenizer_factory():
    return MyCustomTokenizer()
register_tokenizer("Foo/foo_tokenizer", my_tokenizer_factory)

Referenced by

• tinker_cookbook.tokenizer_utils.get_registered_tokenizer_names
• tinker_cookbook.tokenizer_utils.get_tokenizer
• tinker_cookbook.tokenizer_utils.is_tokenizer_registered