# tinker_cookbook.tokenizer_utils.TmlTokenizer

## _class_ [**tinker_cookbook.tokenizer_utils.TmlTokenizer**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/tokenizer_utils.py#L68)( _Protocol_)

Structural type for the underlying `tml_renderers` tokenizer.

Cookbook never constructs this object (`tml_renderers.tokenizers` does), and `tml_renderers` is an optional, lazily imported dependency, so its concrete type is not referenceable here. This protocol pins the small surface the cookbook adapter and renderer actually rely on, making the contract explicit instead of `Any` duck-typing.

**Fields:**

- [**bos_token**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/tokenizer_utils.py#L78) ( _str_)
- [**eos_token**](https://github.com/thinking-machines-lab/tinker-cookbook/blob/main/tinker_cookbook/tokenizer_utils.py#L79) ( _str_)

## Referenced by

- [tinker_cookbook.tokenizer_utils.SupportsTmlTokenizer](https://tinker-docs.thinkingmachines.ai/cookbook/api-reference/tokenizer_utils/supportstmltokenizer/)
- [tinker_cookbook.tokenizer_utils.TmlRenderersTokenizerAdapter](https://tinker-docs.thinkingmachines.ai/cookbook/api-reference/tokenizer_utils/tmlrendererstokenizeradapter/)