tinker.types.LoraConfig

class tinker.types.LoraConfig( StrictBase)

Fields:

• rank ( int) – LoRA rank (dimension of low-rank matrices)
• seed ( Optional[int]) – Seed used for initialization of LoRA weights.

Useful if you need deterministic or reproducible initialization of weights.

• train_unembed ( bool) – Whether to add lora to the unembedding layer

• train_mlp ( bool) – Whether to add loras to the MLP layers (including MoE layers)

• train_attn ( bool) – Whether to add loras to the attention layers