tinker_cookbook.eval.TrainingClientEvaluator

class tinker_cookbook.eval.TrainingClientEvaluator()

An evaluator that uses a TrainingClient to compute metrics (e.g., loss).

class NLLEval(TrainingClientEvaluator):
    async def __call__(self, training_client):
        loss = await compute_nll(training_client, self.eval_data)
        return {"eval/nll": loss}

call( training_client)

Run evaluation and return a dict of metric names to values.

Parameters:

• training_client ( tinker. TrainingClient) – The Tinker training client to evaluate with.

Returns: A dict mapping metric names to float values.