MetaCAT Model Evaluation

MedCAT
1

When training a MedCAT model with some Meta-Annotations you get Precision/Recall/F1 scores outputted (generated by the medcat.metacat.MetaCAT.eval() method).

What is this evaluating - is it performance of the MedCAT model identifying concepts, or is it evaluating the performance of meta models choosing categories?

I’m trying to evaluate the performance of our meta models.

2

Answered my own question - if you have a metacat model object, and call .eval() on it with a filepath to a MedCATtrainer project export as the argument, it evaluates the one meta model.

meta_model.eval('path/to/annotation/project/export.json')

3

Hey!
Yes correct it needs a “gold standard” to compare its predictions against in order to produce metrics. Glad you found it