Multiple NLP models or single one

Healthcare Use Cases
1

Hi… We are using MedCAT on clinical notes of patients. One of the options being considered was training a generalized model and multiple specialized models depending on the disease group. This needs using multiple models on the text and determining the expert for every annotation. Wanted to check if there is merit in using multiple models or stick to a single model. Are there any relevant usecases with Cogstack/MedCAT or papers that one can refer to.

2

The ideal scenario is a single model which has been trained and validated by the respective domain specialists. I.e. Cardiologists validating cardiology reports, Epileptologists validating epilepsy clinic letters etc…

Validating and training all possible concepts at once may not be possible. Therefore at the beginning there are times when you may have to create a model fork (e.g. Epilepsy specialist model) as you have only validated it across the epilepsy domain. In my experience setting up and training models by specialty is easier.

Over time you, once have gathered enough training/validation material across multiple domains from MedCATtrainer and can then retrain a single model.

I would only intentionally look to train different models in two specific circumstances:

  1. If you expect different interpretations of the same spans of text. For example: consider the text: “The pt has Juvenile Myoclonic Epilepsy” the outputs of two different models could be:

Output model 1: Juvenile-> 59223006, Myoclonic->17450006, Epilepsy-> 84757009
Output model 2: Juvenile myoclonic epilepsy → 6204001

  1. If you are annotating and linking with different Ontologies/terminologies e.g. dm+d, SNOMED, UMLS and/or different versions/releases of the ontology. Then I would advise to keep these as separate model, as there may be different concept representations across the same word/phrase span and will consequently clash.
1 Like
3

Thanks… this is useful

4

As Anthony says, an ideal approach would be one model one which is broadly generalisable across all domains. The direct opposite approach is a model zoo where there is one model for each task in each domain.

The main risk with overly generalising is overlapping acronyms that different specialties or clinicians use, but this is not very common (e.g. PD meaning Parkinson’s Disease to some, and Personality Disorder to others).

I think the reality is in between: a broadly generalisable one which is generalisable across most domains and focused very light fine-tuning needed for specific tasks.

1 Like