feat(medcat-performance-estimation): Add MedCAT Performance estimation

[mart-r]

This PR adds a separate perfomance estimation project to the project.

There's a lot in the project README.

But the TLDR I would say is the following:

• We try to estimate the performance
  • We'll be tiering in the end (A, B, C)
• We can't guarantee anything on a per concept basis
  • But in the example
    • With SnomedCT Entity Linking Challenge
    • Concepts in tier A (on average) outperformed those in B (on average)
    • Concepts in tier B (on average) outperformed those in C (on average)
• There is a LOT OF noise in the estimations / output data
  • Some due to missing information (i.e missing synonyms)
  • Others simply due to other missing context

I'll throw in some noisy plots as well

As we can see the average performance for tier A is higher than B, which is higher than C (sorry for the odd layout).

[adam-sutton-1992]

This is big weow.

Before I attempt the nitty gritty theres high level questions.

I see the guide says:

python -m medcat_performance_estimator.cli \
    --model-pack path/to/model_pack.zip \
    --cuis 195967001,22298006 \
    --output estimates.json

as a guide on how to run it.

But in this PR you say:

We can't guarantee anything on a per concept basis

    But in the example
        With SnomedCT Entity Linking Challenge

How is this tiering concept... performance over any single dataset? Unless this model has been trained on that given dataset after you mean?

The tiers should be used to decide where training effort is best spent on average, not to make claims about a specific concept.

I kinda have two or three (or four) different questions for this:

1. Don't you (ideally) train for the distribution of data that you have? Not what this performance estimation would give you.
2. If you had infinite data to train on. Wouldn't you just train for what you're weak on at test time (i.e. high false negatives / false positives).
3. If being in Tier A is desirable. Wouldn't you always train on Tier C concepts if you had the luxury of making that choice?
4. I know we're speaking in general / specifics interchangeably. But could training on a Tier A concept improve the confidence of those Tier C concepts, by reducing the noise on edge cases. And saying what potential concepts could be not?

I think it still has uses. Say in a cardiology project you could say "we're a bit naff at concepts <x,y,z>", for some disclaimers. Or where you're "strong". I'm just sceptical how it informs training - unless you're going to buy annotations concepts that are Tier C... somehow.

I'll look into the code more detailed later. It's a big boy.

[mart-r]

I see the guide says:

python -m medcat_performance_estimator.cli \
    --model-pack path/to/model_pack.zip \
    --cuis 195967001,22298006 \
    --output estimates.json

as a guide on how to run it.

Good point, the guide / README should clearly note that only estimating for a few concepts is unlikely to be super informative.
I've made the change now.

How is this tiering concept... performance over any single dataset? Unless this model has been trained on that given dataset after you mean?

The above comparison is taking the concepts in the Snomed CT linking challenge dataset and estimating their performance. Then it's running get_stats to measure the actual performance. And that's where the comparison comes from - the estimation vs the measured performance.

I kinda have two or three (or four) different questions for this:

1. Don't you (ideally) train for the distribution of data that you have? Not what this performance estimation would give you.
2. If you had infinite data to train on. Wouldn't you just train for what you're weak on at test time (i.e. high false negatives / false positives).
3. If being in Tier A is desirable. Wouldn't you always train on Tier C concepts if you had the luxury of making that choice?
4. I know we're speaking in general / specifics interchangeably. But could training on a Tier A concept improve the confidence of those Tier C concepts, by reducing the noise on edge cases. And saying what potential concepts could be not?

I think it still has uses. Say in a cardiology project you could say "we're a bit naff at concepts <x,y,z>", for some disclaimers. Or where you're "strong". I'm just sceptical how it informs training - unless you're going to buy annotations concepts that are Tier C... somehow.

I think this may not be quite clear in all of the documentation.
But the main idea here is that a clinician and/or a data scientist wants to do a project. They've curated the concepts they'll need to cover. And then they're wondering "How well will MedCAT do for these concepts? Do I need to do fine tuning beforehand or will the model perform well out of the box for these?".
And that's what this is trying to help with (though not fully answer - we do not know what we do not know). It's trying to inform on which set of concepts the model will likely perform better on and which ones it'll likely perform worse at. So if everything they care about is in tier C, then they migth want a different approach or may need to allocate more time for clinicians / coders to annotate data. Yet if everything is in tier A, then they might be able to get away with far less manual annotations.

So this is a tool intended to be used at the early stages of a project. Where there is an idea of what types of concepts will be of interest, but there is no curated data yet and/or there is no annotations / annotators yet available. It's intended to help someone plan for this.

Now, the other side is that given how poorly its estimations are doing, this might not do that well at what it's trying to do. But there is some signal here.

EDIT: To answer the questions more directly:

1. This is intended for the stage where you have no data to train with (yet). It's intended to help inform what type of and how much data might be needed to be annotated in order to train the model.
2. Again, this is for the situation where you have no training data (yet). So there is no empircal knowledge of what the model is strong or weak at. This is trying to fill that gap a bit by providing an estimate.
3. Yes, you would always train tier C, I'd imagine. But the other option would be to look for some other model (or another process altogether) that works better out of the box.
4. Yes, that will almost certainly still be the case. But it's not what this estimation is trying to estimate at this point.

[adam-sutton-1992]

If in:

python -m medcat_performance_estimator.cli \
    --model-pack path/to/model_pack.zip \
    --cuis 195967001,22298006 \
    --output estimates.json

195967001,22298006 are CUIs you know are in the snomed entity challenge set. Why don't you offer the option to link to a trainer export file so all cuis can be done there, rather than manually entering all of them into the estimator. unless I've missed that option. (I now see --cui-file, could this lazily be a trainer export?)

A final high level question before I look at the code; this only works with the core components - right?

[mart-r]

195967001,22298006 are CUIs you know are in the snomed entity challenge set. Why don't you offer the option to link to a trainer export file so all cuis can be done there, rather than manually entering all of them into the estimator. unless I've missed that option. (I now see --cui-file, could this lazily be a trainer export?)

That's precisely counter to the point. If you have a trainer export, you don't need to estimate performance. You can just evaluate it by running metrics.

This estimation business (i.e this PR) is for when you do not have the luxury of existing data.

A final high level question before I look at the code; this only works with the core components - right?

Stage 1 / ontology estimation is a general estimation on how ambiguous something is. So that will - in principle - estimate any components that base their estimations on the CDB (i.e the concepts and their synonyms).
But stage 2 / training aware estimation relies on train counts as well. So that will only work for components that update these.

With that said, for different implementations we'd almost certianly need a different calibration curve (i.e stage 3). So if there is some signal from the same pieces of information, it could still estimate something.

With that said, as you can see from the estimations, they're not super accurate anyway. So I'd assume they'd be even worse for something I didn't try to match them to. But at the same time, I don't know that for a fact.

[adam-sutton-1992]

OKay - i see that the CUIs you used were an example - i.e. you want to see how you do with those cuis before you decide to train. All understood.

[adam-sutton-1992]

This is only useful for SNOMED?

[mart-r]

I suppose it could work for anything that defines pt2ch in the same manner. But that's not really well defined for anything else (in terms of model building that is).

[adam-sutton-1992]

I'd abstract this if you care enough to then. Document it works for snomed if you prefer.

Up to you.

[adam-sutton-1992]

When we say contextual embedding spaces here - and a brief scroll through this file. I think this can only used the vector context model (with xsmall -> xlarge contexts weights). I may be wrong on this...

Just a minor point on terminology - these embedding spaces are not contextual. That would be LSTM embeddings and beyond.

I'm not sure at this point if you can use this on transformer NER. Just checking that this is only for the vector context model trained models.

[mart-r]

In principle, if some other component saves the context vectors in the same manner (and/or changes the config in line with how it saves them), this could work with some other component as well.

But more realistically some other component probably won't even touch these things and thus there will be nothing for this to use for estimations.

And you could use this with transformers NER, I just don't see a world where it'll do a good job at the estimation. Or - perhaps more accurately - I'd expect it to do even worse.

This doesn't touch the components themselves. It touches the central pieces. But many non-default components (such as transformers NER and embedding linker) probably just don't touch these (i.e train counts and per cui context vectors in CDB.cui2info).

As for the terminology, I should have paid more careful attention on what was in there. But the point is that there's no expectation that the range of cosine simularity values spans the entire theoretical range (i.e from -1 to 1).

[adam-sutton-1992]

Cool.