medcat.utils.regression.results

Module Contents

Classes

Finding

Describes whether or how the finding verified.

FindingDeterminer

A helper class to determine the type of finding.

Strictness

The total strictness on which to judge the results.

SingleResultDescriptor

The result descriptor.

ResultDescriptor

The overarching result descriptor that handles multiple phrases.

MultiDescriptor

The descriptor of results over multiple different results (parts).

Attributes

STRICTNESS_MATRIX

class medcat.utils.regression.results.Finding

Bases: enum.Enum

Describes whether or how the finding verified.

The idea is that we know where we expect the entity to be recognised and the enum constants describe how the recognition compared to the expectation.

In essence, we want to know the relative positions of the two pairs of numbers (character numbers): - Expected Start, Expected End - Recognised Start, Recognised End

We can model this as 4 numbers on the number line. And we want to know their position relative to each other. For example, if the expected positions are marked with * and recognised positions with #, we may have something like: ___*__#_______#*______________ Which would indicate that there is a partial, but smaller span recognised.

IDENTICAL

The CUI and the span recognised are identical to what was expected.

BIGGER_SPAN_RIGHT

The CUI is the same, but the recognised span is longer on the right.

If we use the notation from the class doc string, e.g: _*#__*__#

BIGGER_SPAN_LEFT

The CUI is the same, but the recognised span is longer on the left.

If we use the notation from the class doc string, e.g: _#_*__*#_

BIGGER_SPAN_BOTH

The CUI is the same, but the recognised span is longer on both sides.

If we use the notation from the class doc string, e.g: _#__*__*__#_

SMALLER_SPAN

The CUI is the same, but the recognised span is smaller.

If we use the notation from the class doc string, e.g: _*_#_#_*_ (neither start nor end match) _*#_#_*__ (start matches, but end is before expected) _*__#_#*_ (end matches, but start is after expected)

PARTIAL_OVERLAP

The CUI is the same, but the span overlaps partially.

If we use the notation from the class doc string, e.g: _*_#__*_#_ (starts between expected start and end, but ends beyond) _#_*_#_*__ (start before expected start, but ends between expected start and end)

FOUND_DIR_PARENT

The recognised CUI is a parent of the expected CUI but the span is an exact match.

FOUND_DIR_GRANDPARENT

The recognised CUI is a grandparent of the expected CUI but the span is an exact match.

FOUND_ANY_CHILD

The recognised CUI is a child of the expected CUI but the span is an exact match.

FOUND_CHILD_PARTIAL

The recognised CUI is a child yet the match is only partial (smaller/bigger/partial).

FOUND_OTHER

Found another CUI in the same span.

FAIL

The concept was not recognised in any meaningful way.

has_correct_cui()

Whether the finding found the correct concept.

Returns:

bool – Whether the correct concept was found.

Return type:

bool

classmethod determine(exp_cui, exp_start, exp_end, tl, found_entities, strict_only=False, check_children=True, check_parent=True, check_grandparent=True)

Determine the finding type based on the input

Parameters:

  • exp_cui (str) – Expected CUI.

  • exp_start (int) – Expected span start.

  • exp_end (int) – Expected span end.

  • tl (TranslationLayer) – The translation layer.

  • found_entities (Dict[str, Dict[str, Any]]) – The entities found by the model.

  • strict_only (bool) – Whether to use a strict-only mode (either identical or fail). Defaults to False.

  • check_children (bool) – Whether to check the children. Defaults to True.

  • check_parent (bool) – Whether to check for parent(s). Defaults to True.

  • check_grandparent (bool) – Whether to check for grandparent(s). Defaults to True.

Returns:

Tuple[‘Finding’, Optional[str]] – The type of finding determined, and the alternative.

Return type:

Tuple[Finding, Optional[str]]

class medcat.utils.regression.results.FindingDeterminer(exp_cui, exp_start, exp_end, tl, found_entities, strict_only=False, check_children=True, check_parent=True, check_grandparent=True)

A helper class to determine the type of finding.

This is mostly useful to split the responsibilities of looking at children/parents as well as to keep track of the already-checked children to avoid infinite recursion (which could happen in - e.g - a SNOMED model).

Parameters:

  • exp_cui (str) – The expected CUI.

  • exp_start (int) – The expected span start.

  • exp_end (int) – The expected span end.

  • tl (TranslationLayer) – The translation layer.

  • found_entities (Dict[str, Dict[str, Any]]) – The entities found by the model.

  • strict_only (bool) – Whether to use strict-only mode (either identical or fail). Defaults to False.

  • check_children (bool) – Whether or not to check the children. Defaults to True.

  • check_parent (bool) – Whether to check for parent(s). Defaults to True.

  • check_grandparent (bool) – Whether to check for granparent(s). Defaults to True.

__init__(exp_cui, exp_start, exp_end, tl, found_entities, strict_only=False, check_children=True, check_parent=True, check_grandparent=True)
Parameters:

  • exp_cui (str) –

  • exp_start (int) –

  • exp_end (int) –

  • tl (medcat.utils.regression.targeting.TranslationLayer) –

  • found_entities (Dict[str, Dict[str, Any]]) –

  • strict_only (bool) –

  • check_children (bool) –

  • check_parent (bool) –

  • check_grandparent (bool) –

Return type:

None

_determine_raw(start, end)

Determines the raw SPAN-ONLY finding.

I.e this assumes the concept is appropriate. It will return None if there is no overlapping span.

Parameters:

  • start (int) – The start of the span.

  • end (int) – The end of the span.

Raises:
Returns:

Optional[Finding] – The finding, if a match is found.

Return type:

Optional[Finding]

_get_strict()
Return type:

Optional[Finding]

_check_parents()
Return type:

Optional[Tuple[Finding, Optional[str]]]

_check_children()
Return type:

Optional[Tuple[Finding, Optional[str]]]

_descr_cui(cui)
Parameters:

cui (Optional[str]) –

Return type:

Optional[str]

_find_diff_cui()
Return type:

Optional[Tuple[Finding, str]]

determine()

Determine the finding based on the given information.

First, the strict check is done (either identical or not). Then, parents are checked (if required). After that, children are checked (if required).

Returns:

Tuple[Finding, Optional[str]] – The appropriate finding, and the alternative (if applicable).

Return type:

Tuple[Finding, Optional[str]]

_determine()
Return type:

Tuple[Finding, Optional[str]]

class medcat.utils.regression.results.Strictness

Bases: enum.Enum

The total strictness on which to judge the results.

STRICTEST

The strictest option which only allows identical findings.

STRICT

A strict option which allows identical or children.

NORMAL

Normal strictness also allows partial overlaps on target concept and children.

LENIENT

Lenient stictness also allows parents and grandparents.

ANYTHING

Anything stricness allows ANY finding.

This would generally only be relevant when disabling examples for results descriptors.

medcat.utils.regression.results.STRICTNESS_MATRIX: Dict[Strictness, Set[Finding]]
class medcat.utils.regression.results.SingleResultDescriptor

Bases: pydantic.BaseModel

The result descriptor.

This class is responsible for keeping track of all the findings (i.e how many were found to be identical) as well as the examples of the finding on a per-target basis for further analysis.

name: str

The name of the part that was checked

findings: Dict[Finding, int]

The description of failures

examples: List[Tuple[medcat.utils.regression.targeting.FinalTarget, Tuple[Finding, str | None]]] = []

The examples of non-perfect alignment.

report_success(target, found)

Report a test case and its successfulness.

Parameters:

  • target (FinalTarget) – The target configuration

  • found (Tuple[Finding, Optional[str]]) – Whether or not the check was successful

Return type:

None

get_report()

Get the report associated with this descriptor

Returns:

str – The report string

Return type:

str

dict(**kwargs)
Return type:

dict

json(**kwargs)
Return type:

str

class medcat.utils.regression.results.ResultDescriptor

Bases: SingleResultDescriptor

The overarching result descriptor that handles multiple phrases.

This class keeps track of the results on a per-phrase basis and can be used to get the overall report and/or iterate over examples.

per_phrase_results: Dict[str, SingleResultDescriptor]
report(target, finding)

Report a test case and its successfulness

Parameters:

  • target (FinalTarget) – The final targe configuration

  • finding (Tuple[Finding, Optional[str]]) – To what extent the concept was recognised

Return type:

None

iter_examples(strictness_threshold)

Iterate suitable examples.

The strictness threshold at which to include examples.

Any finding that is assumed to be “correct enough” according to the strictness matrix for this threshold will be withheld from examples.

In simpler terms, if the finding is NOT in the strictness matrix for this strictness, the example is recorded.

NOTE: To disable example keeping, set the threshold to Strictness.ANYTHING.

Parameters:

strictness_threshold (Strictness) – The strictness threshold.

Yields:

Iterable[Tuple[FinalTarget, Tuple[Finding, Optional[str]]]] – The placeholder, phrase, finding, CUI, and name.

Return type:

Iterable[Tuple[medcat.utils.regression.targeting.FinalTarget, Tuple[Finding, Optional[str]]]]

get_report(phrases_separately=False)

Get the report associated with this descriptor

Parameters:

phrases_separately (bool) – Whether to output descriptor for each phrase separately

Returns:

str – The report string

Return type:

str

dict(**kwargs)
Return type:

dict

class medcat.utils.regression.results.MultiDescriptor

Bases: pydantic.BaseModel

The descriptor of results over multiple different results (parts).

The idea is that this would likely be used with a regression suite and it would incorporate all the different regression cases it describes.

property findings: Dict[Finding, int]

The total findings.

Returns:

Dict[Finding, int] – The total number of successes.

Return type:

Dict[Finding, int]

name: str

The name of the collection being checked

parts: List[ResultDescriptor] = []

The parts kept track of

iter_examples(strictness_threshold)

Iterate over all relevant examples.

Only examples that are not in the strictness matrix for the specified threshold will be used.

Parameters:

strictness_threshold (Strictness) – The threshold of avoidance.

Yields:

Iterable[Tuple[FinalTarget, Tuple[Finding, Optional[str]]]] – The examples

Return type:

Iterable[Tuple[medcat.utils.regression.targeting.FinalTarget, Tuple[Finding, Optional[str]]]]

_get_part_report(part, allowed_findings, total_findings, hide_empty, examples_strictness, phrases_separately, phrase_max_len)
Parameters:

  • part (ResultDescriptor) –

  • allowed_findings (Set[Finding]) –

  • total_findings (Dict[Finding, int]) –

  • hide_empty (bool) –

  • examples_strictness (Optional[Strictness]) –

  • phrases_separately (bool) –

  • phrase_max_len (int) –

Return type:

Tuple[str, int, int, int]

calculate_report(phrases_separately=False, hide_empty=False, examples_strictness=Strictness.STRICTEST, strictness=Strictness.NORMAL, phrase_max_len=80)

Calculate some of the major parts of the report.

Parameters:

  • phrases_separately (bool) – Whether to include per-phrase information

  • hide_empty (bool) – Whether to hide empty cases

  • examples_strictness (Optional[Strictness.STRICTEST]) – What level of strictness to show for examples. Set to None to disable examples. Defaults to Strictness.STRICTEST.

  • strictness (Strictness) – The strictness of the success / fail overview. Defaults to Strictness.NORMAL.

  • phrase_max_len (int) – The maximum length of the phrase in examples. Defaults to 80.

Returns:

Tuple[int, int, int, int, str] – The total number of examples, the total successes, the total failures, the delegated part, and the number of empty

Return type:

Tuple[int, int, int, str, int]

get_report(phrases_separately, hide_empty=False, examples_strictness=Strictness.STRICTEST, strictness=Strictness.NORMAL, phrase_max_len=80)

Get the report associated with this descriptor

Parameters:

  • phrases_separately (bool) – Whether to include per-phrase information

  • hide_empty (bool) – Whether to hide empty cases

  • examples_strictness (Optional[Strictness.STRICTEST]) – What level of strictness to show for examples. Set to None to disable examples. Defaults to Strictness.STRICTEST.

  • strictness (Strictness) – The strictness of the success / fail overview. Defaults to Strictness.NORMAL.

  • phrase_max_len (int) – The maximum length of the phrase in examples. Defaults to 80.

Returns:

str – The report string

Return type:

str

dict(**kwargs)
Return type:

dict

exception medcat.utils.regression.results.MalformedFinding(*args)

Bases: ValueError

Inappropriate argument value (of correct type).

Parameters:

args (object) –

__init__(*args)

Initialize self. See help(type(self)) for accurate signature.

Parameters:

args (object) –

Return type:

None