Skip to contents

edstr_extract(): text extraction

Source: vignettes/extract.qmd

edstr_extract() is the core of the pipeline. It tokenises cleaned text, matches user-defined concepts via regex, filters false positives, and exports results as XLSX, JSON, and RDS files.

Prerequisites 

edstr_config(
  edstr_dirname = "output/my_study",
  edstr_filename = "my_study",
  edstr_text = "note_text",
  edstr_overwrite = FALSE
)

df_import <- edstr_import()
df_clean <- edstr_clean(data = df_import, replace = c("\\s+" = " "))

Defining concepts

Concepts are named regex patterns that define the clinical entities to search for. A named character vector creates flat, independent concepts:

result <- edstr_extract(
  data = df_clean,
  concepts = c(fracture = "fracture", femur = "femur|fesf"),
  group = "id_pat"
)

A nested named list creates sub-concepts grouped under a root:

result <- edstr_extract(
  data = df_clean,
  concepts = list(
    fracture = list(
      fesf = "fesf|extremite superieure",
      col = "col (du )?femur"
    )
  ),
  group = "id_pat"
)

Tokenisation

The ngram_max argument sets the largest n-gram size tokenised; every smaller size is searched too. Unigrams (ngram_max = 1) are the default; raising it to 2 adds bigrams, which capture multi-word expressions. Do not confuse it with the ngrams argument of edstr_view(), which widens the window displayed around a match instead of the sizes searched.

result <- edstr_extract(
  data = df_clean,
  concepts = c(fracture = "fracture"),
  ngram_max = 2,
  group = "id_pat"
)

By default, starts_with_only = TRUE appends \S*$ to patterns, matching any token that starts with the concept. Set starts_with_only = FALSE for exact matching.

Collapse and intersect

When multiple concepts are defined, two modes alter the matching logic:

  • collapse = TRUE: OR-combines patterns into a single regex, one per root concept as soon as at least one root holds several patterns, otherwise one regex named concepts for the whole set, which drops the root names. Useful when sub-concepts are synonyms that should be treated as one.
  • intersect = TRUE: keeps only documents that match all root-level concepts. Useful for narrowing results to co-occurrences. A document matching some roots but not all becomes a non-case, counted in unmatched$no_concept: the denominator you read is therefore the one for the compound concept, not for any single root. data$match stays pre-intersect, so anti_join(data$match, data$extract, by = id) recovers the partial matches.

Both require at least two concepts.

result <- edstr_extract(
  data = df_clean,
  concepts = c(fracture = "fracture", femur = "femur|fesf"),
  intersect = TRUE,
  group = "id_pat"
)

Exclusions

False positives can be removed through two mechanisms:

Manual exclusions

A regex pattern passed to exclus_manual removes any matched token containing that pattern.

result <- edstr_extract(
  data = df_clean,
  concepts = c(fracture = "fracture"),
  exclus_manual = "fracture ouverte|ancienne fracture",
  group = "id_pat"
)

Automatic exclusions

Heuristic-based filtering on long tokens. exclus_auto_token_min sets the n-gram size above which auto-exclusion applies, in the same unit as ngram_max. Its default of 10 is above any realistic n-gram size, so out of the box no token clears the threshold and the heuristic is skipped: set it to 0 to submit every n-gram size to the scan. exclus_auto_escape removes specific tokens from the match pool before auto-exclusion runs.

Sampling

For development and testing, sample draws a random subset of rows before extraction. Use seed for reproducibility.

result <- edstr_extract(
  data = df_clean,
  concepts = c(fracture = "fracture"),
  sample = 500,
  seed = 42,
  group = "id_pat"
)

Pseudonymisation

Two arguments handle de-identification before extraction:

  • ano_hash: column names whose values are replaced by a 16-character hash.
  • ano_hide: column names whose values are masked with "---".

Both take regex patterns matched case-insensitively against column names, and both apply to the input frame before anything derives from it, so every output carries the transformed values: data$base, data$match, data$extract, data$note, the Excel sheets and the gt tables.

They abort rather than act silently when a pattern matches no column, matches the id or group column, or matches the text column.

Two limits are worth stating plainly. The hash is unsalted and stable across runs, which is what makes a patient traceable between two extractions; it also means a small identifier space can be reversed by enumerating it. This is pseudonymisation, not anonymisation. And neither argument touches the clinical text itself: data$extract, data$note and the highlighted Excel output carry the source unredacted, since showing that text is what they are for.

result <- edstr_extract(
  data = df_clean,
  concepts = c(fracture = "fracture"),
  ano_hash = "ipp",
  ano_hide = "patient_name",
  group = "id_pat"
)

Unmatched documents

Documents that produce no token match are reported separately, split by reason:

  • no_concept: the text was searched and matched no concept. A document whose every match is ruled out by the exclusions belongs here too, those matches having been judged false positives. Under intersect = TRUE the unit is the compound concept, so a document matching some roots but not all lands here as well, even though it did match something.
  • no_source: the source is empty or missing, so there was nothing to search.
  • empty_text: the source holds no text once its markup is stripped.
  • outside_p: the document has text, but it falls outside the <p> blocks the formatter reads.

Only no_concept is a result: it is the set of true negatives, the one to use as a denominator. The other three record documents the pipeline could not evaluate, and keeping them apart is what makes that denominator trustworthy.

The extraction summary always reports how many documents produced no matchable text, splitting the no_source, empty_text and outside_p counts. Those three sets are always populated in the output. The no_concept set can be large, so its rows are materialised only when unmatched_data = TRUE. Its count, unmatched$n_no_concept, is always exact regardless: read it for the denominator, never nrow(unmatched$no_concept), which is 0 both when the set is genuinely empty and when it is merely unmaterialised.

result <- edstr_extract(
  data = df_clean,
  concepts = c(fracture = "fracture"),
  unmatched_data = TRUE,
  group = "id_pat"
)

Output structure

edstr_extract() returns a nested list and saves three files:

File Contents
.xlsx Excel workbook with one sheet per result type: the extraction, the token stage (patterns, matches, counts, exclusions), concept counts, the source-matching stage (replacements, patterns, matches, counts), unmatched, mismatched, and the call parameters
.json Summary tables (by token, by concept) and the call parameters (JSON format)
.rds Full list with all intermediate objects (R-native, used for caching)

The returned list contains:

  • data: data frames (base, match, extract, note)
  • regex: parsed patterns, replacement rules, source-level matches
  • match: initial and final (post-exclusion) matches
  • count: token-level and distinct match counts
  • exclus: excluded matches and counts
  • unmatched: the true-negative count (n_no_concept, always exact) and documents with no token match, split by reason (no_concept, no_source, empty_text, outside_p)
  • mismatched: discrepancies between token and source matching
  • summary: summaries by token and concept, plus call parameters
  • sheets: data frames and optional gt tables for each Excel sheet