Extract structured variables from clinical text

Tokenize source text, match concept patterns (regex), apply exclusions, locate the matched tokens back in the source text, and save results as XLSX, JSON, and RDS files.

Usage

edstr_extract(
  data = NULL,
  text_input = getOption("edstr_text"),
  id = NULL,
  group = NULL,
  sample = NULL,
  seed = NULL,
  ano_hash = NULL,
  ano_hide = NULL,
  ngram_max = 1,
  concepts = NULL,
  collapse = FALSE,
  intersect = FALSE,
  starts_with_only = TRUE,
  exclus_manual = NULL,
  exclus_auto_escape = NULL,
  exclus_auto_token_min = 10,
  regex_replace = NULL,
  unmatched_data = FALSE,
  concept_color = "#0099FF",
  text_color = "#FF0000",
  save_as_gt = FALSE,
  chunk = NULL,
  chunk_sort = NULL,
  dirname_suffix = if (!is.null(sample)) str_glue("sample_{sample}") else NULL,
  filename_suffix = dirname_suffix
)

Arguments

data

<data.frame> Input data containing at least a text column and a unique identifier column. Required unless chunk is set, which reads the Parquet cache edstr_clean() wrote instead.

text_input

<character(1)> Name of the text column to analyse. Defaults to getOption("edstr_text") set by edstr_config().

id

<character(1)> Name of the unique identifier column. If not supplied, detected automatically: the one column with no duplicates and no NA, aborting if none or several qualify.

group

<character(1)> Optional grouping column (e.g. patient ID when rows are documents). If NULL, a sequential id_group is created.

sample

<integer(1)> Optional. Number of rows to randomly sample from data before extraction.

seed

<integer(1)> Optional. Random seed for reproducibility when sample is used.

ano_hash

<character> Regex pattern(s) matched case-insensitively against column names; every matching column is pseudonymised by hashing. Applied to data before extraction, so every output frame (data$base, data$match, data$extract, data$note, sheets, and the XLSX and gt tables built from them) carries the hashed values. The hash is unsalted and truncated to 16 hexadecimal characters: it is a pseudonym, stable across runs and therefore reversible by dictionary attack on a small domain such as a patient identifier. It is not anonymisation.

ano_hide

<character> Regex pattern(s) matched case-insensitively against column names; every matching column is masked (replaced with "---"). Same application point and coverage as ano_hash. A column matched by both arguments is hashed first, then masked, so "---" is what reaches the outputs.

Both arguments abort rather than act silently when a pattern matches no column at all, matches the id or group column (which join the outputs back together), or matches text_input. Neither argument touches the source text: data$extract, data$note and the highlighted XLSX and gt output carry text_input unredacted by design. Neither touches the Parquet caches written upstream by edstr_import() and edstr_clean(), which keep the source in clear.

ngram_max

<integer(1)> Largest n-gram size to tokenise. Every smaller size is searched too, so ngram_max = 2 covers unigrams and bigrams. Default 1 (unigrams only). Distinct from the ngrams argument of edstr_view(), which sets a display window around a match rather than the sizes searched.

concepts

<character|list> Named vector or nested named list of regex patterns defining the concepts to search for. Each name becomes a concept key; nested names create sub-concepts (e.g. list(cancer = list(sein = "sein|mammaire", poumon = "poumon"))). Required.

collapse

<logical(1)> If TRUE, OR-combine concept patterns into a single regex: one per root concept as soon as at least one root holds several patterns, otherwise one regex named concepts for the whole set, which drops the root names. Nesting alone does not decide it: a nested list whose roots each hold a single pattern takes the second branch. Requires at least 2 concepts.

intersect

<logical(1)> If TRUE, keep only documents matching ALL root-level concepts. Requires at least 2 concepts.

starts_with_only

<logical(1)> If TRUE (default), token matching uses prefix mode: the pattern must match the start of a token, and the rest of the token is accepted (\\S*$ appended).

exclus_manual

<character(1)> Optional regex pattern. Matched tokens containing this pattern are excluded (manual false-positive filter).

exclus_auto_escape

<character(1)> Optional regex pattern. Tokens matching this pattern are removed from data_match before auto-exclusion runs.

exclus_auto_token_min

<numeric(1)> Minimum n-gram size for automatic exclusion heuristics (default 10). Auto-exclusions only apply to tokens with n > exclus_auto_token_min, measured in the same unit as ngram_max. The default therefore disables the heuristic for every realistic ngram_max value: with ngram_max = 3 no token exceeds 10. Set it to 0 to submit every n-gram size to the scan. When no n-gram clears the threshold the scan has nothing to look at and is skipped, so the default costs nothing.

regex_replace

<character> Optional named vector of additional regex replacements for source matching (appended to the built-in accent normalisation rules). Every element must be named, the name being the pattern its value replaces, and the value reaches the source pattern as the regex fragment it is written as. Each rule matches the token with the earlier replacements already parked, so a rule whose pattern contains a character a built-in covers (oe, ae, a, e, i, o, u, or whitespace) never fires rather than widening it: "oe" is inert, "y" is not. A rule whose pattern reaches any character (., \\W, a negated class) also reaches that parking and is rejected. Backreferences in a replacement are not supported.

unmatched_data

<logical(1)> If TRUE, materialise the row-level unmatched$no_concept set (documents whose text was searched and matched no concept, or no longer match the intersection when intersect = TRUE), which can be large. Only the rows are gated: the count unmatched$n_no_concept is always exact regardless. The no_source, empty_text and outside_p sets are always populated too. Default FALSE.

concept_color

<character(1)> Hex colour for concept highlighting in XLSX and gt output. Default "#0099FF".

text_color

<character(1)> Hex colour for text/extract highlighting in XLSX and gt output. Default "#FF0000".

save_as_gt

<logical(1)> If TRUE, generate gt::gt() tables alongside XLSX output. Requires the gt package.

chunk

<integer(1)|logical(1)> Optional. Run the extraction in chunks of about chunk documents, each in its own subprocess, and combine the results into one output indistinguishable from a single-pass run. TRUE takes the default of 15000, NULL (default) and FALSE extract in one pass. A chunk is a contiguous range of group, never a split one, so the budget is a soft bound: a group straddling a boundary goes whole into the later chunk, and that chunk exceeds the budget by at most the straddling group's document count. A group larger than the budget itself forms a chunk of its own, which no budget bounds, and warns. Requires the callr, DBI and duckdb packages.

The budget is a document count because the memory a chunk needs is set by the text volume it carries, and hospital report length varies far less than the factor-2 margin 15000 was chosen with: measured over three EDS corpora, about 20,600 characters per document within 25%. Re-derive it (mean(nchar(text)) on the Parquet cache) only past roughly 30,000 characters per document, on another machine, or on documents that are not hospital reports.

chunk_sort

<character> Optional. Column names to order data$extract and data$note by, after group and before id, once the metadata is joined back. Inactive by default, which leaves the group, id order a single-pass run produces. Only used when chunk is set.

dirname_suffix

<character(1)> Optional suffix appended to the extract/ subdirectory, not to edstr_dirname (e.g. extract_sample_500). Defaults to "sample_{sample}" when sample is set.

filename_suffix

<character(1)> Optional suffix appended to output file names. Defaults to dirname_suffix.

Value

Invisibly, a nested list with elements:

data

List of data frames: base (input without text), match (initial matches), extract (final extraction, carrying one 0/1 column per declared concept key, in that order, whether or not the concept matched anything), note (extraction with highlight markup applied).

regex

List: concepts (parsed patterns), replace (replacement rules), final (combined regex), match (source-level matches).

match

List: init (all matches), final (keep/drop after exclusions).

count

List: init (token-level counts), final (distinct match counts).

exclus

List: match (excluded matches), count (exclusion counts).

unmatched

List for documents with no token match. n_no_concept <integer(1)> is the true-negative count, always exact regardless of unmatched_data. Then id/group tibbles: no_concept (text searched, no concept matched; rows materialised only when unmatched_data = TRUE, otherwise a 0-row tibble while n_no_concept still holds the count), no_source (source empty or NA, so never searched), empty_text (source holding no text once markup is stripped), outside_p (text outside <p> blocks). Read n_no_concept for the denominator, never nrow(no_concept). A document whose every match is excluded counts as a non-case in no_concept too, the exclusions having ruled its matches false positives, and so does one whose every match fails source confirmation. Under intersect = TRUE the unit is the compound concept, so a document whose token matches do not cover every root counts as a non-case as well; the intersection is evaluated on those token matches, before the exclusions, so a document that loses a whole root to an exclusion is still delivered, carrying the roots that survived. data$match stays pre-intersect and pre-exclusion, and anti_join(data$match, data$extract, by = id) recovers those sets.

mismatched

Tibble of token vs source discrepancies (token matches not confirmed in the source text). One row per match, not per document, so a document whose matches are only partly confirmed appears here and in data$extract at once.

summary

List: token (summary by token), concept (summary by concept), params (call parameters).

sheets

List: df (data frames per Excel sheet), gt (gt tables if save_as_gt = TRUE).

Details

Requires edstr_config() to be called first.

Single-pass and chunked mode

Without chunk, the extraction runs in one pass over data.

With chunk, it runs over the Parquet cache edstr_clean() wrote, one subprocess per chunk, and the results are combined into a single output: the summary, the three saved files and every frame of the return value are what a single-pass run over the whole corpus produces, and the per-chunk directories are removed once the combine succeeds. A chunked run that is interrupted resumes from the chunks already on disk unless edstr_overwrite is TRUE, and a chunk left over from a call with other arguments aborts the combine rather than being reused.

Four arguments change meaning between the two modes:

• data is required without chunk and rejected with it.

• id and group are optional without chunk and required with it: a chunk is a range of group, and each subprocess receives id, group and the text column alone, so neither key can be detected from that projection. Every other column is joined back after the extraction, which is what keeps a chunk's memory proportional to its text.

• chunk_sort is only read with chunk.

And four are rejected with chunk: sample and seed, which would draw within each chunk rather than over the corpus, and ano_hash and ano_hide, whose target columns no subprocess ever sees.

exclus_manual, exclus_auto_escape and exclus_auto_token_min apply inside each subprocess, so an exclusion that fires is evaluated per chunk and not over the corpus.

Examples

# \donttest{
edstr_config(
  edstr_dirname = tempdir(), edstr_filename = "my_study",
  edstr_text = "note_text", edstr_overwrite = TRUE
)
#> 
#> ── edstr_config ────────────────────────────────────────────────────────────────
#> 
#> ℹ Root : /home/runner/work/edstr/edstr
#> 
#> ℹ Files will be saved in /tmp/RtmpQPktpQ with prefix my_study
#> 
#> ────────────────────────────────────────────────────────────────────────────────

df_clean <- data.frame(
  id = 1:3,
  note_text = c(
    "<p class=\"n\">diabete de type 2</p>",
    "<p class=\"n\">bilan normal</p>",
    "<p class=\"n\">tumeur mammaire</p>"
  )
)

result <- edstr_extract(
  data = df_clean,
  concepts = c(diabete = "diabet", cancer = "cancer|tumeur")
)
#> 
#> ── edstr_extract ───────────────────────────────────────────────────────────────
#> Formatting source text
#> 
#> Formatting source text

#> ✔ Formatting source text [10ms]
#> 
#> Tokenising source text
#> 
#> Tokenising source text

#> ✔ Tokenising source text [77ms]
#> 
#> Matching tokenised text
#> 
#> Matching tokenised text

#> ✔ Matching tokenised text [71ms]
#> 
#> Exclusions
#> 
#> Exclusions

#> ✔ Exclusions [95ms]
#> 
#> Matching source text
#> 
#> Matching source text

#> ✔ Matching source text [80ms]
#> 
#> Unmatched documents and source mismatches
#> 
#> Unmatched documents and source mismatches

#> ✔ Unmatched documents and source mismatches [29ms]
#> 
#> Extraction
#> 
#> Extraction

#> ✔ Extraction [45ms]
#> 
#> Summary
#> 
#> Summary

#> ✔ Summary [100ms]
#> 
#> Saving file my_study_extract.xlsx
#> 
#> Saving file my_study_extract.xlsx

#> ✔ Saving file my_study_extract.xlsx [2.8s]
#> 
#> Saving file my_study_extract.json
#> 
#> Saving file my_study_extract.json

#> ✔ Saving file my_study_extract.json [53ms]
#> 
#> Saving file my_study_extract.rds
#> 
#> Saving file my_study_extract.rds

#> ✔ Saving file my_study_extract.rds [26ms]
#> 
#> 
#> ────────────────────────────────────────────────────────────────────────────────
#> 
#> $token
#> $token$n1
#>   concept note_text match
#> 1  cancer    tumeur     1
#> 2 diabete   diabete     1
#> 
#> 
#> $count
#> $count$total
#>   token concept note_text match id id_group
#> 1     1  cancer    tumeur     1  1        1
#> 2     1 diabete   diabete     1  1        1
#> 
#> $count$token
#>   token match_total match_exclus_auto match_exclus_manual match_final
#> 1     1           2                NA                  NA           2
#>   match_distinct
#> 1              2
#> 
#> $count$concepts
#>   concept match_total match_exclus_auto match_exclus_manual match_final
#> 1  cancer           1                NA                  NA           1
#> 2 diabete           1                NA                  NA           1
#>   match_distinct id id_group
#> 1              1  1        1
#> 2              1  1        1
#> 
#> $count$exclus
#>  [1] mode        concept_key token       concept     note_text   start      
#>  [7] end         start_end   match       id          id_group   
#> <0 rows> (or 0-length row.names)
#> 
#> $count$final
#>   token concept note_text match id id_group
#> 1     1  cancer    tumeur     1  1        1
#> 2     1 diabete   diabete     1  1        1
#> 
#> 
#> $regex
#> $regex$concepts
#> # A tibble: 2 × 3
#>   concept_key concept_name regex                  
#>   <chr>       <chr>        <chr>                  
#> 1 diabete     diabete      "^(diabet)\\S*$"       
#> 2 cancer      cancer       "^(cancer|tumeur)\\S*$"
#> 
#> $regex$replace
#> # A tibble: 8 × 2
#>   pattern replace                         
#>   <chr>   <chr>                           
#> 1 "oe"    "(?:[oô][eéèêë]|œ)"             
#> 2 "ae"    "(?:[aàâ][eéèêë]|æ)"            
#> 3 "a"     "[aàâ]"                         
#> 4 "e"     "[eéèêë]"                       
#> 5 "i"     "[iîï]"                         
#> 6 "o"     "[oô]"                          
#> 7 "u"     "[uùûü]"                        
#> 8 "\\s"   "(?:<br/>)?[\\s\\-']+(?:<br/>)?"
#> 
#> $regex$final
#> # A tibble: 2 × 2
#>   concept note_text                            
#>   <chr>   <glue>                               
#> 1 cancer  (?i)\b(t[uùûü]m[eéèêë][uùûü]r)\b     
#> 2 diabete (?i)\b(d[iîï][aàâ]b[eéèêë]t[eéèêë])\b
#> 
#> $regex$match
#> # A tibble: 2 × 3
#>      id concept match  
#>   <int> <chr>   <chr>  
#> 1     3 cancer  tumeur 
#> 2     1 diabete diabete
#> 
#> 
#> $unmatched
#> $unmatched$n_no_concept
#> [1] 1
#> 
#> $unmatched$no_concept
#> [1] id       id_group
#> <0 rows> (or 0-length row.names)
#> 
#> $unmatched$no_source
#> [1] id       id_group
#> <0 rows> (or 0-length row.names)
#> 
#> $unmatched$empty_text
#> [1] id       id_group
#> <0 rows> (or 0-length row.names)
#> 
#> $unmatched$outside_p
#> [1] id       id_group
#> <0 rows> (or 0-length row.names)
#> 
#> 
#> $mismatched
#> [1] id      concept match  
#> <0 rows> (or 0-length row.names)
#> 
#> $params
#> # A tibble: 18 × 2
#>    arg                   value           
#>    <chr>                 <chr>           
#>  1 sample                NULL            
#>  2 seed                  NULL            
#>  3 id                    id              
#>  4 group                 NULL            
#>  5 ngram_max             1               
#>  6 concepts_root         diabete ; cancer
#>  7 concepts_names        diabete ; cancer
#>  8 collapse              FALSE           
#>  9 intersect             FALSE           
#> 10 starts_with_only      TRUE            
#> 11 exclus_manual         NULL            
#> 12 exclus_auto_escape    NULL            
#> 13 exclus_auto_token_min 10              
#> 14 regex_replace         NULL            
#> 15 unmatched_data        FALSE           
#> 16 concept_color         #0099FF         
#> 17 text_color            #FF0000         
#> 18 save_as_gt            FALSE           
#> 
#> ────────────────────────────────────────────────────────────────────────────────
#> 
#> 
#> ── Timing per step ──
#> 
#> Parse concepts: 0.004 sec elapsed
#> Check ids: 0.006 sec elapsed
#> Anonymise: 0 sec elapsed
#> Format text: 0.002 sec elapsed
#> Tokenise: 0.06 sec elapsed
#> Match token: 0.053 sec elapsed
#> Exclusions: 0.076 sec elapsed
#> Match source: 0.055 sec elapsed
#> Unmatched: 0.01 sec elapsed
#> Extraction: 0.028 sec elapsed
#> Summary: 0.073 sec elapsed
#> Build sheets: 0.008 sec elapsed
#> Build gt: 0 sec elapsed
#> Build xlsx: 2.568 sec elapsed
#> Save xlsx: 0.257 sec elapsed
#> Save json: 0.032 sec elapsed
#> Note markup: 0.005 sec elapsed
#> Save rds: 0.001 sec elapsed
#> 
#> Full steps: 3.497 sec elapsed
#> 
#> ℹ Documents
#> • Total: 3 id
#> 
#> ℹ Total matches
#> • 2 across 2 id (66.7% id)
#> • No exclusions
#> 
#> ℹ Unmatched (no concept): 1 id
#> 
#> ────────────────────────────────────────────────────────────────────────────────
#> 
#> ✔ 2 root concepts: diabete, cancer
#> ✔ 2 matches
#> • 2 id (66.7% id)
#> 
#> ✔ 1 unmatched (no concept)
#> • 1 id (33.3% id)
#> 
#> ✔ 2 distinct matches
#> 
#> ────────────────────────────────────────────────────────────────────────────────
#> 
#> ℹ Root directory: /home/runner/work/edstr/edstr
#> ✔ Files saved in /tmp/RtmpQPktpQ/extract
#> 
#> ────────────────────────────────────────────────────────────────────────────────
# }