Skip to contents

Extract structured variables from clinical text

Source: R/edstr_extract.R
edstr_extract.Rd

Tokenize source text, match concept patterns (regex), apply exclusions, locate the matched tokens back in the source text, and save results as XLSX, JSON, and RDS files.

Usage

edstr_extract(
  data,
  text_input = getOption("edstr_text"),
  id = NULL,
  group = NULL,
  sample = NULL,
  seed = NULL,
  ano_hash = NULL,
  ano_hide = NULL,
  ngram_max = 1,
  concepts = NULL,
  collapse = FALSE,
  intersect = FALSE,
  starts_with_only = TRUE,
  exclus_manual = NULL,
  exclus_auto_escape = NULL,
  exclus_auto_token_min = 10,
  regex_replace = NULL,
  unmatched_data = FALSE,
  concept_color = "#0099FF",
  text_color = "#FF0000",
  save_as_gt = FALSE,
  dirname_suffix = if (!is.null(sample)) str_glue("sample_{sample}") else NULL,
  filename_suffix = dirname_suffix
)

Arguments

data

<data.frame> Input data containing at least a text column and a unique identifier column.

text_input

<character(1)> Name of the text column to analyse. Defaults to getOption("edstr_text") set by edstr_config().

id

<character(1)> Name of the unique identifier column. If not supplied, detected automatically: the one column with no duplicates and no NA, aborting if none or several qualify.

group

<character(1)> Optional grouping column (e.g. patient ID when rows are documents). If NULL, a sequential id_group is created.

sample

<integer(1)> Optional. Number of rows to randomly sample from data before extraction.

seed

<integer(1)> Optional. Random seed for reproducibility when sample is used.

ano_hash

<character> Regex pattern(s) matched case-insensitively against column names; every matching column is pseudonymised by hashing. Applied to data before extraction, so every output frame (data$base, data$match, data$extract, data$note, sheets, and the XLSX and gt tables built from them) carries the hashed values. The hash is unsalted and truncated to 16 hexadecimal characters: it is a pseudonym, stable across runs and therefore reversible by dictionary attack on a small domain such as a patient identifier. It is not anonymisation.

ano_hide

<character> Regex pattern(s) matched case-insensitively against column names; every matching column is masked (replaced with "---"). Same application point and coverage as ano_hash. A column matched by both arguments is hashed first, then masked, so "---" is what reaches the outputs.

Both arguments abort rather than act silently when a pattern matches no column at all, matches the id or group column (which join the outputs back together), or matches text_input. Neither argument touches the source text: data$extract, data$note and the highlighted XLSX and gt output carry text_input unredacted by design. Neither touches the Parquet caches written upstream by edstr_import() and edstr_clean(), which keep the source in clear.

ngram_max

<integer(1)> Largest n-gram size to tokenise. Every smaller size is searched too, so ngram_max = 2 covers unigrams and bigrams. Default 1 (unigrams only). Distinct from the ngrams argument of edstr_view(), which sets a display window around a match rather than the sizes searched.

concepts

<character|list> Named vector or nested named list of regex patterns defining the concepts to search for. Each name becomes a concept key; nested names create sub-concepts (e.g. list(cancer = list(sein = "sein|mammaire", poumon = "poumon"))). Required.

collapse

<logical(1)> If TRUE, OR-combine concept patterns into a single regex: one per root concept as soon as at least one root holds several patterns, otherwise one regex named concepts for the whole set, which drops the root names. Nesting alone does not decide it: a nested list whose roots each hold a single pattern takes the second branch. Requires at least 2 concepts.

intersect

<logical(1)> If TRUE, keep only documents matching ALL root-level concepts. Requires at least 2 concepts.

starts_with_only

<logical(1)> If TRUE (default), token matching uses prefix mode: the pattern must match the start of a token, and the rest of the token is accepted (\\S*$ appended).

exclus_manual

<character(1)> Optional regex pattern. Matched tokens containing this pattern are excluded (manual false-positive filter).

exclus_auto_escape

<character(1)> Optional regex pattern. Tokens matching this pattern are removed from data_match before auto-exclusion runs.

exclus_auto_token_min

<numeric(1)> Minimum n-gram size for automatic exclusion heuristics (default 10). Auto-exclusions only apply to tokens with n > exclus_auto_token_min, measured in the same unit as ngram_max. The default therefore disables the heuristic for every realistic ngram_max value: with ngram_max = 3 no token exceeds 10. Set it to 0 to submit every n-gram size to the scan. When no n-gram clears the threshold the scan has nothing to look at and is skipped, so the default costs nothing.

regex_replace

<character> Optional named vector of additional regex replacements for source matching (appended to the built-in accent normalisation rules). Every element must be named, the name being the pattern its value replaces, and the value reaches the source pattern as the regex fragment it is written as. Each rule matches the token with the earlier replacements already parked, so a rule whose pattern contains a character a built-in covers (oe, ae, a, e, i, o, u, or whitespace) never fires rather than widening it: "oe" is inert, "y" is not. A rule whose pattern reaches any character (., \\W, a negated class) also reaches that parking and is rejected. Backreferences in a replacement are not supported.

unmatched_data

<logical(1)> If TRUE, materialise the row-level unmatched$no_concept set (documents whose text was searched and matched no concept, or no longer match the intersection when intersect = TRUE), which can be large. Only the rows are gated: the count unmatched$n_no_concept is always exact regardless. The no_source, empty_text and outside_p sets are always populated too. Default FALSE.

concept_color

<character(1)> Hex colour for concept highlighting in XLSX and gt output. Default "#0099FF".

text_color

<character(1)> Hex colour for text/extract highlighting in XLSX and gt output. Default "#FF0000".

save_as_gt

<logical(1)> If TRUE, generate gt::gt() tables alongside XLSX output. Requires the gt package.

dirname_suffix

<character(1)> Optional suffix appended to the extract/ subdirectory, not to edstr_dirname (e.g. extract_sample_500). Defaults to "sample_{sample}" when sample is set.

filename_suffix

<character(1)> Optional suffix appended to output file names. Defaults to dirname_suffix.

Value

Invisibly, a nested list with elements:

data

List of data frames: base (input without text), match (initial matches), extract (final extraction, carrying one 0/1 column per declared concept key, in that order, whether or not the concept matched anything), note (extraction with highlight markup applied).

regex

List: concepts (parsed patterns), replace (replacement rules), final (combined regex), match (source-level matches).

match

List: init (all matches), final (keep/drop after exclusions).

count

List: init (token-level counts), final (distinct match counts).

exclus

List: match (excluded matches), count (exclusion counts).

unmatched

List for documents with no token match. n_no_concept <integer(1)> is the true-negative count, always exact regardless of unmatched_data. Then id/group tibbles: no_concept (text searched, no concept matched; rows materialised only when unmatched_data = TRUE, otherwise a 0-row tibble while n_no_concept still holds the count), no_source (source empty or NA, so never searched), empty_text (source holding no text once markup is stripped), outside_p (text outside <p> blocks). Read n_no_concept for the denominator, never nrow(no_concept). A document whose every match is excluded counts as a non-case in no_concept too, the exclusions having ruled its matches false positives, and so does one whose every match fails source confirmation. Under intersect = TRUE the unit is the compound concept, so a document whose token matches do not cover every root counts as a non-case as well; the intersection is evaluated on those token matches, before the exclusions, so a document that loses a whole root to an exclusion is still delivered, carrying the roots that survived. data$match stays pre-intersect and pre-exclusion, and anti_join(data$match, data$extract, by = id) recovers those sets.

mismatched

Tibble of token vs source discrepancies (token matches not confirmed in the source text). One row per match, not per document, so a document whose matches are only partly confirmed appears here and in data$extract at once.

summary

List: token (summary by token), concept (summary by concept), params (call parameters).

sheets

List: df (data frames per Excel sheet), gt (gt tables if save_as_gt = TRUE).

Details

Requires edstr_config() to be called first.

Examples

# \donttest{
edstr_config(
  edstr_dirname = tempdir(), edstr_filename = "my_study",
  edstr_text = "note_text", edstr_overwrite = TRUE
)
#> 
#> ── edstr_config ────────────────────────────────────────────────────────────────
#> 
#> ℹ Root : /home/runner/work/edstr/edstr
#> 
#> ℹ Files will be saved in /tmp/RtmpbqjexN with prefix my_study
#> 
#> ────────────────────────────────────────────────────────────────────────────────

df_clean <- data.frame(
  id = 1:3,
  note_text = c(
    "<p class=\"n\">diabete de type 2</p>",
    "<p class=\"n\">bilan normal</p>",
    "<p class=\"n\">tumeur mammaire</p>"
  )
)

result <- edstr_extract(
  data = df_clean,
  concepts = c(diabete = "diabet", cancer = "cancer|tumeur")
)
#> 
#> ── edstr_extract ───────────────────────────────────────────────────────────────
#> Formatting source text
#> 
#> Formatting source text

#> ✔ Formatting source text [9ms]
#> 
#> Tokenising source text
#> 
#> Tokenising source text

#> ✔ Tokenising source text [22ms]
#> 
#> Matching tokenised text
#> 
#> Matching tokenised text

#> ✔ Matching tokenised text [67ms]
#> 
#> Exclusions
#> 
#> Exclusions

#> ✔ Exclusions [93ms]
#> 
#> Matching source text
#> 
#> Matching source text

#> ✔ Matching source text [66ms]
#> 
#> Unmatched documents and source mismatches
#> 
#> Unmatched documents and source mismatches

#> ✔ Unmatched documents and source mismatches [26ms]
#> 
#> Extraction
#> 
#> Extraction

#> ✔ Extraction [42ms]
#> 
#> Summary
#> 
#> Summary

#> ✔ Summary [123ms]
#> 
#> Saving file my_study_extract.xlsx
#> 
#> Saving file my_study_extract.xlsx

#> ✔ Saving file my_study_extract.xlsx [2.5s]
#> 
#> Saving file my_study_extract.json
#> 
#> Saving file my_study_extract.json

#> ✔ Saving file my_study_extract.json [49ms]
#> 
#> Saving file my_study_extract.rds
#> 
#> Saving file my_study_extract.rds

#> ✔ Saving file my_study_extract.rds [25ms]
#> 
#> 
#> ────────────────────────────────────────────────────────────────────────────────
#> 
#> $token
#> $token$n1
#>   concept note_text match
#> 1  cancer    tumeur     1
#> 2 diabete   diabete     1
#> 
#> 
#> $count
#> $count$total
#>   token concept note_text match id id_group
#> 1     1  cancer    tumeur     1  1        1
#> 2     1 diabete   diabete     1  1        1
#> 
#> $count$token
#>   token match_total match_exclus_auto match_exclus_manual match_final
#> 1     1           2                NA                  NA           2
#>   match_distinct
#> 1              2
#> 
#> $count$concepts
#>   concept match_total match_exclus_auto match_exclus_manual match_final
#> 1  cancer           1                NA                  NA           1
#> 2 diabete           1                NA                  NA           1
#>   match_distinct id id_group
#> 1              1  1        1
#> 2              1  1        1
#> 
#> $count$exclus
#>  [1] mode        concept_key token       concept     note_text   start      
#>  [7] end         start_end   match       id          id_group   
#> <0 rows> (or 0-length row.names)
#> 
#> $count$final
#>   token concept note_text match id id_group
#> 1     1  cancer    tumeur     1  1        1
#> 2     1 diabete   diabete     1  1        1
#> 
#> 
#> $regex
#> $regex$concepts
#> # A tibble: 2 × 3
#>   concept_key concept_name regex                  
#>   <chr>       <chr>        <chr>                  
#> 1 diabete     diabete      "^(diabet)\\S*$"       
#> 2 cancer      cancer       "^(cancer|tumeur)\\S*$"
#> 
#> $regex$replace
#> # A tibble: 8 × 2
#>   pattern replace                         
#>   <chr>   <chr>                           
#> 1 "oe"    "(?:[oô][eéèêë]|œ)"             
#> 2 "ae"    "(?:[aàâ][eéèêë]|æ)"            
#> 3 "a"     "[aàâ]"                         
#> 4 "e"     "[eéèêë]"                       
#> 5 "i"     "[iîï]"                         
#> 6 "o"     "[oô]"                          
#> 7 "u"     "[uùûü]"                        
#> 8 "\\s"   "(?:<br/>)?[\\s\\-']+(?:<br/>)?"
#> 
#> $regex$final
#> # A tibble: 2 × 2
#>   concept note_text                            
#>   <chr>   <glue>                               
#> 1 cancer  (?i)\b(t[uùûü]m[eéèêë][uùûü]r)\b     
#> 2 diabete (?i)\b(d[iîï][aàâ]b[eéèêë]t[eéèêë])\b
#> 
#> $regex$match
#> # A tibble: 2 × 3
#>      id concept match  
#>   <int> <chr>   <chr>  
#> 1     3 cancer  tumeur 
#> 2     1 diabete diabete
#> 
#> 
#> $unmatched
#> $unmatched$n_no_concept
#> [1] 1
#> 
#> $unmatched$no_concept
#> [1] id       id_group
#> <0 rows> (or 0-length row.names)
#> 
#> $unmatched$no_source
#> [1] id       id_group
#> <0 rows> (or 0-length row.names)
#> 
#> $unmatched$empty_text
#> [1] id       id_group
#> <0 rows> (or 0-length row.names)
#> 
#> $unmatched$outside_p
#> [1] id       id_group
#> <0 rows> (or 0-length row.names)
#> 
#> 
#> $mismatched
#> [1] id      concept match  
#> <0 rows> (or 0-length row.names)
#> 
#> $params
#> # A tibble: 18 × 2
#>    arg                   value           
#>    <chr>                 <chr>           
#>  1 sample                NULL            
#>  2 seed                  NULL            
#>  3 id                    id              
#>  4 group                 NULL            
#>  5 ngram_max             1               
#>  6 concepts_root         diabete ; cancer
#>  7 concepts_names        diabete ; cancer
#>  8 collapse              FALSE           
#>  9 intersect             FALSE           
#> 10 starts_with_only      TRUE            
#> 11 exclus_manual         NULL            
#> 12 exclus_auto_escape    NULL            
#> 13 exclus_auto_token_min 10              
#> 14 regex_replace         NULL            
#> 15 unmatched_data        FALSE           
#> 16 concept_color         #0099FF         
#> 17 text_color            #FF0000         
#> 18 save_as_gt            FALSE           
#> 
#> ────────────────────────────────────────────────────────────────────────────────
#> 
#> 
#> ── Timing per step ──
#> 
#> Parse concepts: 0.003 sec elapsed
#> Check ids: 0.007 sec elapsed
#> Anonymise: 0 sec elapsed
#> Format text: 0.002 sec elapsed
#> Tokenise: 0.005 sec elapsed
#> Match token: 0.05 sec elapsed
#> Exclusions: 0.076 sec elapsed
#> Match source: 0.049 sec elapsed
#> Unmatched: 0.009 sec elapsed
#> Extraction: 0.025 sec elapsed
#> Summary: 0.098 sec elapsed
#> Build sheets: 0.008 sec elapsed
#> Build gt: 0 sec elapsed
#> Build xlsx: 2.4 sec elapsed
#> Save xlsx: 0.059 sec elapsed
#> Save json: 0.031 sec elapsed
#> Note markup: 0.005 sec elapsed
#> Save rds: 0.001 sec elapsed
#> 
#> Full steps: 3.065 sec elapsed
#> 
#> ℹ Documents
#> • Total: 3 id
#> 
#> ℹ Total matches
#> • 2 across 2 id (66.7% id)
#> • No exclusions
#> 
#> ℹ Unmatched (no concept): 1 id
#> 
#> ────────────────────────────────────────────────────────────────────────────────
#> 
#> ✔ 2 root concepts: diabete, cancer
#> ✔ 2 matches
#> • 2 id (66.7% id)
#> 
#> ✔ 1 unmatched (no concept)
#> • 1 id (33.3% id)
#> 
#> ✔ 2 distinct matches
#> 
#> ────────────────────────────────────────────────────────────────────────────────
#> 
#> ℹ Root directory: /home/runner/work/edstr/edstr
#> ✔ Files saved in /tmp/RtmpbqjexN/extract
#> 
#> ────────────────────────────────────────────────────────────────────────────────
# }