Versions Compared

Old Version 36

changes.mady.by.user Andrew McMurry

Saved on

compared with

New Version 37

changes.mady.by.user Andrew McMurry

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Info

Note : Scrubber 3.X is being ported to Apache cTAKES, this is an interim BETA release.

Table of Contents



Intended usages

Info

We recommend starting with the default properties and prebuilt train/test models.
The train and test models are anonymized feature sets (not text) generated by scrubber runtime.


Default configuration

Developers can use Scrubber 3.X in "default mode" with the same settings as the provided train and test model files. Input and output settings are managed in scrubber.properties (file paths, database settings, method implementations).

Info

scrubber.properties : all supported config options and features in one place.
Apache UIMA, Apache cTAKES, and WEKA distribution jars are loaded dynamically.



Customize NLP pipeline

Scrubber uses Apache UIMA and Apache cTAKES packages, which together provide the NLP pipeline for lexical parsing and medical concept annotation. Generated feature sets are exported to the SQL database or model file (CSV, ARFF). The UIMA and cTAKES services used by Scrubber are defined and configured using scrubber.properties.

Customize Classifier

Scrubber can use different classifier implementations without recompiling the software.
By default scrubber dynamically loads the popular WEKA C4.5 decision tree classifier with multi-class support.



Software Features



Annotation

  • Annotate word tokens and redact PHI from physician notes
  • cTAKES lexical parsing and medical dictionary annotation
  • WEKA multi-class decision tree classifier (plugin default)
  • Protege UI support for human expert curators (reads output) 
  • Generate feature sets containing lexical properties, medical concept codes, and human defined rules

    Models

  • Prebuilt train and test models can be imported to Weka (default), Matlab, or R
  • (default) Test your local physician notes without retraining
  • (optional) Retrain model using local physician note samples, publications, and medical dictionaries.  

    Classification

  • Distinguish (classify) private patient data from coded medical concepts and commonly used words 

    Compare Text

    Compare lexical properties and distributions of public and private text sources

    How To


    Install / Train / Test / Scrub


    Office Word
    namescrubber-3.x-runtime-guide.doc
    Scrubber Property KEY = VALUE

Anchor
properties
properties


scrubber.properties



Java Object

ScrubberProperties.java statically binds scrubber.properties at startup

Java Template

...

TemplateFileProcessor.java IO and token replacement of default configuration files 

Shell scripts

setClassPath.sh sets the java classpath and exports the shell variables

Shell UnitTest

...

ScrubberPropertiesTest.java demonstrates binding scrubber.properties to shell commands.