Preprocessors

Preprocessors to remove DICOM masks and generate segmentations.

Prepare Dependencies

pip install opencv-python pydicom matplotlib pandas

Command Usage

usage: main.py [-h] --dcm-dir DCM_DIR --label-dir LABEL_DIR
--target-dir TARGET_DIR
--mode {inspector,mask,roi,spreadsheet,cvat,transform,classify,label}
[--filterable-csv-file FILTERABLE_CSV_FILE]
[--filterable-dataset-type {train,valid,test}]
[--filterable-keep-issues]
[--overwrite-label-type {cvat}]
[--overwrite-label-file OVERWRITE_LABEL_FILE]
[--new-shape NEW_SHAPE]
[--crop-image CROP_IMAGE]
[--jobs JOBS]
This is a preprocessor to remove DICOM masks and generate segmentations and
its inspectors, masks and ROIs.
optional arguments:
-h, --help show this help message and exit
--dcm-dir DCM_DIR The DICOM root directory
--label-dir LABEL_DIR
The JSON labels root directory
--target-dir TARGET_DIR
The destination root directory for outputs
--mode {inspector,mask,roi,spreadsheet,cvat}
inspector Generate four-in-one images to compare masks, overlay and noise-eliminated with original image
mask Generate binary masks that will be used as Dataset
for segmentation models
roi Generate region-of-interest images that will be used as Dataset for classification model
spreadsheet Generate CSV files that contains encrypted
patients identifiers and its file name
cvat Generate a XML file that contains segmentation mask polygons
to be uploaded on CVAT
transform Generate the original dataset images but necessarily transformed
classify Generate region-of-interest images that will be used
as Dataset for classification model, but sorts into phase labels
label Generate a CSV file that contains file name and its cancer phases
--filterable-csv-file FILTERABLE_CSV_FILE
The CSV file to be used for filtering broken datasets out
--filterable-dataset-type {train,valid,test}
The type of dataset source directory for querying filterable CSV file
--filterable-keep-issues
A flag to keep issued rows in filterable CSV file
--overwrite-label-type {cvat}
The type of overrideable labels format to parse
cvat CVAT 1.1 XML annotation format
Pass 'annotations.xml' file to --overwrite-label-file argument
--overwrite-label-file OVERWRITE_LABEL_FILE
The label file to be used for overwriting dataset labels
--new-shape NEW_SHAPE
WxH. Resize the output image with desired width and height - e.g.) 224x224
--crop-image CROP_IMAGE
X:Y,W:H. Crop the output image to desired rectangle - e.g.) 90:0,480:480
--jobs JOBS Number of workers

usage: assign.py [-h] --csv-file CSV_FILE --source-dirs SOURCE_DIRS
--target-dir TARGET_DIR
This is an assigner for assigning dataset validation job fairly.
optional arguments:
-h, --help show this help message and exit
--csv-file CSV_FILE The assignees CSV file
--source-dirs SOURCE_DIRS
dir1,dir2,dir3,.. The source directories to be assigned to
--target-dir TARGET_DIR
The destination directory where the assigned directory will be located

Dataset Hierarchy

Dataset:
- {LABEL_DIR}
- *
- ENDO
- *.json
- {DCM_DIR}
- *
- ENDO
- *.dcm

CSV Formats

Filterable CSV File

filterable-dataset-type,patient_id,image_id,assignee,issuetrain,00000001,00000001_0001,John,TRUEvalid,00000002,00000002_0001,James,FALSEtest,00000003,00000003_0001,Alice,FALSE

TRUE means the row have an issue, and the image will be truncated in the result.

Assignees CSV File

John,James,Alice,,00000001,00000002,0000000300000004,00000005,,00000006,

Proper assignees CSV file is required to separate the dataset fairly.

Result:
- John
- 00000001
- *.jpg
- 00000004
- *.jpg
- James
- 00000002
- *.jpg
- 00000005
- *.jpg
- 00000006
- *.jpg
- Alice
- 00000003
- *.jpg