From 3d514eecf32aea5c8519e13a8d5613f171bdf142 Mon Sep 17 00:00:00 2001 From: Baljit Singh Date: Sat, 1 Jul 2017 02:30:34 +0300 Subject: [PATCH 1/3] Updated root README, fixed errors in custom scripts --- README.md | 29 +++++++++++------ src/custom_scripts/convert_csv_to_json.py | 33 ++++++++++++++------ src/custom_scripts/image_augment.py | 2 +- src/custom_scripts/training_data.json | 1 + src/machine_learning/README.md | 38 ----------------------- src/webapp/README.md | 38 ----------------------- 6 files changed, 44 insertions(+), 97 deletions(-) create mode 100644 src/custom_scripts/training_data.json delete mode 100644 src/machine_learning/README.md delete mode 100644 src/webapp/README.md diff --git a/README.md b/README.md index b264935..fee6fb1 100644 --- a/README.md +++ b/README.md @@ -28,6 +28,7 @@ The following pre-requisites need to be satisified for the project to run: * Python 2.7 * Django 1.11.2 * pip +* Pillow 4.1.1 Following commands can be used to install Django on both macOS and Unix machines: ``` @@ -60,7 +61,7 @@ After downloading the dataset csv file from the webapp; use the script `convert_ ## Description Once we have the training data and the validation data, use *TensorBox* to train a machine learning model. A trained model has -already been provided with the name of _save.ckpt-180000_ +already been provided with the name of _save.ckpt-18000_ ## Pre-requisites The following pre-requisites need to be satisified for the machine learning project to run: @@ -87,15 +88,19 @@ To setup TensorBox and evaluate the model, follow instructions below: mv ./src/machine_learning/data/* ./tensorbox/data/ mv ./src/machine_learning/config/* ./tensorbox/hypes/ cd tensorbox - python evaluate.py --weights data/save.ckpt-180000 --test_boxes data/testing_set.json - ``` + python evaluate.py --weights data/save.ckpt-18000 --test_boxes data/testing_set.json + + +The evaluated image results are saved under `data/images_testing_set_18000`. + Below is a solution for a potential error that might come up during TensorBox installation. The commands should be executed inside the `virtualenv`: -2. **Error:** pkg-config python not found +1. **Error:** pkg-config python not found + **Solution for macOS:** export PKG_CONFIG_PATH=/System/Library/Frameworks/Python.framework/Versions/2.7/lib/pkgconfig/; echo $(pkg-config --variable pc_path pkg-config)${PKG_CONFIG_PATH:+:}${PKG_CONFIG_PATH} -_Note: In order to use a custom training dataset, the files should be placed under the folder 'tensorbox/data' and modify the fields training and testing fields in `tensorbox/hypes/overfeat_rezoom.js`_ +_Note: In order to use a custom training dataset, the training and validation files should be placed under 'tensorbox/data' and modify the fields training and testing fields in `tensorbox/hypes/overfeat_rezoom.js`_ ## The Dataset The images used for this project have all been taken from the web. The images are of human skulls, non human skulls and various other objects such as people, cartoon characters, etc. After the completion of data collection, every image was passed through @@ -107,7 +112,9 @@ the web app developed in the first phase. The image metadata is then saved in a ## Results -Full testing output fromm the trainined model can be found under `results/`. In the images, the green boxes are the final predictions after merging, and red are all predictions after applying a threshold of confidence, but before merging. +Full testing output from the trained model can be found under `results/`. In the images, the green boxes are the final predictions after merging, and red are all predictions after applying a threshold of confidence, but before merging. + +For the test dataset used, average accuracy was 90%. #### Positive Images Below are some positive images that were classified correctly. Most of the positively classified images, as below, are images that contain a standalone skull whereas some of them also contain noise in the background. @@ -118,7 +125,7 @@ Below are some positive images that were classified correctly. Most of the posit Most of the negative images were classified correctly, however as we can see below, some of the negative images that were positively classified contain backgorunds, skeletons, faces or facial-like features such as eyes. The trained model drew bounding boxes around random objects in addition to the human skull (if present). -For the test dataset used, average accuracy was 90% + ## Discussion @@ -159,6 +166,8 @@ Couple of points: 1. Use ResNet. TensorBox contains ResNet that we could use for much deeper learning with more resources. 2. Training on a much larger data set that includes more negative class than positives and images with different resolutions. -3. Use more variations of image augmentations such as shearing, rotation to increase the accuracy. -4. For web app, use outlier detection to prevent spam in the training set -5. For web app, fix canvas drawing for large resolution images +3. Train the model for more epochs +4. Use more variations of image augmentations such as shearing, rotation to increase the accuracy. +5. For web app, use outlier detection to prevent spam in the training set +6. For web app, fix canvas drawing for large resolution images + diff --git a/src/custom_scripts/convert_csv_to_json.py b/src/custom_scripts/convert_csv_to_json.py index 8cf6782..49e4901 100644 --- a/src/custom_scripts/convert_csv_to_json.py +++ b/src/custom_scripts/convert_csv_to_json.py @@ -3,7 +3,7 @@ import json import ast import sys, getopt - +import math def parse(ifile, ofile): ''' @@ -107,9 +107,8 @@ def main(argv): # create a list of dictionaries for each line for line in csv_file: d = {} - print line line_split = line.split("\"") - + #print line imgfile_name = img_directory+"/"+line_split[0] d['image_path'] = imgfile_name[:-1] @@ -117,8 +116,18 @@ def main(argv): line_split[1] != "[]" line_split[1] = line_split[1][1:] line_split[1] = line_split[1][:-1] + temp_array = [] + try: + temp_array = ast.literal_eval(line_split[1]) + except SyntaxError: + nan_array = line_split[1].split(",") + for (_, arr_val) in enumerate(nan_array): + ele_len = len(arr_val) + if ele_len>0: + is_nan = math.isnan(float(arr_val)) + if not is_nan: + temp_array.append(float(arr_val)) - temp_array = ast.literal_eval(line_split[1]) #duck-typing var_type = isinstance(temp_array, (list, tuple)) @@ -126,13 +135,17 @@ def main(argv): rects_array = [] if var_type: - # list if there iis more than one rectangle - if type(temp_array) == type([]): - rects_array.append({'x1':temp_array[0], 'y1':temp_array[1], 'x2':temp_array[2], 'y2':temp_array[3]}) + print temp_array + # tuple if there is one rectangle - elif type(temp_array) == type(()): - for (index, _) in enumerate(temp_array): - coords = {'x1':temp_array[index][0], 'y1':temp_array[index][1], 'x2':temp_array[index][2], 'y2':temp_array[index][3]} + if type(temp_array) == type(()): + + rects_array.append({'x1':temp_array[0], 'y1':temp_array[1], 'x2':temp_array[2], 'y2':temp_array[3]}) + # list if there is more than one rectangle + elif type(temp_array) == type([]): + num_rects = len(temp_array)/4 + for index in range(0,num_rects): + coords = {'x1':temp_array[(4*index)+0], 'y1':temp_array[(4*index)+1], 'x2':temp_array[(4*index)+2], 'y2':temp_array[(4*index)+3]} rects_array.append(coords) d['rects'] = rects_array diff --git a/src/custom_scripts/image_augment.py b/src/custom_scripts/image_augment.py index 152e4ad..59a389d 100644 --- a/src/custom_scripts/image_augment.py +++ b/src/custom_scripts/image_augment.py @@ -183,7 +183,7 @@ def main(argv): img = transform_image(image,0,brightness=1) im = Image.fromarray(img) - #generate only 5 augmented versions with big variations in the image + #generate only 4 augmented versions with big variations in the image if i%5: im.save(output_img_directory+"/"+file_name+"_"+str(i)+"."+file_ext) image_new = file_name+"_"+str(i)+"."+file_ext diff --git a/src/custom_scripts/training_data.json b/src/custom_scripts/training_data.json new file mode 100644 index 0000000..407a622 --- /dev/null +++ b/src/custom_scripts/training_data.json @@ -0,0 +1 @@ +[{"rects": [], "image_path": "/Users/baljitsingh/Desktop/SidraFinal/HiringExercise_MLEngineer_Baljit92/tensorbox/data/images/16b1cfa12b64.jpg"}, {"rects": [], "image_path": "/Users/baljitsingh/Desktop/SidraFinal/HiringExercise_MLEngineer_Baljit92/tensorbox/data/images/1a.jpg"}, {"rects": [{"y1": 103, "x2": 259.5, "x1": 121.5, "y2": 184}], "image_path": "/Users/baljitsingh/Desktop/SidraFinal/HiringExercise_MLEngineer_Baljit92/tensorbox/data/images/2007_000027.jpg"}, {"rects": [{"y1": 97.0, "x2": 284.5, "x1": 167.5, "y2": 201.0}], "image_path": "/Users/baljitsingh/Desktop/SidraFinal/HiringExercise_MLEngineer_Baljit92/tensorbox/data/images/2007_000032.jpg"}, {"rects": [], "image_path": "/Users/baljitsingh/Desktop/SidraFinal/HiringExercise_MLEngineer_Baljit92/tensorbox/data/images/2007_000033.jpg"}, {"rects": [], "image_path": "/Users/baljitsingh/Desktop/SidraFinal/HiringExercise_MLEngineer_Baljit92/tensorbox/data/images/2007_000039.jpg"}, {"rects": [{"y1": 78.0, "x2": 472.5, "x1": 324.5, "y2": 270.0}, {"y1": 192.0, "x2": 237.5, "x1": 148.0, "y2": 148.0}, {"y1": 357.0, "x2": 284.0, "x1": 521.5, "y2": 209.0}, {"y1": 148.0, "x2": 521.5, "x1": 237.5, "y2": 357.0}, {"y1": 209.0, "x2": 328.5, "x1": 284.0, "y2": 73.0}, {"y1": 319.0, "x2": 204.0, "x1": 532.5, "y2": 246.0}, {"y1": 126.0, "x2": 534.5, "x1": 153.5, "y2": 354.0}, {"y1": 228.0, "x2": 322.5, "x1": 381.0, "y2": 88.0}, {"y1": 297.0, "x2": 217.0, "x1": 539.5, "y2": 209.0}], "image_path": "/Users/baljitsingh/Desktop/SidraFinal/HiringExercise_MLEngineer_Baljit92/tensorbox/data/images/852e283c084141e3afad609c640a8379.jpg"}] \ No newline at end of file diff --git a/src/machine_learning/README.md b/src/machine_learning/README.md deleted file mode 100644 index 20047d8..0000000 --- a/src/machine_learning/README.md +++ /dev/null @@ -1,38 +0,0 @@ -# Train machine learning model - -## Description - -Once we have the training data and the validation data, use TensorBox to train a machine learning model. A trained model has -already been provided with the name of (write the name)_ - -## Instructions to run -The following pre-requisites need to be satisified for the project to run: -* Python 2.7 -* TensorFlow -* OpenCV -* pip -* pkg-config -* virtualenv -* Pillow 4.1.0 - -In order to install and run TensorBox, follow the instructions on this page [TensorBox setup](https://github.com/baljit92/TensorBox) - -Once TensorBox is setup; **modify** the corresponding fields in `TensorBox/hypes/overfeat_rezoom.json`: - -``` - "train_idl": "", - "test_idl": "" -``` -**Note: The paths should be absolute paths** - -### Evaluation -To evaluate new images, use the evaluation file provided by Tensor box as follows: -`python evaluate.py --weights output/overfeat_rezoom_2017_01_17_15.20/ --test_boxes ` - -### Scripts -Two scripts have ben provided: -1) `convert.py` which converts the .csv file downloaded from the web app to JSON format accepted by TensorBox -2) `image-augment.py` used to generate augmentated versions of a single training/annotated image. Over here; we first -draw bounding box in the area of interest in an image and then use this script to generate an augmented version of the -image. Of course, it is made sure that the image is not rotated to avoid any rectangle coordinate issues. One requirement, -however, to run this is that it uses `Pillow 2.1.0` instead of the latest version. diff --git a/src/webapp/README.md b/src/webapp/README.md deleted file mode 100644 index ecb3ca2..0000000 --- a/src/webapp/README.md +++ /dev/null @@ -1,38 +0,0 @@ -# Web app to annotate/label training images - -## Description - -The web app let's the user annotate images by drawing a red color bounding box around the skulls in the image. The user can label the image with *Skull* or *Not Skull* class. -Once the user clicks the Save button; the metadata of the annotated image such as `(image_name, [[x0, y0, x1, y1, width, height]], class)` is stored in a csv file. -The csv file gets downloaded when the user downloads the training images. - -Non-Maximum Suppression(NSM) support has been added to combine multiple bounding boxes for an image. This helps in suppressing all the image information and displaing only -the essential bounding boxes. - -No annotated images are stored in any directory. Whenever the user wants to view/download the training images; we make use of the coordinates present in the .csv -to draw the bounding box on the fly. user can view/download the training images both with NSM and without NSM. - -Non-Maximum Suppression support has been added to combine multiple bounding boxes for an image. This helps in suppressing all the image information and displaying -only the essential bounding boxes. - - -## Instructions to run -The following pre-requisites need to be satisified for the project to run: -* Python 2.7 -* Django 1.11.2 -* pip - -Following commands can be used to install Django on both macOs and Unix machines: -``` -pip install Django==1.11.2 -pip install Pillow -``` - -Once installed, head to the TrainImage_Annotate project and execute: -``` -python manage.py runserver -``` - -Go to the browser and type in `localhost:8000/draw` - - From ddde650766fae3ca5055fd399eac1acc9343f994 Mon Sep 17 00:00:00 2001 From: Baljit Singh Date: Sat, 1 Jul 2017 02:34:28 +0300 Subject: [PATCH 2/3] Added blind test dataset instructions --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index fee6fb1..f8cca89 100644 --- a/README.md +++ b/README.md @@ -93,6 +93,7 @@ To setup TensorBox and evaluate the model, follow instructions below: The evaluated image results are saved under `data/images_testing_set_18000`. +To use a custom testing dataset, add the image files to `data\images` and replace the testing_set.json with the custom test dataset. The custom dataset should have image_path field as `/images/` and the rects field will contain the true bounding box coordinates. Below is a solution for a potential error that might come up during TensorBox installation. The commands should be executed inside the `virtualenv`: From 29ea2a084588fd5aab9681c03923503bcf759f58 Mon Sep 17 00:00:00 2001 From: baljit92 Date: Sat, 1 Jul 2017 02:35:05 +0300 Subject: [PATCH 3/3] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index f8cca89..80dede0 100644 --- a/README.md +++ b/README.md @@ -93,7 +93,7 @@ To setup TensorBox and evaluate the model, follow instructions below: The evaluated image results are saved under `data/images_testing_set_18000`. -To use a custom testing dataset, add the image files to `data\images` and replace the testing_set.json with the custom test dataset. The custom dataset should have image_path field as `/images/` and the rects field will contain the true bounding box coordinates. +To use a custom testing dataset, add the image files to `data/images` and replace the testing_set.json with the custom test dataset. The custom dataset should have image_path field as `/images/` and the rects field will contain the true bounding box coordinates. Below is a solution for a potential error that might come up during TensorBox installation. The commands should be executed inside the `virtualenv`: