Merged changes

README.md

@@ -29,6 +29,9 @@ The following pre-requisites need to be satisified for the project to run:
 * Django 1.11.2
 * `pip`
 * OpenCV
+* pip
+* Pillow 4.1.1
+
 
 For Ubuntu, install freetype before installing the required modules:
 ```
@@ -66,7 +69,7 @@ After downloading the dataset csv file from the webapp; use the script `convert_
 ## Description
 
 Once we have the training data and the validation data, use *TensorBox* to train a machine learning model. A trained model has
-already been provided with the name of _save.ckpt-180000_
+already been provided with the name of _save.ckpt-18000_
 
 ## Pre-requisites
 The following pre-requisites need to be satisified for the machine learning project to run:
@@ -95,15 +98,20 @@ To setup TensorBox and evaluate the model, follow instructions below:
 	mv ./src/machine_learning/data/* ./tensorbox/data/
 	mv ./src/machine_learning/config/* ./tensorbox/hypes/
 	cd tensorbox
-	python evaluate.py --weights data/save.ckpt-180000 --test_boxes data/testing_set.json
-	```
+	python evaluate.py --weights data/save.ckpt-18000 --test_boxes data/testing_set.json
+
+
+The evaluated image results are saved under `data/images_testing_set_18000`. 
+
+To use a custom testing dataset, add the image files to `data/images` and replace the testing_set.json with the custom test dataset. The custom dataset should have image_path field as `/images/<image_name>` and the rects field will contain the true bounding box coordinates.
 
 Below is a solution for a potential error that might come up during TensorBox installation. The commands should be executed inside the `virtualenv`:
 
-2. **Error:** pkg-config python not found
+1. **Error:** pkg-config python not found
+
    **Solution for macOS:** export PKG_CONFIG_PATH=/System/Library/Frameworks/Python.framework/Versions/2.7/lib/pkgconfig/; echo $(pkg-config --variable pc_path pkg-config)${PKG_CONFIG_PATH:+:}${PKG_CONFIG_PATH}
 
-_Note: In order to use a custom training dataset, the files should be placed under the folder 'tensorbox/data' and modify the fields training and testing fields in `tensorbox/hypes/overfeat_rezoom.js`_
+_Note: In order to use a custom training dataset, the training and validation files should be placed under 'tensorbox/data' and modify the fields training and testing fields in `tensorbox/hypes/overfeat_rezoom.js`_
 
 ## The Dataset
 The images used for this project have all been taken from the web. The images are of human skulls, non human skulls and various other objects such as people, cartoon characters, etc. After the completion of data collection, every image was passed through
@@ -115,7 +123,9 @@ the web app developed in the first phase.  The image metadata is then saved in a
 
 ## Results
 
-Full testing output fromm the trainined model can be found under `results/`. In the images, the green boxes are the final predictions after merging, and red are all predictions after applying a threshold of confidence, but before merging.
+Full testing output from the trained model can be found under `results/`. In the images, the green boxes are the final predictions after merging, and red are all predictions after applying a threshold of confidence, but before merging.
+
+For the test dataset used, average accuracy was 90%.
 
 #### Positive Images
 Below are some positive images that were classified correctly. Most of the positively classified images, as below, are images that contain a standalone skull whereas some of them also contain noise in the background.
@@ -126,7 +136,7 @@ Below are some positive images that were classified correctly. Most of the posit
 Most of the negative images were classified correctly, however as we can see below, some of the negative images that were positively classified contain backgorunds, skeletons, faces or facial-like features such as eyes. The trained model drew bounding boxes around random objects in addition to the human skull (if present). 
 
 
-For the test dataset used, average accuracy was 90%
+
 
 ## Discussion
 
@@ -167,6 +177,8 @@ Couple of points:
 
 1. Use ResNet. TensorBox contains ResNet that we could use for much deeper learning with more resources.
 2. Training on a much larger data set that includes more negative class than positives and images with different resolutions.
-3. Use more variations of image augmentations such as shearing, rotation to increase the accuracy.
-4. For web app, use outlier detection to prevent spam in the training set
-5. For web app, fix canvas drawing for large resolution images
+3. Train the model for more epochs
+4. Use more variations of image augmentations such as shearing, rotation to increase the accuracy.
+5. For web app, use outlier detection to prevent spam in the training set
+6. For web app, fix canvas drawing for large resolution images
+

src/custom_scripts/convert_csv_to_json.py

@@ -3,7 +3,7 @@
 import json
 import ast
 import sys, getopt
-
+import math
 
 def parse(ifile, ofile):
 	'''
@@ -107,32 +107,45 @@ def main(argv):
 			# create a list of dictionaries for each line
 			for line in csv_file:
 				d = {}
-				print line
 				line_split = line.split("\"")
-
+				#print line
 				imgfile_name = img_directory+"/"+line_split[0]
 				d['image_path'] = imgfile_name[:-1]
 
 				try:
 					line_split[1] != "[]"
 					line_split[1] = line_split[1][1:]
 					line_split[1] = line_split[1][:-1]
+					temp_array = []
+					try:
+						temp_array = ast.literal_eval(line_split[1])
+					except SyntaxError:
+						nan_array = line_split[1].split(",")
+						for (_, arr_val) in enumerate(nan_array):
+							ele_len = len(arr_val)
+							if ele_len>0:
+								is_nan = math.isnan(float(arr_val))
+								if not is_nan:
+									temp_array.append(float(arr_val))
 
-					temp_array = ast.literal_eval(line_split[1])
 
 					#duck-typing
 					var_type = isinstance(temp_array, (list, tuple))
 
 					rects_array = []
 
 					if var_type:
-						# list if there iis more than one rectangle
-						if type(temp_array) == type([]):
-							rects_array.append({'x1':temp_array[0], 'y1':temp_array[1], 'x2':temp_array[2], 'y2':temp_array[3]})
+						print temp_array
+
 						# tuple if there is one rectangle
-						elif type(temp_array) == type(()):
-							for (index, _) in enumerate(temp_array):
-								coords = {'x1':temp_array[index][0], 'y1':temp_array[index][1], 'x2':temp_array[index][2], 'y2':temp_array[index][3]}
+						if type(temp_array) == type(()):
+
+							rects_array.append({'x1':temp_array[0], 'y1':temp_array[1], 'x2':temp_array[2], 'y2':temp_array[3]})
+						# list if there is more than one rectangle
+						elif type(temp_array) == type([]):
+							num_rects = len(temp_array)/4
+							for index in range(0,num_rects):
+								coords = {'x1':temp_array[(4*index)+0], 'y1':temp_array[(4*index)+1], 'x2':temp_array[(4*index)+2], 'y2':temp_array[(4*index)+3]}
 								rects_array.append(coords)
 					d['rects'] = rects_array
 

src/custom_scripts/image_augment.py

@@ -183,7 +183,7 @@ def main(argv):
 					img = transform_image(image,0,brightness=1)
 					im = Image.fromarray(img)
 
-					#generate only 5 augmented versions with big variations in the image
+					#generate only 4 augmented versions with big variations in the image
 					if i%5:
 						im.save(output_img_directory+"/"+file_name+"_"+str(i)+"."+file_ext)
 						image_new = file_name+"_"+str(i)+"."+file_ext

src/custom_scripts/training_data.json

@@ -0,0 +1 @@
+[{"rects": [], "image_path": "/Users/baljitsingh/Desktop/SidraFinal/HiringExercise_MLEngineer_Baljit92/tensorbox/data/images/16b1cfa12b64.jpg"}, {"rects": [], "image_path": "/Users/baljitsingh/Desktop/SidraFinal/HiringExercise_MLEngineer_Baljit92/tensorbox/data/images/1a.jpg"}, {"rects": [{"y1": 103, "x2": 259.5, "x1": 121.5, "y2": 184}], "image_path": "/Users/baljitsingh/Desktop/SidraFinal/HiringExercise_MLEngineer_Baljit92/tensorbox/data/images/2007_000027.jpg"}, {"rects": [{"y1": 97.0, "x2": 284.5, "x1": 167.5, "y2": 201.0}], "image_path": "/Users/baljitsingh/Desktop/SidraFinal/HiringExercise_MLEngineer_Baljit92/tensorbox/data/images/2007_000032.jpg"}, {"rects": [], "image_path": "/Users/baljitsingh/Desktop/SidraFinal/HiringExercise_MLEngineer_Baljit92/tensorbox/data/images/2007_000033.jpg"}, {"rects": [], "image_path": "/Users/baljitsingh/Desktop/SidraFinal/HiringExercise_MLEngineer_Baljit92/tensorbox/data/images/2007_000039.jpg"}, {"rects": [{"y1": 78.0, "x2": 472.5, "x1": 324.5, "y2": 270.0}, {"y1": 192.0, "x2": 237.5, "x1": 148.0, "y2": 148.0}, {"y1": 357.0, "x2": 284.0, "x1": 521.5, "y2": 209.0}, {"y1": 148.0, "x2": 521.5, "x1": 237.5, "y2": 357.0}, {"y1": 209.0, "x2": 328.5, "x1": 284.0, "y2": 73.0}, {"y1": 319.0, "x2": 204.0, "x1": 532.5, "y2": 246.0}, {"y1": 126.0, "x2": 534.5, "x1": 153.5, "y2": 354.0}, {"y1": 228.0, "x2": 322.5, "x1": 381.0, "y2": 88.0}, {"y1": 297.0, "x2": 217.0, "x1": 539.5, "y2": 209.0}], "image_path": "/Users/baljitsingh/Desktop/SidraFinal/HiringExercise_MLEngineer_Baljit92/tensorbox/data/images/852e283c084141e3afad609c640a8379.jpg"}]