Bad Accuracy with Object Detection

barry5 · May 10, 2022, 1:24am

Hi,

We’re creating a project where we do object detection of a stop sign as well as red, green, and yellow traffic lights. Here are the images we have selected to use:

We labeled 60 photos (that we took with a Raspberry Pi camera) of each object, and we trained the model on 142 inputs / validating on 36 inputs. The training/test split is about 80/20. Number of training cycles = 50. Learning rate = 0.001.

However, the accuracy of our model testing results is very low (38.71%). It doesn’t detect the yellow or red traffic lights and/or will mislabel them as the green traffic lights.

The Feature Explorer shows good separation of the labeled objects:

We want to either increase the images in the dataset or increase the number of training cycles to improve model accuracy, but we have the Enterprise version with a 20 minute time limit.

Do you have any suggestions on improving the accuracy?

Thank you in advance!

louis · May 10, 2022, 4:31pm

Hello @barry5 ,

Can you share your project ID please so I can have a deeper look?

Regards,

Louis

barry5 · May 10, 2022, 5:55pm

Hi @louis ,

Our project ID is 103750 (project name is barry5-project-1).

louis · May 10, 2022, 6:36pm

Hello @barry5 ,

I am afraid this has to do mainly with your dataset…

Your “objects” are pretty small in the image, MobileNetV2 SSD is not great for detecting small objects in a frame. You can try to take closer pictures of your object to improve that
You’ll probably need more data to improve the accuracy, especially given your traffic lights objects share a lot of common features.
I am not sure if the end goal of your project is to recognize the real stop signs and real traffic lights. If so, you’re model will probably won’t work as you have some white background around your objects that will likely not be there in real conditions.

I hope that helps,

Regards,

Louis

barry5 · May 10, 2022, 8:12pm

Hi @louis ,

We’ll try taking closer pictures of our object!
We have more data that we have already collected, but with the 20 minute time limit, we are unable to add more photos to our project without the model training’s time expiring.
The end goal of our project is to display these photos on our mobile phones in front of our Raspberry Pi camera. We don’t plan on having our project recognize real traffic signs.

Is there a way to increase our time limit so that we can create a more robust dataset? Thank you for your suggestions and help!

louis · May 10, 2022, 8:20pm

Hello @barry5,

I just increased your time limit. I hope that can increase your accuracy.

Regards,

Louis

barry5 · May 13, 2022, 5:34pm

Hi @louis ,

Thank you for your previous suggestions and for increasing our time limit. We followed your tips and retook closer photos of our objects.

Currently, the highest accuracy we can get is 66.67% (with around 200 photos using the 80/20 split) using 175 training cycles, 0.001 learning rate, and 20% validation set size. Our biggest obstacle is that our model cannot differentiate between the red, yellow, and green lights.

If we are using MobileNetV2 SSD, would increasing the size of our captured image help? We are taking 320x320 size photos, so would increasing the size to 500x500 affect anything?

Thank you in advance!

louis · May 13, 2022, 6:52pm

Hello @barry5,

you won’t be able to use 500x500px using our pre-trained MobileNetV2 SSD.
Have you also tried the standard image classification? If you have only one object in the image, it should also work (not sure it matches your needs though).

Try with different parameters. I’d try to use 0.15 as the learning rate and 50 epochs to get started and then slight decrease the LR if you’re converging to fast.

Regards,

Louis

barry5 · May 17, 2022, 12:28am

Hi @louis ,

Thank you for your help! We trained our model with 50 training cycles, 0.15 learning rate, and a 20% validation size. With this, we were able to get a much higher accuracy!

However, when we played around with the training model and lowered the learning rate to 0.13, we received these errors:

Attached to job 2841135…
Calculating performance metrics…
Calculating inferencing time…
ERROR: /app/tensorflow-2.4.1/tensorflow/lite/kernels/detection_postprocess.cc:426 ValidateBoxes(decoded_boxes, num_boxes) was not true.
ERROR: Node number 156 (TFLite_Detection_PostProcess) failed to invoke.

ERROR: /app/tensorflow-2.4.1/tensorflow/lite/kernels/detection_postprocess.cc:426 ValidateBoxes(decoded_boxes, num_boxes) was not true.
ERROR: Node number 156 (TFLite_Detection_PostProcess) failed to invoke.

Calculating inferencing time OK
Profiling float32 model…

Model training complete

Job completed
Model

Creating job… OK (ID: 2841210)

Generating features for Image…
Not generating new features: features already generated and no options or files have changed.

Reducing dimensions for Image…
Scheduling job in cluster…
Job started
Reducing dimensions for visualizations…
ERR: Expected 2D array, got 1D array instead:
array=[].
Reshape your data either using array.reshape(-1, 1) if your data has a single feature or array.reshape(1, -1) if it contains a single sample.
Application exited with code 1

Could you please help us understand what caused this error and how to avoid it in the future?

Thank you again!

louis · May 17, 2022, 10:39am

Hello @barry5 ,

That’s odd. Did you change anything in your ML architecture?

barry5 · May 18, 2022, 12:06am

Hi @louis ,

Other than decreasing the learning rate, nothing was changed. We have seen these errors a few times when playing around with the training model.

louis · May 18, 2022, 3:59pm

Hello @barry5,

@dansitu will have a look at your project soon.
FYI, I just created a version called Edge Impulse Support so Dan can clone it to reproduce your issue.

Regards,

Louis

dansitu · May 20, 2022, 12:42am

Hi @barry5,

Thank you so much for reporting these errors! The first one, ERROR: Node number 156 (TFLite_Detection_PostProcess) failed to invoke., is safe to ignore—we’re in the process of fixing it, but it won’t get in the way of your training.

For the second error (ERR: Expected 2D array, got 1D array instead:) I wasn’t able to reproduce in the support version of your project, so I’d love to get a bit more context. Did this happen during Model Testing, or in during Generate Features? Is it still happening in your original project?

Warmly,
Dan

robhazes · May 22, 2022, 7:31pm

Hi, I had the same problem with trying to detect and recognize speed limit signs. To make separate detector and classifier may solve your problem.