Bad precision in object detection

Hello again,

we have the following problem:

we want to detect traffic signs with the help of object detection and for this, we have picked out the following signs:

We’ve downloaded 100 similar looking pictures for each sign from the internet and have labeled them.


Training/ Test split is 80/20
We also tried a few training settings and the best for us were:
number of cycles 30
learning rate 0.02

As a result, we’ve found out that the system is very sensible because it detects signs even if they aren’t there. Furthermore, it can’t detect real test signs with a high precision.

For this reason, we’ve decided to limit the pics to the ones which look very similar and have eliminated the sign “Gewaehren”, because it was the one which made most of the problems.

before
image

after
image

Now we have 50 pics of each sign but the problems are still there. (trainings settings are the same as before, precision score is 89%, accuracy while model testing is 76% with threshold 0.5).

Do you have any tips or experiences with how to increase the precision of the sign detection and reduce the problem of the detection of non-existing signs?

Thanks in advance!

Hi,

These images are pretty complex and have a lot of information inside.
I would suggest collecting more images, and initially start focusing on classifying only one class (e.g., 30 km sign).
I would suggest that you do the following:

  1. For instance start collecting 150 images for the 30 km sign.
  2. Collect at least 150 images for different types of signs other than the 30 km sign or non-sign images (label them other).
  3. Train the model for these images and try to classify either 30 km sign or other sign.
  4. Increase the number of cycles to at least 100 or even more if you can.

Let me know if this improves the model testing accuracy.

Thank,
Omar

Hey Omar,

thank you for your answer.

I think 100 of training cycles aren’t possible because we don’t use the enterprise version and only have 20 min to train the model.
Which learning rate do you suggest.

Regards,
Jonas

@HTWG_Elite Related to the 20 minutes limitation, I got an extension for one particular project by e-mailing hello@edgeimpulse.com and describing what I wanted to achieve. I had many thousands of images in the training and test set, and was able to achieve my goal with the time extension.

Hello @HTWG_Elite,

We can indeed increase the time limit depending on the projects and the use cases (for example we don’t do it if you want to retrain a model on the entire MNIST dataset or on the entire MS COCO dataset).
But here your project looks interesting and I’d be happy to hear more about what you are trying to achieve? Where do you plan to use this model, on which harware target, etc…

By the way, I just increased your time limit to 60 min instead of 20 on your project called OD3 :wink:

Have a great day,

Louis

4 Likes

Hi @Louis,

thank you for the time extension on our project, we will start an other training as soon as possible. At the moment we paused the project to write our exams, but in a week we will continue with the project and post our result with the new time limit.

To our project: we’re two students who want to build a mobile sign detection device, that can be mounted in a car. For that we run the object detection on a raspberry pi 4 with 8 GB RAM, raspberry camera and an 5’’ Touch Screen. All electronics will be packed inside a 3D-printed case. The case will be mounted on the windscreen via GoPro-mount. To get a high enough framerate for the useage in the car, we will need an accelerator for the raspberry pi, but for the moment we want a proper working model first.

Have a great day,

Jonas

2 Likes

Hello @louis,

We trained the same pics with new parameters and got the following result:

but now, when we want to test the model we get the same error everytime:

Creating job… OK (ID: 2167733)

Generating features for Image…
Not generating new features: features already generated and no options or files have changed.
Classifying data for Object detection…
Classifying data for float32 model…
Scheduling job in cluster…
Job started
Traceback (most recent call last):
** File “/home/classify_float32.py”, line 42, in **
** ei_tensorflow.inference.classify_keras(input_x_file, input_y_file, mode, output_file, dir_path, model_path,**
** File “/app/./resources/libraries/ei_tensorflow/inference.py”, line 343, in classify_keras**
** gt_y = ei_tensorflow.utils.load_y_structured(dir_path, ‘y_classify.npy’, len(input))**
** File “/app/./resources/libraries/ei_tensorflow/utils.py”, line 7, in load_y_structured**
** Y_structured_file = json.loads(file.read())**
** File “/usr/lib/python3.8/json/init.py”, line 357, in loads**
** return _default_decoder.decode(s)**
** File “/usr/lib/python3.8/json/decoder.py”, line 337, in decode**
** obj, end = self.raw_decode(s, idx=_w(s, 0).end())**
** File “/usr/lib/python3.8/json/decoder.py”, line 355, in raw_decode**
** raise JSONDecodeError(“Expecting value”, s, err.value) from None**
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Application exited with code 1 (Error)

Job failed (see above)

With the number of training cycles equal or less than 30 everything works great.

Do you have any ideas what to to?

Best regards,
Jonas

Hi @HTWG_Elite,

Could you try again? There were some issues with the Studio earlier today (see this announcement). I cloned your project and training/testing worked fine for me.

Hey @shawn_edgeimpulse,

the testing works properly now.

Thank you.

2 Likes

Hi @shawn_edgeimpulse , I was searching the forum and found this thread.

I am creating features using a custom model with a local web server on my machine (as per the Edge Impulse custom feature tutorial).

It seemed to be working great until the end then I received the following output:


[9301/9690] Creating features…
[9351/9690] Creating features…
[9401/9690] Creating features…
[9451/9690] Creating features…
[9501/9690] Creating features…
Traceback (most recent call last):
File “/home/create_features.py”, line 102, in
ret = res.json()
File “/usr/local/lib/python3.7/dist-packages/requests/models.py”, line 897, in json
return complexjson.loads(self.text, **kwargs)
File “/usr/lib/python3.7/json/init.py”, line 348, in loads
return _default_decoder.decode(s)
File “/usr/lib/python3.7/json/decoder.py”, line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File “/usr/lib/python3.7/json/decoder.py”, line 355, in raw_decode
raise JSONDecodeError(“Expecting value”, s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
Application exited with code 1

Failed to connect to custom block: 502: Bad Gateway - <!doctype html>

.....

On the local side, I received the following with the python server:
segmentation fault python3 dsp-server.py

any clues??

Thanks!
Jamie

Hi @jhaas,

Could you share your project ID so we can try to replicate your error?

Hi @shawn_edgeimpulse ,

Thanks for the quick response.
Sure, it’s: 227514

@shawn_edgeimpulse , here is another clue. When I run a local Python server with the Edge Impulse tutorial custom feature dsp block, it runs and I do not get this error.

I am locally running python based on the following example code (and this code runs fine):

Hi @jhaas,

This appears to be a problem with your custom dsp-server.py. I suspect there’s an error there (causing the segfault), which then causes the job to fail in Studio. I would take a look at how your dsp-server.py differs from the one in the example (example-custom-processing-block-python/dsp-server.py at master · edgeimpulse/example-custom-processing-block-python · GitHub). If you’re able to share your dsp-server.py, we can also try to take a look.

Hi @shawn_edgeimpulse ,

ok, I don’t think anything was changed in the dsp-server.py but I could be missing something.

Here is the dsp-server.py code:

This is a generic Edge Impulse DSP server in Python

AIStorm - Monarch

import sys, importlib, os, socket, json, math, traceback
from http.server import HTTPServer, BaseHTTPRequestHandler
from socketserver import ThreadingMixIn
import threading
from urllib.parse import urlparse, parse_qs
import traceback
import logging
import numpy as np
from dsp import generate_features

def get_params(self):
with open(‘parameters.json’, ‘r’) as f:
return json.loads(f.read())

def single_req(self, fn, body):
if (not body[‘features’] or len(body[‘features’]) == 0):
raise ValueError(‘Missing “features” in body’)
if (not ‘params’ in body):
raise ValueError(‘Missing “params” in body’)
if (not ‘sampling_freq’ in body):
raise ValueError(‘Missing “sampling_freq” in body’)
if (not ‘draw_graphs’ in body):
raise ValueError(‘Missing “draw_graphs” in body’)

args = {
    'draw_graphs': body['draw_graphs'],
    'raw_data': np.array(body['features']),
    'axes': np.array(body['axes']),
    'sampling_freq': body['sampling_freq'],
    'implementation_version': body['implementation_version']
}

for param_key in body['params'].keys():
    args[param_key] = body['params'][param_key]

processed = fn(**args)
if (isinstance(processed['features'], np.ndarray)):
    processed['features'] = processed['features'].tolist()

body = json.dumps(processed)

self.send_response(200)
self.send_header('Content-Type', 'application/json')
self.end_headers()
self.wfile.write(body.encode())

def batch_req(self, fn, body):
if (not body[‘features’] or len(body[‘features’]) == 0):
raise ValueError(‘Missing “features” in body’)
if (not ‘params’ in body):
raise ValueError(‘Missing “params” in body’)
if (not ‘sampling_freq’ in body):
raise ValueError(‘Missing “sampling_freq” in body’)

base_args = {
    'draw_graphs': False,
    'axes': np.array(body['axes']),
    'sampling_freq': body['sampling_freq'],
    'implementation_version': body['implementation_version']
}

for param_key in body['params'].keys():
    base_args[param_key] = body['params'][param_key]

total = 0
features = []
labels = []
output_config = None

for example in body['features']:
    args = dict(base_args)
    args['raw_data'] = np.array(example)
    f = fn(**args)
    if (isinstance(f['features'], np.ndarray)):
        features.append(f['features'].tolist())
    else:
        features.append(f['features'])

    if total == 0:
        if ('labels' in f):
            labels = f['labels']
        if ('output_config' in f):
            output_config = f['output_config']

    total += 1

body = json.dumps({
    'success': True,
    'features': features,
    'labels': labels,
    'output_config': output_config
})

self.send_response(200)
self.send_header('Content-Type', 'application/json')
self.end_headers()
self.wfile.write(body.encode())

class Handler(BaseHTTPRequestHandler):
def do_GET(self):
url = urlparse(self.path)
params = get_params(self)

    if (url.path == '/'):
        self.send_response(200)
        self.send_header('Content-Type', 'text/plain')
        self.end_headers()
        self.wfile.write(('Edge Impulse DSP block: ' + params['info']['title'] + ' by ' +
                          params['info']['author']).encode())

    elif (url.path == '/parameters'):
        self.send_response(200)
        self.send_header('Content-Type', 'application/json')
        self.end_headers()
        params['version'] = 1
        self.wfile.write(json.dumps(params).encode())

    else:
        self.send_response(404)
        self.send_header('Content-Type', 'text/plain')
        self.end_headers()
        self.wfile.write(b'Invalid path ' + self.path.encode() + b'\n')

def do_POST(self):
    url = urlparse(self.path)
    try:
        if (url.path == '/run'):
            content_len = int(self.headers.get('Content-Length'))
            post_body = self.rfile.read(content_len)
            body = json.loads(post_body.decode('utf-8'))
            single_req(self, generate_features, body)

        elif (url.path == '/batch'):
            content_len = int(self.headers.get('Content-Length'))
            post_body = self.rfile.read(content_len)
            body = json.loads(post_body.decode('utf-8'))
            batch_req(self, generate_features, body)

        else:
            self.send_response(404)
            self.send_header('Content-Type', 'text/plain')
            self.end_headers()
            self.wfile.write(b'Invalid path ' + self.path.encode() + b'\n')


    except Exception as e:
        print('Failed to handle request', e, traceback.format_exc())
        self.send_response(200)
        self.send_header('Content-Type', 'application/json')
        self.end_headers()
        self.wfile.write(json.dumps({'success': False, 'error': str(e)}).encode())

def log_message(self, format, *args):
    return

class ThreadingSimpleServer(ThreadingMixIn, HTTPServer):
pass

def run():
host = ‘0.0.0.0’ if not ‘HOST’ in os.environ else os.environ[‘HOST’]
port = 4446 if not ‘PORT’ in os.environ else int(os.environ[‘PORT’])

server = ThreadingSimpleServer((host, port), Handler)
print('Listening on host', host, 'port', port)
server.serve_forever()

if name == ‘main’:
run()

Hi @jhaas,

The example dsp-server.py worked for me on your project. That leads me to believe one of two things happened for you:

  • You reached the 20 min free tier time limit on Edge Impulse
  • You reached the 2 hour free tier time limit on ngrok

If it was an issue with Edge Impulse, then I recommend removing a few hundred samples and trying again to see if it still works.

If it was an issue with ngrok, then you will need to restart the ngrok tool. See here: Using Ngrok in 2022

Hi @shawn_edgeimpulse ,

I really appreciate the support! I ran it this morning and it worked fine. I had also changed some of the items in the parameters.json file and I am wondering if that had anything to do with it since I also changed that back to the default text labels that where in there. You explanation makes more sense!

Thanks again,
Jamie

1 Like