FOMO run detection method from inference.py locally

exodiakiller · May 11, 2023, 7:59pm

I am trying to create the “live classification” feature locally using the run_object_detection_inference method from inference.py . Here’s how I call the method:

interpreter = tf.lite.Interpreter(model_path=“…”)
interpreter.allocate_tensors()
specific_input_shape=[320, 320, 1]
y_data = [40, 40, 2]
img = Image.open(“…jpg”)
img = img.convert(“L”)
numpydata = asarray(img)

minimum_confidence_rating = 0.5
num_classes = 1
t.run_object_detection_inference(
interpreter,
numpydata,
specific_input_shape,
minimum_confidence_rating,
y_data,
num_classes
)

However, when running the code, I am encountering a list index out of range error for the command interpreter.get_tensor(output_details[2])[0].tolist() in inference.py .

I would appreciate any help with this issue.

louis · May 12, 2023, 9:31am

Hello @exodiakiller,

Can you share the link of the inference.py please?
I am not sure which example you are referring to.

Best,

Louis

exodiakiller · May 12, 2023, 1:50pm

Thanks for the quick reply. I’m referring to inference.py , which I obtained from the “edit block locally” option. I have uploaded the files to GitHub: Fomodetection/inference.py at main · Exodiakiller/Fomodetection · GitHub

matkelcey · May 13, 2023, 1:43am

do you have a standalone reproduction case that can be run? the out of bounds you mention could be triggered by a number of different things

mat

exodiakiller · May 13, 2023, 5:20pm

I noticed that the output_details I get through interpreter.get_output_details() only has one output tensor named ‘StatefulPartitionedCall:0’. If I understand the code correctly, there should be a total of four output tensors. Have you been able to execute a FOMO model using the method run_object_detection_inference from inference.py? Unfortunately, I don’t have an exact standalone reproduction case.

matkelcey · May 16, 2023, 9:41pm

i’ve only ever run this piece within the context of studio. e.g. edit block locally, train model in a custom way, then reupload back into studio to then deploy to a specific target.

is there a reason you’re trying to roll your own inference piece? ( as opposed to using a deployment? )

mat

exodiakiller · May 16, 2023, 11:39pm

I plan to extract the bounding boxes/centroids after detection in order to perform further recognition on the respective objects. My objects are so small that they fit within the circle.

The project is initially intended to run in Python and then on the ESP32.

Is there perhaps a sample project that runs the TFLite FOMO model in Python?

dansitu · May 17, 2023, 5:54pm

Hi @exodiakiller,

Thanks for using Edge Impulse! We only really provide this Python code for training; it’s not designed for efficient or convenience inference and we recommend users stick to our typical deployment options (C++, etc).

That said, if you are really set on doing FOMO inference in Python you can use the run_segmentation_inference() function for inspiration:

github.com

Exodiakiller/Fomodetection/blob/249a286726865b6e1ed28b0bcef225b7bc7b0dc2/ei_tensorflow/inference.py#L321


      
              if not y_data:
                  raise ValueError('y_data must be provided for object detection')
              if not num_classes:
                  raise ValueError('num_classes must be provided for object detection')
              if not minimum_confidence_rating:
                  raise ValueError('minimum_confidence_rating must be provided for object detection')
              raw_scores = process_output_object_detection(output_details, interpreter, minimum_confidence_rating)
              scores = compute_performance_object_detection(raw_scores, width_height, y_data, num_classes)
              return scores
          
          
def run_segmentation_inference(interpreter: Interpreter, item: np.ndarray, specific_input_shape: 'list[int]',
                                         minimum_confidence_rating: float, y_data: list):
              """Runs inference that produces an object detection output
              """
          
          
    if not y_data:
                  raise ValueError('y_data must be provided for object detection')
              if not minimum_confidence_rating:
                  raise ValueError('minimum_confidence_rating must be provided for object detection')
          
          
    width, height, _channels = specific_input_shape

You won’t be able to use run_object_detection_inference() since that is for SSD models.

run_segmentation_inference() does a bunch of things, including evaluation against a labelled input. The important call for you is this one, which takes the raw output of the FOMO model and turns it into bounding boxes:

    # convert model output to list of BoundingBoxLabelScores including fusing
    # of adjacent boxes. retains class=0 from segmentation output.
    y_pred_boxes_labels_scores = convert_segmentation_map_to_object_detection_prediction(
        output, minimum_confidence_rating, fuse=True)

You can also use the output “raw” and treat it as a segmentation map showing class probabilities for each region of the image.

Warmly,
Dan

MMarcial · May 17, 2023, 7:07pm

@exodiakiller have you seen this python code.

exodiakiller · May 19, 2023, 3:06pm

Thank you for the response. I haven’t looked at it yet, but I will check it out.

MMarcial · May 19, 2023, 9:42pm

Yes, I think it will accomplish what you want but using other higher level Edge Impulse routines.