Getting a different result from the Edge Impulse CLI and my local Python app

Help
#1

Hey everyone! I was excited to see Linux SBC support drop a few weeks back and have been working on a project using Edge Impulse to perform image classification on an analog pressure gauge using the Pi and Pi camera v2. My model is tuned pretty well and I am happy with the results, but I am seeing different results when using the Python SDK vs edge-impulse-linux-runner.

Here’s what I am seeing when using the Python SDK runner.classifier with the output dict sorted and reversed so the “winner” comes out on to (note that tank-pressure-low is inferred as the most-likely result):

classification runner response
 [('tank-pressure-low', 0.9246085286140442), ('tank-pressure-off', 0.03462480753660202), ('tank-pressure-high', 0.03320816159248352), ('tank-pressure-normal', 0.007558567449450493)]
classification runner response
 [('tank-pressure-low', 0.922396719455719), ('tank-pressure-off', 0.04392895847558975), ('tank-pressure-high', 0.023367837071418762), ('tank-pressure-normal', 0.010306425392627716)]
classification runner response
 [('tank-pressure-low', 0.928776204586029), ('tank-pressure-off', 0.04399576783180237), ('tank-pressure-high', 0.015807965770363808), ('tank-pressure-normal', 0.011420144699513912)]
classification runner response
 [('tank-pressure-low', 0.9372159838676453), ('tank-pressure-off', 0.04013027250766754), ('tank-pressure-normal', 0.012561420910060406), ('tank-pressure-high', 0.010092317126691341)]
classification runner response
 [('tank-pressure-low', 0.9334703683853149), ('tank-pressure-off', 0.043883539736270905), ('tank-pressure-normal', 0.016150416806340218), ('tank-pressure-high', 0.006495747249573469)]

And here’s what I get when I run that exact some model using edge-impulse-linux-runner (note that tank-pressure-normal is inferred as the most likely result:

classifyRes 38ms. {
  'tank-pressure-high': '0.0008',
  'tank-pressure-low': '0.0001',
  'tank-pressure-normal': '0.9989',
  'tank-pressure-off': '0.0002'
}
classifyRes 13ms. {
  'tank-pressure-high': '0.0007',
  'tank-pressure-low': '0.0001',
  'tank-pressure-normal': '0.9991',
  'tank-pressure-off': '0.0002'
}
classifyRes 15ms. {
  'tank-pressure-high': '0.0006',
  'tank-pressure-low': '0.0002',
  'tank-pressure-normal': '0.9991',
  'tank-pressure-off': '0.0002'
}
classifyRes 17ms. {
  'tank-pressure-high': '0.0004',
  'tank-pressure-low': '0.0001',
  'tank-pressure-normal': '0.9994',
  'tank-pressure-off': '0.0001'
}
classifyRes 11ms. {
  'tank-pressure-high': '0.0002',
  'tank-pressure-low': '0.0001',
  'tank-pressure-normal': '0.9996',
  'tank-pressure-off': '0.0001'
}

The CLI is giving me the correct, expected result and the Python SDK is not. I have run these one after another and nothing has changed about the placement of the camera, gauge, or anything else.

I can dig deeper into the Python SDK to see if there’s an issue there, but I figured I’d post here in case I am doing something wrong. I’m using the image classifier example.

#2

Hi Brandon,

I couldn’t reproduce your issue using the Python SDK image example and the CLI runner. Could you share the code snippet when you sort the output dict?

Thanks,
Aurelien

#3

Thanks @aurel, just to take my own code out of the mix, I pulled the example classify.py source and ran it unmodified. Here’s the result (low is the consistent classification):

Result (36 ms.) tank-pressure-high: 0.00        tank-pressure-low: 0.96 tank-pressure-normal: 0.04      tank-pressure-off: 0.00
Result (11 ms.) tank-pressure-high: 0.00        tank-pressure-low: 0.97 tank-pressure-normal: 0.03      tank-pressure-off: 0.00
Result (18 ms.) tank-pressure-high: 0.00        tank-pressure-low: 0.98 tank-pressure-normal: 0.02      tank-pressure-off: 0.00
Result (13 ms.) tank-pressure-high: 0.00        tank-pressure-low: 0.99 tank-pressure-normal: 0.01      tank-pressure-off: 0.00
Result (14 ms.) tank-pressure-high: 0.00        tank-pressure-low: 0.99 tank-pressure-normal: 0.01      tank-pressure-off: 0.00
Result (14 ms.) tank-pressure-high: 0.00        tank-pressure-low: 0.99 tank-pressure-normal: 0.01      tank-pressure-off: 0.00
Result (14 ms.) tank-pressure-high: 0.00        tank-pressure-low: 1.00 tank-pressure-normal: 0.00      tank-pressure-off: 0.00
Result (14 ms.) tank-pressure-high: 0.00        tank-pressure-low: 1.00 tank-pressure-normal: 0.00      tank-pressure-off: 0.00
Result (14 ms.) tank-pressure-high: 0.00        tank-pressure-low: 1.00 tank-pressure-normal: 0.00      tank-pressure-off: 0.00
Result (16 ms.) tank-pressure-high: 0.00        tank-pressure-low: 1.00 tank-pressure-normal: 0.00      tank-pressure-off: 0.00

And here’s the output from edge-impulse-linux-runner (off is the consistent classification):

classifyRes 14ms. {
  'tank-pressure-high': '0.0035',
  'tank-pressure-low': '0.0012',
  'tank-pressure-normal': '0.2027',
  'tank-pressure-off': '0.7926'
}
classifyRes 13ms. {
  'tank-pressure-high': '0.0035',
  'tank-pressure-low': '0.0012',
  'tank-pressure-normal': '0.1574',
  'tank-pressure-off': '0.8378'
}
classifyRes 15ms. {
  'tank-pressure-high': '0.0038',
  'tank-pressure-low': '0.0015',
  'tank-pressure-normal': '0.2010',
  'tank-pressure-off': '0.7938'
}
classifyRes 13ms. {
  'tank-pressure-high': '0.0039',
  'tank-pressure-low': '0.0012',
  'tank-pressure-normal': '0.1909',
  'tank-pressure-off': '0.8040'
}
classifyRes 12ms. {
  'tank-pressure-high': '0.0023',
  'tank-pressure-low': '0.0014',
  'tank-pressure-normal': '0.1660',
  'tank-pressure-off': '0.8304'
}
classifyRes 14ms. {
  'tank-pressure-high': '0.0023',
  'tank-pressure-low': '0.0017',
  'tank-pressure-normal': '0.2096',
  'tank-pressure-off': '0.7864'
}

Just to be certain I was running the same model, I used the downloaded model pulled in by edge-impulse-linux-runner and got the same result.

It seems odd that I’d be getting a consistently different result using the same model moments apart. Only difference I can see is that one is using the Python SDK and one is not.

Anyway, I know this is hard to repro since you don’t have my camera set-up, but I am happy to provide any other context needed.

CLI and Python give different results on Raspberry Pi
#4

@bsatrom I’ve checked the pipeline in Python with an example from your test set in your project and that goes well, so I’m wondering if it’s a camera issue.

Do you have a screen hooked up by any chance? If so, can you enable show_camera in classify.py - it will output the current image as it’s classified by the network. Wonder if it looks distorted…

1 Like
#5

I don’t currently, but I’ll see if I can hook one up, or just output the image to a file.

1 Like
#6

Thanks @bsatrom that’d be helpful.