Edge impulse on coral edgetpu

kashif · July 3, 2021, 1:34pm

So I managed to get the edge-impulse-linux working on coral edgetpu and I see the device in my devices dashboard

mendel@arid-ibis:~$ edge-impulse-linux
Edge Impulse Linux client v1.2.6

[SER] Using microphone hw:0,0
[SER] Using camera i.MX6S_CSI starting...
[SER] Connected to camera
[WS ] Connecting to wss://remote-mgmt.edgeimpulse.com
[WS ] Connected to wss://remote-mgmt.edgeimpulse.com

However when I try to get camera data I get:

[WS ] Incoming sampling request {
  path: '/api/training/data',
  label: 'edge-tpu-1',
  length: 60000,
  interval: 0,
  hmacKey: '0f5eda6c3f1cc8e5380fb619a6df620d',
  sensor: 'Camera (640x480)'
}
[WS ] Failed to sample data Timeout

Any idea what I could be doing wrong here? Some debugging info:

mendel@arid-ibis:~$ gst-device-monitor-1.0
Probing devices...


Device found:

	name  : i.MX6S_CSI
	class : Video/Source
	caps  : video/x-raw, format=(string)YUY2, width=(int)2592, height=(int)1944, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)15/1;
	        video/x-raw, format=(string)YUY2, width=(int)1920, height=(int)1080, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction){ 15/1, 30/1 };
	        video/x-raw, format=(string)YUY2, width=(int)1280, height=(int)720, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)30/1;
	        video/x-raw, format=(string)YUY2, width=(int)720, height=(int)480, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)30/1;
	        video/x-raw, format=(string)YUY2, width=(int)640, height=(int)480, pixel-aspect-ratio=(fraction)1/1, framerate=(fraction)30/1;
	properties:
		udev-probed = true
		device.bus_path = platform-30a90000.csi1_bridge
		sysfs.path = /sys/devices/platform/30a90000.csi1_bridge/video4linux/video0
		device.subsystem = video4linux
		device.product.name = i.MX6S_CSI
		device.capabilities = :capture:
		device.api = v4l2
		device.path = /dev/video0
		v4l2.device.driver = mx6s-csi
		v4l2.device.card = i.MX6S_CSI
		v4l2.device.bus_info = platform:30a90000.csi1_bridge
		v4l2.device.version = 265826 (0x00040e62)
		v4l2.device.capabilities = 2216689665 (0x84200001)
		v4l2.device.device_caps = 69206017 (0x04200001)
	gst-launch-1.0 v4l2src ! ...

mendel@arid-ibis:~$ v4l2-ctl --list-formats-ext --device /dev/video0 
ioctl: VIDIOC_ENUM_FMT
	Type: Video Capture

	[0]: 'YUYV' (YUYV 4:2:2)
		Size: Discrete 640x480
			Interval: Discrete 0.033s (30.000 fps)
		Size: Discrete 720x480
			Interval: Discrete 0.033s (30.000 fps)
		Size: Discrete 1280x720
			Interval: Discrete 0.033s (30.000 fps)
		Size: Discrete 1920x1080
			Interval: Discrete 0.067s (15.000 fps)
			Interval: Discrete 0.033s (30.000 fps)
		Size: Discrete 2592x1944
			Interval: Discrete 0.067s (15.000 fps)
		Size: Discrete 0x0

and

mendel@arid-ibis:~$ gst-launch-1.0 v4l2src device=/dev/video0 num-buffers=1 ! video/x-raw ! videoconvert ! jpegenc ! multifilesink location=test%05d.jpg
Setting pipeline to PAUSED ...
Pipeline is live and does not need PREROLL ...
Setting pipeline to PLAYING ...
New clock: GstSystemClock
Got EOS from element "pipeline0".
Execution ended after 0:00:02.587398250
Setting pipeline to PAUSED ...
Setting pipeline to READY ...
Setting pipeline to NULL ...
Freeing pipeline ...

outputs a valid test image file.

louis · July 5, 2021, 1:33pm

Hello @kashif,

What is the distribution of your main computer on which the Coral Edge TPU is plugged?
Also, could you please share the output of edge-impulse-linux --verbose?

Regards,

kashif · July 5, 2021, 1:47pm

Thanks for the questions @louis, So coral is running Mendel Linux which is a lightweight derivative of Debian Linux. I followed the recipe of installing the Linux cli on Jetson nano and it worked on the coral apart from the fact I needed to:

mendel@arid-ibis:~$ sudo apt install libjpeg62-turbo-dev

Anyways, to answer your questions, I rebooted and tried, and it’s working!!

[WS ] Incoming sampling request {
  path: '/api/training/data',
  label: 'test',
  length: 60000,
  interval: 0,
  hmacKey: '0f5eda6c3f1cc8e5380fb619a6df620d',
  sensor: 'Camera (640x480)'
}

very cool! So I needed to reboot it seems! Is there someplace I can update the documentation since it seems to work with coral devices too?

kashif · July 5, 2021, 2:08pm

So the next issue which I suspected would the blocker is the compilation of the tflite model:

Edge Impulse Linux runner v1.2.6

[RUN] Downloading model...
[BLD] Created build job with ID 1053031
[BLD] Writing templates OK
[BLD] Scheduling job in cluster...
[BLD] Job started
[BLD] Exporting TensorFlow Lite model...
[BLD] Found operators ['Conv2D', 'FullyConnected', 'MaxPool2D', 'Reshape', 'Softmax']
[BLD] Exporting TensorFlow Lite model OK
[BLD]
[BLD] Removing clutter...
[BLD] Removing clutter OK
[BLD]
[BLD] Copying output...
[BLD] Copying output OK
[BLD]
[BLD] Job started
[BLD] Building binary...
[BLD] aarch64-linux-gnu-g++ -MD -Wall -g -Wno-strict-aliasing -I. -Isource -Imodel-parameters -Itflite-model -Ithird_party/ -Os -DNDEBUG -g -DEI_CLASSIFIER_USE_FULL_TFLITE=1 -Iedge-impulse-sdk/tensorflow-lite -Dfloat16_t=float -std=c++14 -c source/main.cpp -o source/main.o
[BLD] aarch64-linux-gnu-g++ -MD -Wall -g -Wno-strict-aliasing -I. -Isource -Imodel-parameters -Itflite-model -Ithird_party/ -Os -DNDEBUG -g -DEI_CLASSIFIER_USE_FULL_TFLITE=1 -Iedge-impulse-sdk/tensorflow-lite -Dfloat16_t=float -std=c++14 -c tflite-model/tflite-trained.cpp -o tflite-model/tflite-trained.o
[BLD] Writing templates OK
[BLD] Scheduling job in cluster...
[BLD] Job started
[BLD] Exporting TensorFlow Lite model...
[BLD] Found operators ['Conv2D', 'FullyConnected', 'MaxPool2D', 'Reshape', 'Softmax']
[BLD] Exporting TensorFlow Lite model OK
[BLD]
[BLD] Removing clutter...
[BLD] Removing clutter OK
[BLD]
[BLD] Copying output...
[BLD] Copying output OK
[BLD]
[BLD] Job started
[BLD] Building binary...
[BLD] aarch64-linux-gnu-g++ -MD -Wall -g -Wno-strict-aliasing -I. -Isource -Imodel-parameters -Itflite-model -Ithird_party/ -Os -DNDEBUG -g -DEI_CLASSIFIER_USE_FULL_TFLITE=1 -Iedge-impulse-sdk/tensorflow-lite -Dfloat16_t=float -std=c++14 -c source/main.cpp -o source/main.o
[BLD] aarch64-linux-gnu-g++ -MD -Wall -g -Wno-strict-aliasing -I. -Isource -Imodel-parameters -Itflite-model -Ithird_party/ -Os -DNDEBUG -g -DEI_CLASSIFIER_USE_FULL_TFLITE=1 -Iedge-impulse-sdk/tensorflow-lite -Dfloat16_t=float -std=c++14 -c tflite-model/tflite-trained.cpp -o tflite-model/tflite-trained.o
ions/arm_mat_inverse_f32.o edge-impulse-sdk/CMSIS/DSP/Source/MatrixFunctions/arm_mat_cmplx_mult_q31.o edge-impulse-sdk/CMSIS/DSP/Source/MatrixFunctions/arm_mat_sub_q15.o edge-impulse-sdk/CMSIS/DSP/Source/MatrixFunctions/arm_mat_sub_f32.o edge-impulse-sdk/CMSIS/DSP/Source/MatrixFunctions/arm_mat_inverse_f64.o edge-impulse-sdk/CMSIS/DSP/Source/MatrixFunctions/arm_mat_sub_q31.o edge-impulse-sdk/CMSIS/DSP/Source/MatrixFunctions/arm_mat_trans_q15.o edge-impulse-sdk/CMSIS/DSP/Source/MatrixFunctions/arm_mat_init_q15.o edge-impulse-sdk/CMSIS/DSP/Source/MatrixFunctions/arm_mat_add_q15.o edge-impulse-sdk/CMSIS/DSP/Source/MatrixFunctions/arm_mat_init_f32.o edge-impulse-sdk/CMSIS/DSP/Source/MatrixFunctions/arm_mat_add_f32.o edge-impulse-sdk/CMSIS/DSP/Source/MatrixFunctions/arm_mat_mult_f32.o edge-impulse-sdk/CMSIS/DSP/Source/MatrixFunctions/arm_mat_trans_f32.o edge-impulse-sdk/CMSIS/DSP/Source/MatrixFunctions/arm_mat_trans_q31.o edge-impulse-sdk/CMSIS/DSP/Source/StatisticsFunctions/arm_mean_q7.o edge-impulse-sdk/CMSIS/DSP/Source/StatisticsFunctions/arm_std_q15.o edge-impulse-sdk/CMSIS/DSP/Source/StatisticsFunctions/arm_max_no_idx_f32.o edge-impulse-sdk/CMSIS/DSP/Source/StatisticsFunctions/arm_max_q31.o edge-impulse-sdk/CMSIS/DSP/Source/StatisticsFunctions/arm_rms_q15.o edge-impulse-sdk/CMSIS/DSP/Source/StatisticsFunctions/arm_mean_q31.o edge-impulse-sdk/CMSIS/DSP/Source/StatisticsFunctions/arm_rms_f32.o edge-impulse-sdk/CMSIS/DSP/Source/StatisticsFunctions/arm_entropy_f64.o edge-impulse-sdk/CMSIS/DSP/Source/StatisticsFunctions/arm_max_q15.o edge-impulse-sdk/CMSIS/DSP/Source/StatisticsFunctions/arm_max_q7.o edge-impulse-sdk/CMSIS/DSP/Source/StatisticsFunctions/arm_max_f32.o edge-impulse-sdk/CMSIS/DSP/Source/StatisticsFunctions/arm_std_q31.o edge-impulse-sdk/CMSIS/DSP/Source/StatisticsFunctions/arm_power_q15.o edge-impulse-sdk/CMSIS/DSP/Source/StatisticsFunctions/arm_power_q7.o edge-impulse-sdk/CMSIS/DSP/Source/StatisticsFunctions/arm_rms_q31.o edge-impulse-sdk/CMSIS/DSP/Source/StatisticsFunctions/arm_power_f32.o edge-impulse-sdk/CMSIS/DSP/Source/StatisticsFunctions/arm_mean_f32.o edge-impulse-sdk/CMSIS/DSP/Source/StatisticsFunctions/arm_power_q31.o edge-impulse-sdk/CMSIS/DSP/Source/StatisticsFunctions/arm_var_q15.o edge-impulse-sdk/CMSIS/DSP/Source/StatisticsFunctions/arm_var_f32.o edge-impulse-sdk/CMSIS/DSP/Source/StatisticsFunctions/arm_logsumexp_f32.o edge-impulse-sdk/CMSIS/DSP/Source/StatisticsFunctions/arm_std_f32.o edge-impulse-sdk/CMSIS/DSP/Source/StatisticsFunctions/arm_var_q31.o edge-impulse-sdk/CMSIS/DSP/Source/StatisticsFunctions/arm_logsumexp_dot_prod_f32.o edge-impulse-sdk/CMSIS/DSP/Source/StatisticsFunctions/arm_kullback_leibler_f32.o edge-impulse-sdk/CMSIS/DSP/Source/StatisticsFunctions/arm_min_q15.o edge-impulse-sdk/CMSIS/DSP/Source/StatisticsFunctions/arm_min_q7.o edge-impulse-sdk/CMSIS/DSP/Source/StatisticsFunctions/arm_min_f32.o edge-impulse-sdk/CMSIS/DSP/Source/StatisticsFunctions/arm_entropy_f32.o edge-impulse-sdk/CMSIS/DSP/Source/StatisticsFunctions/arm_kullback_leibler_f64.o edge-impulse-sdk/CMSIS/DSP/Source/StatisticsFunctions/arm_min_q31.o edge-impulse-sdk/CMSIS/DSP/Source/StatisticsFunctions/arm_mean_q15.o source/main.o tflite-model/tflite-trained.o edge-impulse-sdk/dsp/kissfft/kiss_fft.o edge-impulse-sdk/dsp/kissfft/kiss_fftr.o edge-impulse-sdk/dsp/dct/fast-dct-fft.o ./edge-impulse-sdk/dsp/memory.o edge-impulse-sdk/porting/posix/ei_classifier_porting.o edge-impulse-sdk/porting/posix/debug_log.o edge-impulse-sdk/porting/mingw32/ei_classifier_porting.o edge-impulse-sdk/porting/mingw32/debug_log.o  -o build/linux-impulse-runner -lm -lstdc++ -L./tflite/linux-aarch64 -ldl -ltensorflow-lite -lcpuinfo -lfarmhash -lfft2d_fftsg -lfft2d_fftsg2d -lruy -lXNNPACK -lpthread
[BLD] Building binary OK
[RUN] Downloading model OK
[RUN] Failed to run impulse [Error: EXDEV: cross-device link not permitted, rename '/tmp/tmp/ei-1625494032916Xx9q2b/runner-linux-aarch64' -> '/home/mendel/.ei-linux-runner/models/32387/v2/model.eim'] {
  errno: -18,
  code: 'EXDEV',
  syscall: 'rename',
  path: '/tmp/tmp/ei-1625494032916Xx9q2b/runner-linux-aarch64',
  dest: '/home/mendel/.ei-linux-runner/models/32387/v2/model.eim'
}

I suspect on the coral one would need to compile the TFLite model against libedgetpu.so as in here: https://coral.ai/docs/edgetpu/tflite-cpp/#build-your-project-with-libedgetpu

louis · July 6, 2021, 7:35am

From the Dashboard view in the studio, you can download the tflite file. Could you try to compile it with the documentation you provided and see if there is any issue please?

Regards,

janjongboom · July 7, 2021, 3:15pm

@kashif Actually compilation seems fine, it’s a rename that fails… If you manually copy /tmp/tmp/ei-1625494032916Xx9q2b/runner-linux-aarch64 to ~/.ei-linux-runner/models/v2/model.eim and then run:

$ edge-impulse-linux-runner --model-file ~/.ei-linux-runner/models/v2/model.eim

Does that work? If so, we can change CLI to do a copy and then a delete instead of a move.

kashif · July 8, 2021, 4:04pm

@janjongboom Yes you are right that works thanks! My question was that this model is just a regular TFLite model? How can I go about getting an edge-tpu compiled TFLIte model which is also integrated with edge-impulse?

In anycase I think with these above changes I have a basic image classification model working on Coral EdgeTPU, if you can tell me where I can submit some documentation additions I can send a PR.

janjongboom · July 9, 2021, 7:26am

@kashif Great, what’s needed to make this an edge-tpu compiled TFLite model?

If you can put some docs somewhere (either here, or in a Gist or something) @aurel and team can then integrate it in the docs.

dhruvsheth · August 14, 2021, 12:50pm

@kashif could you share the steps for installing Edge-Impulse-Cli on Google Coral dev board? Seems like I’m facing error while installation on the board.

kashif · August 22, 2021, 1:12pm

@dhruvsheth so I followed the script for installing on jetson nano, however I needed to do

sudo apt install libjpeg62-turbo-dev

apart from that it all worked…

dhruvsheth · August 26, 2021, 5:51pm

Thanks a lot @kashif