TFLite external delegate with Linux C++ SDK

#1

Hi, is there a way to run or integrate my external delegate with the Linux C++ SDK?

I currently have a working .so dynamic library that uses the TensorFlow Lite external delegate mechanism.

#2

Hi @kolejnik,

Yes you can. For example you can place your <your-lib>.so in one of the subfolders in tflite/ for the target your building for. Then in Makefile you adapt the build flags for the target you’re buidling, for example to link against you’re library add LDFLAGS+= -l<your-lib>.

// Raul

#3

Hi @rjames,

If I understand correctly, first I need to compile my external delegate as a static library. Then, I should link it in the Makefile as you described, and it should select my external delegate without any changes in the C++ code. Is that correct?

I’m asking because I don’t think you can link a dynamic library (.so) as a static library (please correct me if I’m wrong).

Thanks for your help.

#4

Hi @kolejnik,

No need to build your library as a static library. You can link against your dynamic library. We do this already for example with our flex delegates (see download_flex_delegates.sh and Makefile).
Note that because it’s linking dynamically you’ll have to copy the flex delegate library also to the target (e.g. in /usr/local/lib) before execution.

#5

Hi @rjames,

I’ve linked my dynamic library, but when I run CUSTOM APP, I still see the log message INFO: Created TensorFlow Lite XNNPACK delegate for CPU.

Are we sure there aren’t any additional changes needed in the C++ code? I found the function TfLiteDelegate* TfLiteExternalDelegateCreate(const TfLiteExternalDelegateOptions* options); in the TensorFlow repository, and I’m wondering if we need to call this function in the Edge Impulse SDK to activate the external delegate.

#6

Hi @kolejnik,

Yes, I think you’ll hack it into the SDK, at least for an MVP.
Take a look at edge-impulse-sdk/classifier/inferencing_engines/tflite_tidl.h around line 94 where we dynamically load delegate and use it. Maybe it sets you in the right direction.

If you have a public repository, please share.

To see what libraries your custom app depends on run:

readelf -d build/custom | grep NEEDED

See if you’re lib is in the list.

1 Like
#7

Hi @rjames,

I successfully added the external delegate, and everything is working perfectly now.

Thank you so much for your guidance and support, I really appreciate it!