I tried to applied sensor fusion to garbage classification :
I have (in different projects) :
a collection of measure of the electrical capacity of different objects (a collection of labeled time series - *.csv files)
a collection of the sound emitted by the object when it falls into the container ( a collection of labeled time series - *.wav files)
a collection of images of objects ( a collection of labeled *.jpg files).
My idea is to “combine” these 3 features (the image, the sound and the electrical capacity) to determine the nature of an object (paper, plastic, metal, glass, cardboard).
my idea is to use 3 “preprocess blocks” to generate the feature associated to these 3 variables and then feed an unique neural network (NN classifier) with all the generated features.
I read the tutorial “Smoke detection” but after reading I have difficulties to adapt these use case to my own use case.
Can you help me to achieve this task (if it is feasible with Edge Impulse…) ?
All constructive comments will be warmly welcomed…!
Unfortunately, we do not support image + audio / time series for sensor fusion at the moment.
If you want to combine audio + time series, you can combine them into Edge Impulse Data Acquisition format with multiple axis although I understand it can require some more work to integrate your data.
Please see: https://docs.edgeimpulse.com/reference#data-acquisition-format
Yeah an easy way would be to extract the int16 values from the WAV file (16,000 samples) and then lay them out in the CSV file as well (resample electrical capacity to 16000Hz too).
Regarding the sensor fusion with the 3 features (image + electrical capacity + sound).
Do you think it is feasible manually outside Edge Impulse in Python ?
My idea is :
to recreate the features extraction process of an image (performed currently by the “image” preprocess block),
to recreate the features extraction of the audio file (performed currently by the “MFCC” preprocess block),
extract the features of the time serie of the electrical capacity record.
Then I want to feed an unique deep-learning model with all the features generated above.
I will be happy to have your opinion on this possible methodology ==> Do you think it is feasible ?
In this case, can you describe what is exactly performed in the “image” preprocessing block ?
(regarding the preprocessing block of the .wav file, I think I’m able to reproduce what is performed in Edge Impulse, thanks to the Librosa python’s library…)
This ones are the official processing blocks that you can see in the studio, if you are looking at the C++ integration, they are present in our inferencing SDK: https://github.com/edgeimpulse/inferencing-sdk-cpp
Let us know if you managed to make it work for your project.
@janjongboom, do you think this would be feasible in Edge Impulse if @krukiou creates a custom processing block taking the 3 sensors as an input to generate his features or having image + time serie / audio would definitely be too complicated at the moment?