I am trying to deploy the trained network on my hardware by creating C++ library. After several trials about setting parameters, I noticed that the C++ library created for the trained model does not work on the hardware only when I set “Normalization Window Size” to zero. If it is set to “101”, everything is fine and I can see the results on the hardware output. Here is the point:
My ideal model is when the Normalization Window Size is set to zero. Because when the training is completed and I test the trained network via “Live Classification” section of Edge Impulse website (by selecting Show options/ Use your computer and switching to classification mode and giving permission to the desired audio source), it can output my desired results.
When I go for deployment and create C++ library and deploy it on my hardware, it outputs only noise with a constant value of 0.9906 for everything.
Only when I change the Normalization Window Size to 101 (default), and create C++ library, it works on my hardware; however, the results are not as accurate as when it is set to zero (I mean as accurate as the results that I receive from the Live Classification section when it is set to zero).
I wonder why C++ library does not work properly on the hardware when the Normalization Window Size is set to zero (which means disabling normalization)? This is how when I train the model considering Normalization Window Size of zero and test it via Live Classification of Edge Impulse website, it works perfectly.
I feel there might be a bug in creating C++ library when this parameter is set to zero. Just one point, I tested many cases by changing all possible parameters, and I ended up with the issue is with only setting Normalization Window Size to zero.
I do not know how to share a link of my project with you? It is a project created under an account registered in Edge Impulse website. My project has source data uploaded along with settings for MFCC and NN Classifier parameters.
Please let me know how do you guys usually share a link?
Just go to any page while you have your EI project loaded in your web browser. It can be any page (Deployment, DSP, etc) Copy the link from the browser and paste it here.
1- I noticed the NN was no longer trained, so I trained it in your project.
2- Do you happen to still have the zip file with the library you deployed? Did you deploy as int8? Live classification runs as float32, so sometimes there can be a modest discrepancy.
I’m still going to run one sample of yours myself to see what kind of results I get. But just wanted to ask about int8.
Since my reply contains attachments which I could not attach here, I tried to reply to your post via email. Please let me know if you received my reply along with two screenshots and the Zip file?
I just wanted to give you an update. I tried to build C++ libraries for deployment on my hardware with two following scenarios:
Considering Model Version: int8 in NN Classifier and setting the optimization for NN Classifier on deployment section to int8 again. The generated Zip file was used in the hardware, and it was detecting everything as noise with the confidence level of 0.9906.
Considering Model Version: float32 in NN Classifier and setting the optimization for NN Classifier on deployment section to unoptimized(float32) again. The generated Zip file was used in the hardware, and it was detecting everything as noise with the confidence level of 1 this time.
In both scenarios, the model was trained considering Normalization Window Size equal to zero. I also, tested it with Live Classification (as said before), and Live Classification can distinguish between noise and cough very well (in the case of putting Normalization Windows Size = 0). I think something is happening when C++ library is used to generate the zip file.
I’m unable to reproduce your error running locally. Here is a zip file you can try:
If you’re running on a Mac, you can run the included binary:
./build/edge-impulse-standalone features.txt
If you need to build it, from the root directory:
make clean
make -j
I tried several windows from your test sample “cough” and results matched. If you can find a window that doesn’t match, can you tell me which one and I will try it?
Where are you copying the input test data from? Can you send a screen shot?
Also, what are you testing on? Locally or on device? If device, which one?
Thank you for the update. Since I cannot attach file here, I sent an email to you containing all steps that I am following along with the required screenshots.
Please see if my explanations and screenshots help.
Let me know how that zip file works out on your device! In the meantime, here are the answers to your questions:
Thank you. I will test the zip file sent by you on my hardware. a few questions:
The size of the zip file you sent is 31 MB, while the ones that I generate are usually less than 5 MB. Is there any difference in your build?
That zip file is not the same deployment you get from the studio. It INCLUDES all of that code, but it’s also all of the code in example-standalone-inferencing, and also, I included my build folder in case you’re on Intel mac, and then you can just run the example out of the box.
What do you mean by window size? I follow the steps below:
What I mean is, I tested with some of your data from your project. In your “test” set (not the training one), you have two files: noise, and cough. Those files are much larger than the window size. So if you look at “Live Classification” you’ll see a whole list of results. This is the result of chopping your sample into “windows”, the size of which you set during impulse design. Each window has its own classification result. I tested a few different windows by copying the Raw Features from Studio and passing them to the example-standalone program (see here for instructions)
Thank you for early response. I just tested it. The zip file worked on my hardware; however, the output results are not promising. It detects the simple speech voice as cough too (the results as what I get in case of putting the Normalization Window Size equal to 101).
For the window size, can you please see the screenshots that I sent via email? Those are all settings that I chose during “Create Impulse”, “MFCC” and “NN Classifier” sections.
My guess is that the Live Classification Section is making local normalization on received data from the chosen microphone regardless of setting Normalization Window Size to zero or 101. However, my hardware does not do any normalization on the received data and just passes the buffer of data to the trained network. Since I do the training considering the Normalization Window Size of zero, therefore, if there is no normalization, the range of data amplitude might be more than a threshold and that is why it is detecting everything as a constant value of noise (0.9906). Is that correct?
The zip file worked on my hardware; however, the output results are not promising.
Sorry, what do you mean by “output results are not promising?” Do the results you get by running previously acquired samples disagree with the results you see in Studio? If you have a sample that has a discrepancy between SDK and Studio, please post it here and I will try it.
Normalization happens the same way in Studio vs SDK. If you set it to zero, there will be no normalization in training, Live Classification, or in the SDK.
Now, it could be that when you run your classifier on your hardware, the model could be running correctly, but not necessarily accurately. Without any normalization, you may have issues when dealing with all the variety you see in the field
Try a normalization window of 5 or 11 and see if that improves real world performance.