Error windowing Downsample error code was not 0, but was 1

[alvaro]

Hello,

I am new to Edge Impulse and I am certainly doing something wrong but have no idea what it is. What I have is 20 files with sensor data like this to train my model:

timestamp,exercise,sample,ACC X,ACC Y,ACC Z,GYR X,GYR Y,GYR Z,MAG X,MAG Y,MAG Z
-26045, 1, 20, -162, 203, 994, 0, -31, -4, -111, -411, -211
-26039, 1, 20, -144, 197, 961, -7, -27, 0, -109, -412, -222

Each file has the records for one sample and has (roughly) between 90 and 120 lines. These are the readings for a patient moving her leg up and down. I want to train a model for anomaly detection.

This is what I have done so far:

1. Create new project selecting “Something else”.

2. In “Data acquisition”, I uploaded 18 files using the “Upload data” option. My files are in csv format. (It took me a lot of time to figure out how to pass the timestamp. It would be great to have more detailed specifications about the data format required by the system).

3. “Upload data” screen, I have tried both “Automatically split between training and testing” and “Training”, having similar issues. For label, I wrote “Ex_1”.

4. I click "Begin Upload and it works fine.

5. Go to “Create impulse” and this is where all my problems start. The default values on screen are: Axes: 1000ms, Window increase: 1000ms, Frequency (Hz): 166.66666666666666, Zero-pad data: Selected.

6. Add “Spectral Analysis” as processing block. (Tried with Flatten and Spectogram and got the same issues).

7. Add “Anomaly Detection (K-means)” as learning block.

8. Click on “Save Impulse.”

9. Go to “Spectral Features”. These are the parameters I found:
   

   

10. Click on “Save parameters”.

11. In “Generate features”, I get this:
    Data in training set: 12s
    Classes: 1 (Ex_1)
    Window length: 1000 ms.
    Window Increase: 1000 ms.
    Training windows: 0

12.Click on “Generate features” and I get this error:

Creating job... OK (ID: 1821099)

Scheduling job in cluster...
Job started
Creating windows from 18 files...
[ 0/18] Creating windows from files...
[ 1/18] Creating windows from files...
[18/18] Creating windows from files...
 
[ 1/18] Resampling windows...
Traceback (most recent call last):
  File "/app/input-block-preprocessing/downsample.py", line 86, in <module>
    window = sn.resample_poly(window, target_freq, sample_freq, axis=ds_axis)
  File "/usr/local/lib/python3.7/dist-packages/scipy/signal/signaltools.py", line 3089, in resample_poly
    h = firwin(2 * half_len + 1, f_c, window=window)
  File "/usr/local/lib/python3.7/dist-packages/scipy/signal/fir_filter_design.py", line 457, in firwin
    m = np.arange(0, numtaps) - alpha
MemoryError: Unable to allocate array with shape (166666666666666656,) and data type int64
Error windowing Downsample error code was not 0, but was 1
Downsample error code was not 0, but was 1

Application exited with code 1 (Error)

Job failed (see above)

What am I doing wrong?

Thanks for your help.

Alvaro

[alvaro]

After experimenting with many options and variables, I found that for an unknown reason, not all data files are accepted even though they are very similar. On page “Spectral features”, Edge Impulse shows this error message under “DSP result”:

Error: Unable to allocate array with shape (166666666666666656,) and data type int64

I checked the data but found no inconsistencies. What can be the reason? In this step I had to disable about 25% of the files. (Suggestion: It would be great if there is a way to see and disable at once all files that have this problem. I had to select one file at a time in the “Spectral features” page to see if it generated the error and then go to “Data acquisition” to disable it.). Once all files displayed graphs correctly, I moved to the next step.

Then, in the “Generate features” page, I still got more errors. The only way I found to discover which files were causing the problem was to disable all files and add one file at a time to see if it broke the feature generation. After I completed this process, I had disabled 13 files out of 28 for training, and 2 out of 13 for testing. As I said before, I still do not understand why they are not accepted (any ideas?).

The project can be found here:

Dashboard - Just ACC - Edge Impulse

[louis]

Hello @alvaro,

I just had a quick look at your project and it seems that the frequency is not constant:

vs:

Do you think you can fix this?
If not you can use the smallest frequency by downsampling your data samples here:

Regards,

Louis

[janjongboom]

Hi, this looks like a bug in our downsampling code. Investigating!

edit: @alvaro the underlying issue is with the many digits behind the decimal in your sampling frequency (e.g. 142.85714285714286) as to normalize frequencies we have to get integers which we do by moving the comma to the back => 14285714285714286. However, that’s a lot of memory Will fix this by limiting the max. resolution here while downsampling. In the short term you can fix this by changing the interval_ms in your samples (e.g. Export data => modify => Upload again) to be normalized to e.g. 0.007042253521126761 -> which will be 142Hz which doesn’t probably have this issue.

[alvaro]

Thank you for replying, Louis. I will analyze the frequencies applying Jan’s idea to see if that solves the problem.

[alvaro]

Hi, Jan. Thank you very much for replying. I am analyzing the firmware in C++ and my code in Python searching for the place where I can set the frequency.

[janjongboom]

@alvaro This has now been fixed in our codebase, and will be released during the next patch release (probably tomorrow).

[stencht]

Hi @janjongboom, same issue here, but the workaround works.

I exported my code and checked all the interval times, which vary between 15ms and 18ms.
I modified all the interval times to be the same value (15ms), uploaded the files again and I can confirm that it now works

PS: I tried uploading the exported files without modifying the interval time (so containing 15,16,17 or 18ms), but this didn’t solve it. Not sure if this matters, but just saying !

[alvaro]

Hi Jan,

Thank you for replying and most importantly for fixing the system. I tried uploading the data and worked fine. However, I am still having issues generating features using the same files. I got that it should be corrected by adjusting the frequency, but it was not easy to do on the sensor side. It would be much easier just to generate more samples and then disable those that cannot be processed in Edge Impulse.

For that reason, I just took my batch of files and started again. I finally could identify the files that cannot be processed, but you can see in this YouTube video how cumbersome this task is. I hope this helps to continue improving this wonderful system!

Regards,

Alvaro

[alvaro]

I removed columns ‘exercise’ and ‘sample’ and it worked better. Then, after the fix, the error under DSP result does not show up. However, as I mentioned in my reply to Jan, I am still having issues generating features. I have to remove some files to make it work. I guess this is due to the frequencies as you said but changing that in my system is not easy. For that reason, the only option I see so far is just removing any file that has an unacceptable frequency rate. This is the error I get:

Creating job... OK (ID: 1863312)

Scheduling job in cluster...
Job started
Creating windows from 15 files...
[ 0/15] Creating windows from files...
[ 1/15] Creating windows from files...
[15/15] Creating windows from files...
 
Traceback (most recent call last):
  File "/app/input-block-preprocessing/downsample.py", line 86, in <module>
    window = sn.resample_poly(window, target_freq, sample_freq, axis=ds_axis)
  File "/usr/local/lib/python3.7/dist-packages/scipy/signal/signaltools.py", line 3089, in resample_poly
    h = firwin(2 * half_len + 1, f_c, window=window)
  File "/usr/local/lib/python3.7/dist-packages/scipy/signal/fir_filter_design.py", line 457, in firwin
    m = np.arange(0, numtaps) - alpha
MemoryError: Unable to allocate array with shape (166666666666666656,) and data type int64
Error windowing Downsample error code was not 0, but was 1
Downsample error code was not 0, but was 1

Application exited with code 1 (Error)

Job failed (see above)

It can be reproduced by enabling any of the disabled files in my project:

Spectral features - Project Ex 1 - Edge Impulse