Error when generating features

[benlim]

Creating job... OK (ID: 2086869)

Scheduling job in cluster...
Job started
Creating windows from 13 files...
[2/3] Pre-caching files...
[3/3] Pre-caching files...
Pre-caching files OK

[ 1/13] Creating windows from files...
[13/13] Creating windows from files...
 
Traceback (most recent call last):
  File "/app/input-block-preprocessing/downsample.py", line 86, in <module>
    window = sn.resample_poly(window, target_freq, sample_freq, axis=ds_axis)
  File "/usr/local/lib/python3.7/dist-packages/scipy/signal/signaltools.py", line 3089, in resample_poly
    h = firwin(2 * half_len + 1, f_c, window=window)
  File "/usr/local/lib/python3.7/dist-packages/scipy/signal/fir_filter_design.py", line 457, in firwin
    m = np.arange(0, numtaps) - alpha
ValueError: Maximum allowed size exceeded
Error windowing Downsample error code was not 0, but was 1
Downsample error code was not 0, but was 1

Application exited with code 1 (Error)

Job failed (see above)

I keep running into the above error. I reduced my sample sizes but I had no luck getting the generate features to work.

[louis]

Hello @benlim,

I just checked you project and it seems the features have been generated successfully.
Have you changed anything in your project between when you posted this message and now?

Regards,

Louis

[louis]

Also, I just noticed that your window size is smaller than your window increase, this might lead to an error.

• The window size is the size of the raw features that is used for the training
• The window increase is used to create more features (and feed the NN with more information)

I tried to draft something to explain but don’t judge my design skills

So I’d suggest that you keep the windows increase less or equal to your window size or you’ll loose information from your dataset.

I hope that was clear.

Regards,

Louis

[benlim]

Oh! So that means that I want to overlap the windows? Your drawing was quite clear!

I changed the sample period - I also noticed that the sample frequency reported by the data ingestion service is determined by the timestamp difference between the first two samples instead of the average sample rate in the dataset.

I’m still running into this error:

MemoryError: Unable to allocate array with shape (57471264367816104,) and data type int64

What should I look at to resolve this?

[louis]

Hello @benlim,

The frequency should be constant in all your data samples (but I think you figured it out, just checked your project again and you disabled the samples that were not using the 16.666…Hz frequency .
You may want to have a look at this thread too (maybe you did already): Error windowing Downsample error code was not 0, but was 1

Regards,

Louis

[janjongboom]

Hi @benlim you’ve hit the same underlying issue as Error windowing Downsample error code was not 0, but was 1 - I thought we patched this everywhere, but apparently we didn’t. Will get fixed in the next patch release!

[benlim]

@janjongboom & @louis Thanks for your quick replies! The issue is that my sampling rate has a 1 ~ 2ms jitter when sampling, so I’m not always able to hit the same interval between samples. Is there a way to downsample to the lowest frequency sample?

How would this affect the subsequent inference when I port the code over to my device?