Error in sampling frequency detection edge-impulse-data-forwarder

paulphilip · June 23, 2020, 10:43am

@janjongboom
Hi,
In the Project 3600, I had uploaded data sampled at 104Hz and detected as 108Hz by the uploader.
Today when I ran the uploader, it is detecting as 110Hz. When I am run re-train with a single data, I am getting the following error

Creating job... OK (ID: 196551)

Retraining Spectral features...
Creating windows from 32 files...
[ 1/32] Creating windows from files...
WARN: failed to process Test_M_104: Shape of data does not match earlier data. Expected 1135 columns, but got 1153. Does all your data have the same frequency?

The model does get generated.

When I add more datasets, I am getting the following error

Creating job... OK (ID: 196563)

Retraining Spectral features...
Creating windows from 38 files...
[ 1/38] Creating windows from files...
WARN: failed to process O.1avi65ss: Shape of data does not match earlier data. Expected 1153 columns, but got 1135. Does all your data have the same frequency?
WARN: failed to process O.1avi65ta: Shape of data does not match earlier data. Expected 1153 columns, but got 1135. Does all your data have the same frequency?
WARN: failed to process O.1avi65tk: Shape of data does not match earlier data. Expected 1153 columns, but got 1135. Does all your data have the same frequency?
WARN: failed to process O.1avi65u0: Shape of data does not match earlier data. Expected 1153 columns, but got 1135. Does all your data have the same frequency?
WARN: failed to process O.1avi65t5: Shape of data does not match earlier data. Expected 1153 columns, but got 1135. Does all your data have the same frequency?
WARN: failed to process O.1avi65t4: Shape of data does not match earlier data. Expected 1153 columns, but got 1135. Does all your data have the same frequency?
WARN: failed to process O.1avi65t0: Shape of data does not match earlier data. Expected 1153 columns, but got 1135. Does all your data have the same frequency?
WARN: failed to process O.1avi65tc: Shape of data does not match earlier data. Expected 1153 columns, but got 1135. Does all your data have the same frequency?
WARN: failed to process O.1avi65tb: Shape of data does not match earlier data. Expected 1153 columns, but got 1135. Does all your data have the same frequency?
WARN: failed to process O.1avi65ps: Shape of data does not match earlier data. Expected 1153 columns, but got 1135. Does all your data have the same frequency?
WARN: failed to process O.1avi65pl: Shape of data does not match earlier data. Expected 1153 columns, but got 1135. Does all your data have the same frequency?
WARN: failed to process M.1avi65j5: Shape of data does not match earlier data. Expected 1153 columns, but got 1135. Does all your data have the same frequency?
WARN: failed to process O.1avi65jk: Shape of data does not match earlier data. Expected 1153 columns, but got 1135. Does all your data have the same frequency?
WARN: failed to process M.1avi65jf: Shape of data does not match earlier data. Expected 1153 columns, but got 1135. Does all your data have the same frequency?
WARN: failed to process M.1avi65ji: Shape of data does not match earlier data. Expected 1153 columns, but got 1135. Does all your data have the same frequency?
WARN: failed to process O.1avi65jv: Shape of data does not match earlier data. Expected 1153 columns, but got 1135. Does all your data have the same frequency?
WARN: failed to process M.1avi65ju: Shape of data does not match earlier data. Expected 1153 columns, but got 1135. Does all your data have the same frequency?
WARN: failed to process O.1avi65jp: Shape of data does not match earlier data. Expected 1153 columns, but got 1135. Does all your data have the same frequency?
WARN: failed to process M.1avi65kk: Shape of data does not match earlier data. Expected 1153 columns, but got 1135. Does all your data have the same frequency?
WARN: failed to process O.1avi65jo: Shape of data does not match earlier data. Expected 1153 columns, but got 1135. Does all your data have the same frequency?
WARN: failed to process M.1avi65jj: Shape of data does not match earlier data. Expected 1153 columns, but got 1135. Does all your data have the same frequency?
WARN: failed to process M.1avi65if: Shape of data does not match earlier data. Expected 1153 columns, but got 1135. Does all your data have the same frequency?
WARN: failed to process M.1avi65j9: Shape of data does not match earlier data. Expected 1153 columns, but got 1135. Does all your data have the same frequency?
WARN: failed to process M.1avi65im: Shape of data does not match earlier data. Expected 1153 columns, but got 1135. Does all your data have the same frequency?
WARN: failed to process M.1avi65jg: Shape of data does not match earlier data. Expected 1153 columns, but got 1135. Does all your data have the same frequency?
WARN: failed to process M.1avi65in: Shape of data does not match earlier data. Expected 1153 columns, but got 1135. Does all your data have the same frequency?
WARN: failed to process M.1avi65is: Shape of data does not match earlier data. Expected 1153 columns, but got 1135. Does all your data have the same frequency?
WARN: failed to process M.1avi65iv: Shape of data does not match earlier data. Expected 1153 columns, but got 1135. Does all your data have the same frequency?
WARN: failed to process M.1avi65ig: Shape of data does not match earlier data. Expected 1153 columns, but got 1135. Does all your data have the same frequency?
WARN: failed to process M.1avi65j2: Shape of data does not match earlier data. Expected 1153 columns, but got 1135. Does all your data have the same frequency?
WARN: failed to process M.1avi65hi: Shape of data does not match earlier data. Expected 1153 columns, but got 1135. Does all your data have the same frequency?
[38/38] Creating windows from files...
Created 56 windows: M: 40, O: 16

Scheduling job in cluster...
Job started
Creating features
[ 1/56] Creating features...
[56/56] Creating features...
Created features
Retraining Spectral features OK

Retraining NN Classifier...
Copying features from processing blocks...
Copying features from processing blocks OK

Scheduling job in cluster...
Job started
Splitting data into training and validation sets...
Splitting data into training and validation sets OK

Training model
Training on 44 inputs, validating on 12 inputs
Epoch 1/100
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/tensorflow/python/keras/engine/data_adapter.py", line 1136, in catch_stop_iteration
    yield
  File "/usr/local/lib/python3.7/site-packages/tensorflow/python/keras/engine/training.py", line 1081, in evaluate
    tmp_logs = test_function(iterator)
  File "/usr/local/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py", line 580, in __call__
    result = self._call(*args, **kwds)
  File "/usr/local/lib/python3.7/site-packages/tensorflow/python/eager/def_function.py", line 650, in _call
    return self._concrete_stateful_fn._filtered_call(canon_args, canon_kwds)  # pylint: disable=protected-access
  File "/usr/local/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 1665, in _filtered_call
    self.captured_inputs)
  File "/usr/local/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 1746, in _call_flat
    ctx, args, cancellation_manager=cancellation_manager))
  File "/usr/local/lib/python3.7/site-packages/tensorflow/python/eager/function.py", line 598, in call
    ctx=ctx)
  File "/usr/local/lib/python3.7/site-packages/tensorflow/python/eager/execute.py", line 60, in quick_execute
    inputs, attrs, num_outputs)
tensorflow.python.framework.errors_impl.OutOfRangeError:  End of sequence
	 [[node IteratorGetNext (defined at home/train.py:55) ]] [Op:__inference_test_function_1055]

Function call stack:
test_function


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/train.py", line 413, in <module>
    main_function()
  File "/home/train.py", line 358, in main_function
    X_train, X_test, Y_train, Y_test, len(X_train), classes)
  File "/home/train.py", line 55, in train_model
    model.fit(train_dataset, epochs=100, validation_data=validation_dataset, verbose=2, callbacks=callbacks)
  File "/usr/local/lib/python3.7/site-packages/tensorflow/python/keras/engine/training.py", line 66, in _method_wrapper
    return method(self, *args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/tensorflow/python/keras/engine/training.py", line 872, in fit
    return_dict=True)
  File "/usr/local/lib/python3.7/site-packages/tensorflow/python/keras/engine/training.py", line 66, in _method_wrapper
    return method(self, *args, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/tensorflow/python/keras/engine/training.py", line 1088, in evaluate
    callbacks.on_test_batch_end(step, logs)
  File "/usr/local/lib/python3.7/contextlib.py", line 130, in __exit__
    self.gen.throw(type, value, traceback)
  File "/usr/local/lib/python3.7/site-packages/tensorflow/python/keras/engine/data_adapter.py", line 1153, in catch_stop_iteration
    self._inferred_steps))
TypeError: unsupported operand type(s) for *: 'int' and 'NoneType'

Application exited with code 1 (Error)

Job failed (see above)

I downgraded from 1.7.0 to 1.6.6 and still I am facing error.

Kindly advise on how to handle this scenario.

janjongboom · June 23, 2020, 11:16am

Hi @paulphilip, once the frequency starts to change we can’t properly generate windows anymore, consistent frequency is key here. You can fix the frequency by opening ~/edge-impulse-config.json and setting it there yourself, but if it differs too much it’ll reset again.

How are you setting the frequency to 104Hz? I think you’re not consistently writing data if this differs between runs. If you’re on Mbed I’d do something like this:

        int64_t sample_length = 10000; // in milliseconds
        int64_t interval = 1000000 / 104; // in nanoseconds

        int64_t break_at = timer.read_us() + (sample_length * 1000);

        Timer timer;

        timer.start();

        while (1) {
            int64_t next_tick = timer.read_us() + interval;

            // do sensor reading things here

            // let's sleep for (wait_time / 1000) - 1 ms. then busy loop from there
            uint64_t wait_time = next_tick - timer.read_us();

            // sleep OK (/1000 already floors it)
            ThisThread::sleep_for((wait_time / 1000) - 1);

            // busy loop til next tick
            while (next_tick > timer.read_us());

            if (timer.read_us() > break_at) break;
        }

paulphilip · June 23, 2020, 11:27am

@janjongboom
I have configured the Accelerometer every 104Hz to generate a GPIO interrupt whenever data is ready. This is to ensure that the data upload frequency doesn’t drift much. Can you share the location of edge-impulse-config.json in Windows 10?

paulphilip · June 23, 2020, 11:32am

Hi I found the edge-impulse-config.json on `C:\Users<user_name> folder. Is editing that be sufficient?

janjongboom · June 23, 2020, 11:39am

Should be yes.

I have configured the Accelerometer every 104Hz to generate a GPIO interrupt whenever data is ready. This is to ensure that the data upload frequency doesn’t drift much.

Interesting, and sounds like the right approach. I’m still wondering why the data forwarder sees the drift though.

davwang · July 21, 2021, 3:03am

I have also found a similar problem.

davwang · July 21, 2021, 3:04am

微信截图_20210721110430

davwang · July 21, 2021, 3:10am

It is great, I have tried this method.

janjongboom · July 21, 2021, 8:32am

Hi @davwang and others that read this thread. We’ll be adding support for normalizing frequencies for timeseries data in the studio in the very near term, this won’t fix the underlying issue, but at least the studio will do the normalization for you

davwang · August 8, 2021, 8:45am

C:\Users\19002\edge-impulse-config.json