WARN: failed to process:Does all your data have the same frequency?

Report bugs
#1

I’m getting allot of this warning when flattening features.

WARN: failed to process 44494/training/0812typeDH_Session1_Shimmer_1_Calibrated_PC.csv.2dkcituf.ingestion-66fc59f86-n6ng2.json: Shape of data does not match earlier data. Expected 3388 columns, but got 2254. Does all your data have the same frequency?

We collected this data from an external device at 75Hz and imported it as CSV.
The only possible issue I can find is that there is a single instance of a missed sample where it interval is 26 ms instead of 13ms. This anomaly would seem to be a common issue since the XL typically samples on it’s own clock, the the log is timestamped by the MCUs RTC. This slight mismatch in rates will result in a periodic beat of an extra or missing sample.

I am left wondering:

  1. If the data rate must be perfect, why do we import timestamps at all?
  2. What are my choices in dealing with this?

Thanks.

#2

Hi @Panometric, remove the rogue data sample (with the wrong frequency) from the dataset and retrain. On Data acquisition you can click the ‘make larger’ button to see the data frequency:

image

  1. If the data rate must be perfect, why do we import timestamps at all?

So we know when data with a wrong frequency is added :wink: