I am building a proof of concept system for anomaly detection and I’m at a the point where I need to upload data to EI.
The problem is I have 3 physical sensors:
accelerometer: 3 floats for acceleration, sampled at 100Hz
RMS current sensor: 1 float for Amperes, sampled at 5Hz
microphone: passed through FFT on the board and outputs bucketed frequency power (16 floats) at 4Hz
All these samples are gathered for a window of 1 sec and transmitted to a small server I’ve written. The server has a window time option that allows me to gather e.g 3 seconds of data and write it to JSON file.
My problem is now how to upload this data using the Ingestion service, since the above sensors are sampled at different rates and the data acquisition format only seems to support one sensor rate via interval_ms.
Should I create a synthetic sensor combining all 3 and use 1Hz = 1000ms as interval_ms? And how would this translate in EI (DSP blocks, axis, etc)?
Or should I break it into 3 uploads, one for each sensor at its appropriate sample rate?
I should clarify that I can already send data through the Ingestion service API with no problem.
I would have used the Data Forwarder also, but I need to intercept and store the data in another system for archival, before it is uploaded to EI.
That’s why I opted for using the EI Ingestion service.
My question was more of how to “shape” the data.
For the moment I’m uploading 3 files, one for each sensor in the format <label>.<sensor>.<uuid>.
All 3 files represent the same, 1 sec window of sensing but split up by sensor (and thus sampling rate)
I’ll try to feed these into separate DSP blocks and the DSP outputs to an NN & K-means block.
This is a good question. All sensors need to have the same sampling frequency (same interval_ms) to upload the samples.
Usually the best way is to upsample to the highest frequency, 100 Hz in your case. You can just copy-paste the last values to make it simple. For instance with your RMS sensor at 5Hz, each value will be duplicated 20 times in order to upsample to 100 Hz.
I am using Python Pandas to do this, on my backend, before uploading to EI with Ingestion.
May I suggest that this tip could be added to the documentation (e.g. in data acquisition format docs) since it probably concerns everyone that is doing sampling with more than one sensor simultaneously?