Using LSTM on accelerometer raw data

Hi,

I have been trying to use LSTM network to train on my raw accelerometer data. I made changes to the NN Classifier by changing the shape of train_dataset and validation_dataset from 2D to 3D (LSTM requirement). Then, I added an LSTM layer, a dropout layer, a dense layer and an output layer. I added model.compile and model.fit to train the network.

After the above changes, when I click on “Start Training”, nothing is returned from server. It shows no error and job fails as no data is returned from the server.

It will be great if someone can help me out on this.

Hi @ravi_bhushan, which project is this for? I don’t see any LSTM layers in the two projects that I checked.

Hi @janjongboom, I was wokring on “ACCELEROMETER_FOKLIFT_MOVE_STOP V1 V3 V1” .

@dansitu Any idea why the training process does not log anything on project 9189 (I’ve added you there)? I see that TF starts, but nothing is logged until eventually the container is evicted. Tried adding some extra callbacks but that does not yield anything either…

Hi @ravi_bhushan,

During and after training, our system uses the train and validation datasets in train_dataset and validation_dataset to profile and optimize your model. These variables are instances of the tf.data.Dataset class which are set up to stream data from disk. They are the same variables that should be passed into Keras during the training process.

In your code, you’re creating two new datasets based on these—which have a different format—and passing them into model.fit. You then disable the BatchLoggerCallback callback This causes problems for a number of reasons:

  • Without the logger callback, Edge Impulse Studio no longer has a mechanism for keeping track of the training progress, which is why you see no logs printed
  • Since our post-training profiling and optimization code use the original train_dataset and validation_dataset variables, their data format is not compatible with the new format you have defined in your code, which will cause the process to fail
  • Your dataset manipulation code results in the entire dataset being loaded into RAM, which is likely to fail for large datasets

In addition, even if training were successful, the change in input data format will result in other failures across Edge Impulse Studio and the on-device SDK, since it will expect your model to accept the data format returned from the DSP block you are using rather than the format that you define in your Python model training code.

Here are some tips that may help:

  • Avoid disabling the batching of datasets, or the logging callback. If something about their behavior is not compatible with your workflow, let us know and we’ll help you figure out a solution.

  • If you need to change the input data format for your model, try to do this by adding a Reshape layer at the beginning of your model. This means that the change in format will be baked into your model and will work across Studio and on the device you are deploying to.

  • Alternatively, you might consider creating a custom DSP block that outputs the data in your desired format, meaning you won’t need to reshape the data at all.

  • If you decide that you need to transform your data in other ways during training, for example to apply data augmentation, use the map function of tf.data.Dataset to do so without reading the entire dataset into RAM. For example, the following would add one to each element in the training dataset:

    train_dataset = train_dataset.map(lambda x: x + 1,
      num_parallel_calls=tf.data.experimental.AUTOTUNE)
    

It’s also worth noting that right now, LSTM models are not supported by EON Compiler or TensorFlow Lite. This means that if you’re aiming to deploy to a microcontroller you may run into issues. You can expect LSTM support to arrive some time in 2021.

Thank you for being an advanced Edge Impulse user, and I’m sorry for the above limitations! I’d love to hear what you’re trying to achieve with your dataset manipulation and see if we can figure out a way to do it within the platform.

Warmly,
Dan

Hi @dansitu and @janjongboom,

Thank you very much for your help. I was able to train the model by adding a Reshape layer. As Dan mentioned, since TensorFlow Lite does not support LSTM, the process failed during quantization.

We were trying to improve our model’s accuracy. Just wanted to check how LSTM performs on our raw data. We tried Spectral Analysis and Flatten before. The model was 86% accurate. Will try with a different set of features.

Thank you,
Ravi

@ravi_bhushan Yeah, interesting. We currently use TFLite internally to drive things like the Model Testing / Live Classification pages so models that use unsupported ops cannot be trained. You could export as iPython notebook => train yourself to do a quick validation or observe the val_accuracy in the final step of the logs (it will be close to reported accuracy later on) to see if this improves over your previous approaches
.

@janjongboom Thanks. Got the model accuracy in logs.

Sounds good. We’ll work on improving the experience for unsupported model architectures so that we make the issue clear and still provide as much information as possible.