Converting SWIN Transformers from Pytorch through ONNX to Edge Impulse

Help
#1

Question/Issue:
Struggling to Convert Pytorch → ONNX to Edge impulse

Project ID:
216047(Not very Interesting)

Context/Use case:
Thesis for University - Trying to Convert Model

Hi Everyone,

I am a student at the University of Amsterdam who is currently writing his thesis about SWIN Transformers for Motion Amplification.
I have spoken to a few people from the Edge Impulse team and I am seeking the vibrant community for some help. (Thanks in advance and thanks again Team!)

What’s the Issue

My Code can be found here
The checkpoints (Which should be put in a ckpt/ directory) can be found here.

  • Currently in Possession of a PyTorch Model (Trained in PyTorch 1.11 but also a version with 10 Epochs in PyTorch 2.0.1) converted to ONNX with set magnification factors but: The input of this model is 3 input tensors (Two tensors that are images and a magnification tensor). Model from Ricard Lato.
  • Unable to Convert this model into ONNX (unsure if ONNX is successful)

Possible Solutions / Suggestions from Me
(Open to any other ideas)

  1. Convert from PyTorch —> Tensorflow using HuggingFace such as done in there Transformer examples (Unable to link as do not have permissions to add more than 2)
  2. Convert from PyTorch —> ONNX —> Tensorflow light
  3. Convert Model from Pytorch to TensorFlow by rewriting the model from the original model reference above and retraining
  4. As Edge Impulse only supports one input tensor convert the two frames into one new tensor each time and set a standard magnification on export to ONNX.

Uncertainties

  • Is this even possible in regards to converting SWIN Transformers from PyTorch —> ONNX —> TFlite

What occurred when uploading any of the ONNX files to Edge Impulse

Traceback (most recent call last):
File "/app/convert-via-tf-onnx.py", line 68, in <module>
ei_tensorflow.onnx.conversion.onnx_to_tflite(args.onnx_file, file_float32, file_int8,
File "/app/./resources/libraries/ei_tensorflow/onnx/conversion.py", line 69, in onnx_to_tflite
raise Exception('More than 1 input tensor, not supported')
Exception: More than 1 input tensor, not supported

Conversion using onnx-tf also failed, cannot use this ONNX file - contact your solutions engineer, or post the logs on the forum.
As I mentioned in the call there are a couple of restrictions to converting models with our tooling:
- Must have 1 input tensor
- batch == 1
- You are restricted to operations that Tensorflow Lite supports (https://www.tensorflow.org/lite/guide/ops_compatibility)
If you can solve these for your model then things should work. It is possible you may have more luck converting a tensorflow model rather than oonx. I'm not an expert on these kinds of models but I'm sure we can find you some expertise on the forum!
#2

If anything is unclear please let me know so I can provide more information

#3

Hi Raoul,
Carl from our ML team just let me know he has successfully managed to convert swin_tiny_patch4_window7_224 from timm which has a single input tensor to onnx and then upload it into edge impulse.

Perhaps try this out first, then look into combining your input tensors into a single input as that looks like the main roadblock

1 Like
#4

Hi Jim,

Do you know what the framework type was that Carl used? Could do you know how he converted it into onnx from pytorch or similar? Thanks in advance.

Kind regards,

Raoul

#5

@raoulritter here’s a github gist of how he converted the model in Google Colab

1 Like