WASM Sound, Acceleration, Camera Demos

There is a really good Edge Impulse demo about using WASM (Web Assembly) on web pages, to demonstrate to people who don’t have your microcontroller what it does but on a webpage. The demo is about shower sounds and is here

This feature is really useful and not as hard as it sounds (Assembly language is very scary and web assembly just sounds extra scary), since Edge Impulse makes the WASM and you just have to ask it to classify information.

Since Github allows WebPages (Gitpages), it is very easy and useful to test things out on a Github site. @janjongboom and I proved this worked in a very different thread here

The only issue is the above demo is for sounds and I can’t find a demo for the camera or accelerometer. Anyone @janjongboom @aurel @dansitu have a link to some javascript example that uses the webcam to grab RGB and/or GRAYSCALE data from a web browser to feed to a classifier?

As I have spent many years working on TensorflowJS (hmmmm, if your not paid is it really working!) I should be able to do this using vanilla javascript, but if someone else has already done the work I would love to see the code. :flushed:

I will put my demos about using WASM on my Gitpage here

1 Like

Hi @Rocksetta,

You can maybe check this repo: https://github.com/edgeimpulse/balena-cam-tinyml.
Though it’s using a Raspberry Pi, the way to classify the image will be similar once you can capture the frame.

The image is sent through a websocket in base64 encoding, you can probably just bypass this part and directly load the image buffer using sharp, see:

Aurelien

1 Like

@Rocksetta The mobile client does exactly that. See e.g.:

In essence: 1) use WebRTC to render picture, 2) pull pixels out (here), 3) run classifier as you’d do normally.

There’s also examples for accelerometer in the mobile client.

1 Like

Thanks @janjongboom and @aurel I am actually doing better than I thought I would. :grinning: Is there any way from javascript to access what we have on the Arduino exported code:

EI_CLASSIFIER_INPUT_WIDTH

or height since it is always square?

If that property is public my code can auto correct for different sized models.

Yeah you can probe the WASM library for them. See getProperties() in classifier.ts in the mobile client.

1 Like

Ever do a stupid Javascript programming mistake that takes away hours of your life, because it kind of works but the results don’t make any sense.

myOutputString = "0x77444a, 0x734247, 0x613239" // example raw RGB data

// Hours of pain
results = await classifier.classify([myOutputString]);

// Works    
results = await classifier.classify( myOutputString.split(', ')  ); 

Very happy now :grinning:

@janjongboom I think getProperties() is private.

The reference link is here

   var classifier = new EdgeImpulseClassifier();
   await classifier.init();
    
  //  I think this line makes it private not public
  // https://github.com/edgeimpulse/mobile-client/blob/10b9b5d163e18addd68438b8fcf753b6033e6ac5/client/classifier.ts#L27

  // the following does not run
  // let myImpulseProperties =  await classifier.get_properties()
  // console.log(myImpulseProperties)

@Rocksetta, add the following to the EdgeImpulseClassifier class:

    getProperties() {
        return Module.get_properties();
    }

Then you can do classifier.getProperties():

    let props = classifier.getProperties();

    console.log('sensor', props.sensor);
    console.log('frequency', props.frequency);
    console.log('frame_sample_count', props.frame_sample_count);
    console.log('input_width', props.input_width);
    console.log('input_height', props.input_height);

Which for a test model gives:

sensor 3
frequency 0
frame_sample_count 102400
input_width 320
input_height 320
1 Like

Thank you, that works really well. Any idea why the frequency is zero, I was expecting the average time in ms for each analysis?

So I needed a way to see my students assignments, so I made it really easy to view using a single index.html file and the downloaded WASM 2 files from edge impulse. (Must be loaded to an https URL, the easiest way to do that is using github GITPAGES although they are a pain to setup.)

Presently the code is really raw, (I write messy code on a good day, so when I say it is raw it is a real mess :slight_smile: ) but it is really useful to see what your are trying to work with. I was having troubles with inverting the left and the right side and this method really helped me. Here is a students webpage. His model tested with my shoe.

1 Like

@Rocksetta the frequency is 0 because there is no data frequency. An audio signal has 16000 samples per second (typically) but one image is just that: one image.

1 Like

Well that was fast to get deprecated. The new multi-object detection WASM export does not work with my browser viewer. Should not be terribly hard to fix as Javascript canvas overlays easily.

Here is a link to my old working pen-cup model here

and here is a link to the new object detection pen-cup not working model here

Any suggestions? Looks like a minor change with classification results storage is messing with the values. On second thought I can probably figure it out.

P.S. The mulit object detection is very nice.

Hi @Rocksetta, yeah should be easy. I’ve updated the WebAssembly docs here https://docs.edgeimpulse.com/docs/through-webassembly.

Essentially you’ll just get x, y, width, height properties extra which you can use to render the bounding box.