WASM Sound, Acceleration, Camera Demos

Rocksetta · April 4, 2021, 3:34pm

There is a really good Edge Impulse demo about using WASM (Web Assembly) on web pages, to demonstrate to people who don’t have your microcontroller what it does but on a webpage. The demo is about shower sounds and is here

This feature is really useful and not as hard as it sounds (Assembly language is very scary and web assembly just sounds extra scary), since Edge Impulse makes the WASM and you just have to ask it to classify information.

Since Github allows WebPages (Gitpages), it is very easy and useful to test things out on a Github site. @janjongboom and I proved this worked in a very different thread here

The only issue is the above demo is for sounds and I can’t find a demo for the camera or accelerometer. Anyone @janjongboom @aurel @dansitu have a link to some javascript example that uses the webcam to grab RGB and/or GRAYSCALE data from a web browser to feed to a classifier?

As I have spent many years working on TensorflowJS (hmmmm, if your not paid is it really working!) I should be able to do this using vanilla javascript, but if someone else has already done the work I would love to see the code.

I will put my demos about using WASM on my Gitpage here

aurel · April 6, 2021, 7:39am

Hi @Rocksetta,

You can maybe check this repo: https://github.com/edgeimpulse/balena-cam-tinyml.
Though it’s using a Raspberry Pi, the way to classify the image will be similar once you can capture the frame.

The image is sent through a websocket in base64 encoding, you can probably just bypass this part and directly load the image buffer using sharp, see:

github.com

edgeimpulse/balena-cam-tinyml/blob/master/edgeimpulse-inference/app/server.js#L78


const wss = new WebSocket.Server({ port: 8080 });
// Incoming websocket connection
wss.on('connection', function connection(ws) {
    ws.on('message', function incoming(message) {
        console.log('received new image');
        // Retrieve raw information from image in argument and parse it for the classifier
        let raw_features = [];
        const buf = sharp(Buffer.from(message, 'base64')).raw().toBuffer()
            .then(img_buffer => {
                let buf_string = img_buffer.toString('hex');
                
                // store RGB pixel value and convert to integer
                for (let i=0; i<buf_string.length; i+=6) {
                    raw_features.push(parseInt(buf_string.slice(i, i+6), 16));
                }
            })
            .catch(err => {
                console.error('Failed to load image', err);

Aurelien

janjongboom · April 6, 2021, 9:44am

@Rocksetta The mobile client does exactly that. See e.g.:

In essence: 1) use WebRTC to render picture, 2) pull pixels out (here), 3) run classifier as you’d do normally.

There’s also examples for accelerometer in the mobile client.

Rocksetta · April 7, 2021, 1:16am

Thanks @janjongboom and @aurel I am actually doing better than I thought I would. Is there any way from javascript to access what we have on the Arduino exported code:

EI_CLASSIFIER_INPUT_WIDTH

or height since it is always square?

If that property is public my code can auto correct for different sized models.

janjongboom · April 7, 2021, 7:26am

Yeah you can probe the WASM library for them. See getProperties() in classifier.ts in the mobile client.

Rocksetta · April 7, 2021, 10:09pm

Ever do a stupid Javascript programming mistake that takes away hours of your life, because it kind of works but the results don’t make any sense.

myOutputString = "0x77444a, 0x734247, 0x613239" // example raw RGB data

// Hours of pain
results = await classifier.classify([myOutputString]);

// Works    
results = await classifier.classify( myOutputString.split(', ')  );

Very happy now

Rocksetta · April 8, 2021, 6:01am

@janjongboom I think getProperties() is private.

The reference link is here

   var classifier = new EdgeImpulseClassifier();
   await classifier.init();
    
  //  I think this line makes it private not public
  // https://github.com/edgeimpulse/mobile-client/blob/10b9b5d163e18addd68438b8fcf753b6033e6ac5/client/classifier.ts#L27

  // the following does not run
  // let myImpulseProperties =  await classifier.get_properties()
  // console.log(myImpulseProperties)

janjongboom · April 13, 2021, 5:46am

@Rocksetta, add the following to the EdgeImpulseClassifier class:

    getProperties() {
        return Module.get_properties();
    }

Then you can do classifier.getProperties():

    let props = classifier.getProperties();

    console.log('sensor', props.sensor);
    console.log('frequency', props.frequency);
    console.log('frame_sample_count', props.frame_sample_count);
    console.log('input_width', props.input_width);
    console.log('input_height', props.input_height);

Which for a test model gives:

sensor 3
frequency 0
frame_sample_count 102400
input_width 320
input_height 320

Rocksetta · April 14, 2021, 5:16am

Thank you, that works really well. Any idea why the frequency is zero, I was expecting the average time in ms for each analysis?

Rocksetta · April 14, 2021, 7:06pm

So I needed a way to see my students assignments, so I made it really easy to view using a single index.html file and the downloaded WASM 2 files from edge impulse. (Must be loaded to an https URL, the easiest way to do that is using github GITPAGES although they are a pain to setup.)

Presently the code is really raw, (I write messy code on a good day, so when I say it is raw it is a real mess ) but it is really useful to see what your are trying to work with. I was having troubles with inverting the left and the right side and this method really helped me. Here is a students webpage. His model tested with my shoe.

janjongboom · April 19, 2021, 9:10am

@Rocksetta the frequency is 0 because there is no data frequency. An audio signal has 16000 samples per second (typically) but one image is just that: one image.

Rocksetta · April 20, 2021, 5:38am

Well that was fast to get deprecated. The new multi-object detection WASM export does not work with my browser viewer. Should not be terribly hard to fix as Javascript canvas overlays easily.

Here is a link to my old working pen-cup model here

and here is a link to the new object detection pen-cup not working model here

Any suggestions? Looks like a minor change with classification results storage is messing with the values. On second thought I can probably figure it out.

P.S. The mulit object detection is very nice.

janjongboom · April 20, 2021, 6:37am

Hi @Rocksetta, yeah should be easy. I’ve updated the WebAssembly docs here https://docs.edgeimpulse.com/docs/through-webassembly.

Essentially you’ll just get x, y, width, height properties extra which you can use to render the bounding box.

Rocksetta · April 22, 2021, 6:35pm

Amazing to do that much different analysis with such a small change in code. Really impressive @janjongboom .

Still working on getting my Web Viewer working but I at least understand what the changes are.

Rocksetta · April 27, 2021, 6:18am

@janjongboom not sure if this is a feature or a bug, but with the multi object detection WASM if there is no new classification the results object is maintained from the last successful classification and not cleared.

Fine with that as a feature, but if I wanted to have the results object cleared after an unsuccessful classification, any suggestions how?

janjongboom · April 27, 2021, 6:32am

@Rocksetta, what do you mean? The WebAssembly package returns the old bounding boxes, that does not happen for me on the mobile client which uses the same WebAssembly build… Or are you not clearing the bounding boxes on the image?

Rocksetta · April 27, 2021, 6:44am

I am still working on getting the bounding boxes in the correct place, that should be fun. That is not the issue. I will assume I have messed something up but I get these repeating results, until a new cup or pen has been found.

https://hpssjellis.github.io/my-examples-of-edge-impulse/public/edge-models/mult01/index.html

Rocksetta · June 6, 2021, 5:27am

The Blog about my version of using the WASM impulse just came out.