Direct import of data from wandb Artifacts into EI Studio

[Joeri]

Hello EI-Team,

I am using W&B Artifacts to track datasets. For example, below, you see an ‘interim dataset’ (testing and training), stored as an Artifact in wandb. This data is obtained after data-preprocessing (raw data —> | data-preprocessing component | —> interim data).

Currently, I download the data to a local drive and upload it to edge impulse using edge-impulse-uploader. I am building a small pipeline to automate this.

My question: Can I import datasets stored in wandb Artifiact directly into EI studio, i.e. without downloading them to a local drive?

Thanks for the feedback.

Regards,
Joeri

[shawn_edgeimpulse]

Hi @Joeri,

Right now, you have a couple of options:

• Create a Transformation Block (Transforming research data - Edge Impulse Documentation) to automatically read/transform/upload data from another source
• Script something with the API (Ingestion API - Edge Impulse API) or CLI to automatically pull from wandb, do any necessary transforms, and upload

[Joeri]

Hi @shawn_edgeimpulse,

The implementation that I am working on is based on your second option, a combination of API and CLI:

@shawn_edgeimpulse

Script something with the API (Ingestion API - Edge Impulse API ) or CLI to automatically pull from wandb, do any necessary transforms, and upload

So I will proceed with this.

Thanks for the feedback!