How to load object detection dataset from .npy file?

#1

Hi Team,
You guys have been a great support. Really appreciate your work. Hope I’m not bothering you with too many questions. :grinning:

I have annotated the dataset with the help of edge impulse for object detection problem. I want to train the model locally, so I have downloaded the image training data and image training labels which is in .npy format.

But when trying to load the dataset using

X = np.load('ei-test_tube_detection-image-X.npy')
Y = np.load('ei-test_tube_detection-image-y.npy')

Im facing the following error:

---------------------------------------------------------------------------
    ValueError                                Traceback (most recent call last)
    C:\Users\Public\AppData\Local\Temp/ipykernel_3364/3608968536.py in <module>
          1 X = np.load('ei-test_tube_detection-image-X.npy')
    ----> 2 Y = np.load('ei-test_tube_detection-image-y.npy')

    C:\Program Files\Python39\lib\site-packages\numpy\lib\npyio.py in load(file, mmap_mode, allow_pickle, fix_imports, encoding)
        443             # Try a pickle
        444             if not allow_pickle:
    --> 445                 raise ValueError("Cannot load file containing pickled data "
        446                                  "when allow_pickle=False")
        447             try:

    ValueError: Cannot load file containing pickled data when allow_pickle=False

when I changed allow_pickle=True, the following error shows up.

 ---------------------------------------------------------------------------
UnpicklingError                           Traceback (most recent call last)
C:\Program Files\Python39\lib\site-packages\numpy\lib\npyio.py in load(file, mmap_mode, allow_pickle, fix_imports, encoding)
    447             try:
--> 448                 return pickle.load(fid, **pickle_kwargs)
    449             except Exception as e:

UnpicklingError: invalid load key, '{'.

The above exception was the direct cause of the following exception:

OSError                                   Traceback (most recent call last)
C:\Users\Public\AppData\Local\Temp/ipykernel_3364/2394030490.py in <module>
      1 X = np.load('ei-test_tube_detection-image-X.npy')
----> 2 Y = np.load('ei-test_tube_detection-image-y.npy',allow_pickle=True)

C:\Program Files\Python39\lib\site-packages\numpy\lib\npyio.py in load(file, mmap_mode, allow_pickle, fix_imports, encoding)
    448                 return pickle.load(fid, **pickle_kwargs)
    449             except Exception as e:
--> 450                 raise IOError(
    451                     "Failed to interpret file %s as a pickle" % repr(file)) from e
    452 

OSError: Failed to interpret file 'ei-test_tube_detection-image-y.npy' as a pickle

How to load the dataset properly?

Thanks and Regards,
Ramson Jehu K

1 Like
#2

Hi @Ramson,

No worries! The labels (y) are actually stored in JSON format (and not npy). This is a bug with how the studio names the dataset files for object detection projects. I’ve let the development team know.

You can use a JSON parser to read the file. For example:

import json
with open("ei-test_tube_detection-image-y.npy") as f:
  data = json.load(f)

Best,

Shawn

1 Like
#3

Hi @shawn_edgeimpulse,

Thank you so much for the answer. Appreciate your work.

Regards,
Ramson Jehu K

1 Like