Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

preprocess on nuscenes #11

Open
KYYOnull opened this issue Jun 24, 2024 · 1 comment
Open

preprocess on nuscenes #11

KYYOnull opened this issue Jun 24, 2024 · 1 comment

Comments

@KYYOnull
Copy link

maps/
| ├── basemaps/
| ├── expansion/
| ├── prediction/
are there no files in these three dirs?

i have .pngs and .jsons in these dirs but when preproccessing i have error like this:

python src/datascripts/dataloader_nuscenes.py --DATAROOT .\nuScenes\ --STOREDIR .\preprocessed\

Traceback (most recent call last):
File "src/datascripts/dataloader_nuscenes.py", line 959, in
dataset.extract_multiprocess()
File "src/datascripts/dataloader_nuscenes.py", line 895, in extract_multiprocess
each.start()
File "D:\work\anaconda\envs\car\lib\multiprocessing\process.py", line 121, in start
self._popen = self._Popen(self)
File "D:\work\anaconda\envs\car\lib\multiprocessing\context.py", line 224, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
File "D:\work\anaconda\envs\car\lib\multiprocessing\context.py", line 327, in _Popen
return Popen(process_obj)
File "D:\work\anaconda\envs\car\lib\multiprocessing\popen_spawn_win32.py", line 93, in init
reduction.dump(process_obj, to_child)
File "D:\work\anaconda\envs\car\lib\multiprocessing\reduction.py", line 60, in dump
ForkingPickler(file, protocol).dump(obj)
AttributeError: Can't pickle local object 'NuScenesData.extract_multiprocess..run'
Traceback (most recent call last):
File "", line 1, in
File "D:\work\anaconda\envs\car\lib\multiprocessing\spawn.py", line 116, in spawn_main
exitcode = _main(fd, parent_sentinel)
File "D:\work\anaconda\envs\car\lib\multiprocessing\spawn.py", line 126, in _main
self = reduction.pickle.load(from_parent)
EOFError: Ran out of input

@marcelmmm
Copy link

The problem is the run function inside of the extract_multiprocess function: AttributeError: Can't pickle local object 'NuScenesData.extract_multiprocess..run'. Pickle cannot handle this. It should work if you move the definition of the run ftc outside of the extract_multiprocessing fct.

def run(self, queue: multiprocessing.Queue, queue_res: multiprocessing.Queue):
        process_id = queue.get()
        if process_id == self.args['cores'] - 1:
            li = list(range(process_id * self.data_per_core, self.data_num))
        elif process_id is not None:
            li = list(range(process_id * self.data_per_core, (process_id + 1) * self.data_per_core))

        for idx in tqdm(li):
            a = self.__getitem__(idx)
            if not self.args['img_only']:
                if a is None:
                    pass
                else:
                    data_compress = zlib.compress(pickle.dumps(a))
                    queue_res.put(data_compress)

def extract_multiprocess(self):
    """
    the parallel process of extracting data
    """
    ex_list = []

    queue = multiprocessing.Queue(self.args['cores'])
    queue_res = multiprocessing.Queue()

    processes = [Process(target=self.run, args=(queue, queue_res)) for _ in range(self.args['cores'])]

    for each in processes:
        each.start()

    for i in range(self.args['cores']):
        queue.put(i)
        # pbar.update(1)

    while not queue.empty():
        pass

    save_pbar = tqdm(range(self.data_num))
    if not self.args['img_only']:
        for _ in save_pbar:
            try:
                a = queue_res.get(block=True, timeout=20)
                ex_list.append(a)
                save_pbar.update(1)
            except Exception as e:
                break

    for each in processes:
        each.join()

    print("all thread end")

    print("length of ex list: ", len(ex_list))
    if not self.args['img_only']:
        os.makedirs(self.data_dir, exist_ok=True)
        if 'train' in self.args['split']:
            with open(os.path.join(self.data_dir, 'ex_list'), 'wb') as f:
                pickle.dump(ex_list, f)
        elif 'val' in self.args['split']:
            with open(os.path.join(self.data_dir, 'eval.ex_list'), 'wb') as f:
                pickle.dump(ex_list, f)
    print("dump finished!")

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants