Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: Pool 资源泄漏 #59

Merged
merged 2 commits into from
Apr 5, 2024
Merged

fix: Pool 资源泄漏 #59

merged 2 commits into from
Apr 5, 2024

Conversation

helsonxiao
Copy link
Contributor

@helsonxiao helsonxiao commented Apr 4, 2024

改动如下:

  1. 调整 Pool 的创建顺序,在 __init__ 最后 run 方法内执行创建操作,避免被 exception 打断导致资源泄漏
  2. 唯一对外的 run 增加 try & except & finally 处理,增加可靠性

@helsonxiao
Copy link
Contributor Author

@virusdefender @mikucat0309 大佬们看看这个 bug

@mikucat0309
Copy link
Member

mikucat0309 commented Apr 5, 2024

感谢 PR,的确引发例外会导致 Pool 没有被清理掉。

这里提出一个想法:
让 JudgeClient 实作 context manager,把 Pool 生命周期管理放在 __enter__()__exit__(),搭配 with as 语句,引发例外也会自动清理,这样会比单独 try except 好些?

@helsonxiao
Copy link
Contributor Author

由于评测结束后需要阻塞等待所有结果,run 方法内总是需要执行 closejoin 操作,同时我们假定异常是小概率情况的话,通过 __exit__ 来关闭 pool 的做法会使得每次评测都多出一些额外不必要的 pool 操作,而且这样还需要修改两三处对于JudgeClient 的调用,个人感觉这个改动不够内聚,没有必要扩大修改范围。

如果说要调整的话,其实我更倾向于把 self._pool 改到 run 方法内部,如下:

    def run(self):
        tmp_result = []
        result = []
        pool = Pool(processes=psutil.cpu_count())
        try:
            for test_case_file_id, _ in self._test_case_info["test_cases"].items():
                tmp_result.append(pool.apply_async(_run, (self, test_case_file_id)))
        except Exception as e:
            raise e
        finally:
            pool.close()
            pool.join()
            print('join done', tmp_result)
        for item in tmp_result:
            # exception will be raised, when get() is called
            # # http://stackoverflow.com/questions/22094852/how-to-catch-exceptions-in-workers-in-multiprocessing
            result.append(item.get())
        return result

PR 现有的改动可以视为方案 A,context manager 的方案可以视作方案 B,我这边新提出的可以视为方案 C,辛苦看下哪种是可被合并的,我再进行调整 @mikucat0309

@mikucat0309
Copy link
Member

有道理,Pool 甚至不需要放在实例变数,那就方案 C 吧,再麻烦你了

@helsonxiao
Copy link
Contributor Author

@mikucat0309 修改好咯,请再看看

@mikucat0309 mikucat0309 merged commit b28aa56 into QingdaoU:master Apr 5, 2024
@helsonxiao helsonxiao deleted the fix-pool branch April 6, 2024 04:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants