在 Pythonmultiprocessing库中,是否有pool.map支持多个参数的变体?
multiprocessing
pool.map
import multiprocessing text = "test" def harvester(text, case): X = case[0] text + str(X) if __name__ == '__main__': pool = multiprocessing.Pool(processes=6) case = RAW_DATASET pool.map(harvester(text, case), case, 1) pool.close() pool.join()
对此的答案取决于版本和情况。JF Sebastian下面首先描述了最新版本的 Python(自 3.3 起)的最通用答案。1它使用Pool.starmap接受一系列参数元组的方法。然后它会自动解包每个元组的参数并将它们传递给给定的函数:
Pool.starmap
import multiprocessing from itertools import product def merge_names(a, b): return '{} & {}'.format(a, b) if __name__ == '__main__': names = ['Brown', 'Wilson', 'Bartlett', 'Rivera', 'Molloy', 'Opie'] with multiprocessing.Pool(processes=3) as pool: results = pool.starmap(merge_names, product(names, repeat=2)) print(results) # Output: ['Brown & Brown', 'Brown & Wilson', 'Brown & Bartlett', ...
对于早期版本的 Python,您需要编写一个辅助函数来显式解包参数。如果你想使用with,你还需要编写一个包装器来Pool变成一个上下文管理器。(感谢muon指出这一点。)
with
Pool
import multiprocessing from itertools import product from contextlib import contextmanager def merge_names(a, b): return '{} & {}'.format(a, b) def merge_names_unpack(args): return merge_names(*args) @contextmanager def poolcontext(*args, **kwargs): pool = multiprocessing.Pool(*args, **kwargs) yield pool pool.terminate() if __name__ == '__main__': names = ['Brown', 'Wilson', 'Bartlett', 'Rivera', 'Molloy', 'Opie'] with poolcontext(processes=3) as pool: results = pool.map(merge_names_unpack, product(names, repeat=2)) print(results) # Output: ['Brown & Brown', 'Brown & Wilson', 'Brown & Bartlett', ...
在更简单的情况下,使用固定的第二个参数,您也可以使用partial,但仅限于 Python 2.7+。
partial
import multiprocessing from functools import partial from contextlib import contextmanager @contextmanager def poolcontext(*args, **kwargs): pool = multiprocessing.Pool(*args, **kwargs) yield pool pool.terminate() def merge_names(a, b): return '{} & {}'.format(a, b) if __name__ == '__main__': names = ['Brown', 'Wilson', 'Bartlett', 'Rivera', 'Molloy', 'Opie'] with poolcontext(processes=3) as pool: results = pool.map(partial(merge_names, b='Sons'), names) print(results) # Output: ['Brown & Sons', 'Wilson & Sons', 'Bartlett & Sons', ...
1. 这在很大程度上受到了他的回答的启发,而他的回答可能应该被接受。但由于这个卡在顶部,似乎最好为未来的读者改进它。