阅读背景:

使用python 3.6将多个文件并行加载到内存中的最佳方法是什么?

来源:互联网 

I have 6 large files which each of them contains a dictionary object that I saved in a hard disk using pickle function. It takes about 600 seconds to load all of them in sequential order. I want to start loading all them at the same time to speed up the process. Suppose all of them have the same size, I hope to load them in 100 seconds instead. I used multiprocessing and apply_async to load each of them separately but it runs like sequential. This is the code I used and it doesn't work. The code is for 3 of these files but it would be the same for six of them. I put the 3rd file in another hard disk to make sure the IO is not limited.I have 6 large files which each of them contain




你的当前访问异常,请进行认证后继续阅读剩余内容。

分享到: