阅读背景:

如何减少AWS Lambda的打包python zip文件的大小

来源:互联网 

Afternoon,

I recently came across AWS Lambda and Azure Functions. AWS imposes a limit on the size of zipped as well as unzipped files, which for python scripts need to include all of the dependent modules. I have been using lambda-uploader to package my script and it's module dependencies, but the pandas package is too big.

我最近遇到了AWS Lambda和Azure Functions。 AWS对压缩文件和解压缩文件的大小施加了限制,这对于python脚本需要包含所有相关模块。我一直在使用lambda-uploader打包我的脚本和它的模块依赖,但是pandas包太大了。

I have seen examples of people completing machine learning and using pandas on AWS Lambda (a little outdated though) but I can't see how they're doing it. Any suggestions?

我已经看过人们在AWS Lambda上完成机器学习和使用pandas的例子(虽然有点过时),但我看不出他们是怎么做的。有什么建议么?

2 个解决方案

#1


3  

The package that you upload to lambda should not contain anything but the code and support modules required for Lambda to run your code. The Lambda console UI limits the file size to 10MB but you can upload zip files up to 50MB if you place them in an S3 bucket and then request that Lambda load them from S3.

您上传到lambda的软件包不应包含Lambda运行代码所需的代码和支持模块。 Lambda控制台UI将文件大小限制为10MB,但如果将zip文件放在S3存储桶中,则可以上传最大50MB的zip文件,然后请求Lambda从S3加载它们。

Any other assets that you require for execution such as machine learning models should be uploaded separately to S3 and then downloaded from within your Lambda function at execution time. The Lambda function can write to a /tmp folder but keep in mind it only has access to 512MB of disk space. Also keep in mind that the Lambda function has a maximum runtime of 300 seconds so downloading really large files will take time away from your function doing real work with the data you're downloading.

您需要执行的任何其他资产(如机器学习模型)应单独上载到S3,然后在执行时从Lambda函数中下载。 Lambda函数可以写入/ tmp文件夹,但请记住它只能访问512MB的磁盘空间。另外请记住,Lambda函数的最大运行时间为300秒,因此下载非常大的文件需要花费一些时间来完成您正在下载的数据的实际工作。

#2


0  

Try using Zappa. Add slim_handler to true in the zappa_settings.json which you make using zappa init.

尝试使用Zappa。在使用zappa init创建的zappa_settings.json中将slim_handler添加到true。


分享到: