阅读背景:

如何为AWS Lambda捆绑Python

来源:互联网 

I have a project I'd like to run on AWS Lambda but it is exceeding the 50MB zipped limit. Right now it is at 128MB zipped and the project folder with the virtual environment sits at 623MB and includes (top users of space):

我有一个项目我想在AWS Lambda上运行,但它超过了50MB的压缩限制。现在它的压缩为128MB,带有虚拟环境的项目文件夹位于623MB并包含(空间的顶级用户):

  • scipy (~187MB)
  • pandas (~108MB)
  • numpy (~74.4MB)
  • lambda_packages (~71.4MB)

Without the virtualenv the project is <2MB. The requirements.txt is:

如果没有virtualenv,该项目<2MB。 requirements.txt是:

click==6.7
cycler==0.10.0
ecdsa==0.13
Flask==0.12.2
Flask-Cors==3.0.3
future==0.16.0
itsdangerous==0.24
Jinja2==2.10
MarkupSafe==1.0
matplotlib==2.1.2
mpmath==1.0.0
numericalunits==1.19
numpy==1.14.0
pandas==0.22.0
pycryptodome==3.4.7
pyparsing==2.2.0
python-dateutil==2.6.1
python-dotenv==0.7.1
python-jose==2.0.2
pytz==2017.3
scipy==1.0.0
six==1.11.0
sympy==1.1.1
Werkzeug==0.14.1
xlrd==1.1.0

I deploy using Zappa, so my understanding of the whole infrastructure is limited. My understanding is that some (very few) of the libraries do not get uploaded so for e.g. numpy, that part does not get uploaded and Amazon's version gets used that is already available in that environment.

我使用Zappa进行部署,因此我对整个基础架构的理解有限。我的理解是,一些(很少)的库不会上传,例如numpy,那部分没有上传,亚马逊的版本在该环境中已经可用。

I propose the following workflow (without using S3 buckets for slim_handler):

我建议使用以下工作流程(不使用sladhandler的S3存储桶):

  1. delete all the files that match "test_*.py" in all packages
  2. 删除所有包中与“test _ * .py”匹配的所有文件

  3. manually tree shake scipy as I only use scipy.minimize, by deleting most of it and re-running my tests
  4. 手动树摇晃scipy因为我只使用scipy.minimize,删除大部分并重新运行我的测试

  5. minify all the code and obfuscate using pyminifier
  6. 缩小所有代码并使用pyminifier进行混淆

  7. zappa deploy

Or:

  1. run compileall to get .pyc files
  2. 运行compileall以获取.pyc文件

  3. delete all *.py files and let zappa upload .pyc files instead
  4. 删除所有* .py文件,让zappa上传.pyc文件

  5. zappa deploy

I've had issues with slim_handler: true, either my connection drops and the upload fails or some other error occurs and at ~25% of the upload to S3 I get Could not connect to the endpoint URL. For the purposes of this question, I'd like to get the dependencies down to manageable levels.

我遇到了关于slim_handler的问题:是的,我的连接断开,上传失败或发生其他一些错误,并且上传到S3的约25%我得到无法连接到端点URL。出于这个问题的目的,我希望将依赖关系降低到可管理的水平。

Nevertheless, over half a gig of dependencies with the main app being less than 2MB has to be some sort of record.

然而,主应用程序小于2MB的超过一半的依赖关系必须是某种记录。

My questions are:

我的问题是:

  1. What is the unzipped limit for AWS? Is it 250MB or 500MB?
  2. AWS的解压缩限制是多少?是250MB还是500MB?

  3. Am I on the right track with the above method for reducing package sizes?
  4. 我是否采用上述减少包装尺寸的方法走上正轨?

  5. Is it possible to go a step further and use .pyz files?
  6. 是否可以更进一步使用.pyz文件?

  7. Are there any standard utilities out there that help with the above?
  8. 是否有任何标准实用程序可以帮助解决上述问题?

  9. Is there no tree shaking library for python?
  10. 是不是有用于python的树摇动库?

1 个解决方案

#1


0  

  1. The limit in AWS is for unpacked 250MB of code (as seen here https://hackernoon.com/exploring-the-aws-lambda-deployment-limits-9a8384b0bec3)
  2. AWS中的限制是针对解压缩的250MB代码(如https://hackernoon.com/exploring-the-aws-lambda-deployment-limits-9a8384b0bec3所示)

  3. I would suggest going for second method and compile everything. I think you should also consider using serverless framework. It does not force you to create virtualenv which is very heavy.
  4. 我建议去第二种方法并编译所有内容。我认为你也应该考虑使用无服务器框架。它不会强迫你创建非常沉重的virtualenv。

I've seen that all your packages can be compressed up to 83MB (just the packages).

我已经看到你的所有软件包都可以压缩到83MB(只是软件包)。

My workaround would be:

我的解决方法是:

  1. use serverless framework (consider moving from flask directly to API Gateway)
  2. 使用无服务器框架(考虑从flask直接移动到API网关)

  3. install your packages locally on the same folder using:

    使用以下方法在本地安装您的软件包:

    pip install -r requirements.txt -t .
    
  4. try your method of compiling to .pyc files, and remove others.

    尝试编译.pyc文件的方法,并删除其他文件。

  5. Deploy:

    sis deploy
    
  6. 部署:sis deploy

Hope it helps.

希望能帮助到你。


分享到: