阅读背景:

AWS Lambda解压缩gzip文件,无需将文件保存在本地

来源:互联网 

I'm trying to get a file from a S3 bucket (.gzip) and unzip it to another bucket. I couldn't find a way to do it without saving the file on local (my PC). Is there a way to 'save' the file on Lambda and unzip it directly on S3? Thank you!

我正在尝试从S3存储桶(.gzip)获取文件并将其解压缩到另一个存储桶。如果不将文件保存在本地(我的电脑)上,我找不到办法。有没有办法在Lambda上“保存”文件并直接在S3上解压缩?谢谢!

1 个解决方案

#1


1  

Here's a lambda code for example (gist):

这是一个lambda代码,例如(gist):

let path = require('path');
let aws = require('aws-sdk');
let s3Client = new aws.S3();
let zlib = require('zlib');
let s3s = require('s3-streams');

const output_bucket = "stackoverflow-bucket";

exports.handler = (event, context, callback) => {
    context.callbackWaitsForEmptyEventLoop = false;

    event.Records.forEach(record => {
        const params = {
            Bucket: record.s3.bucket.name,
            Key: record.s3.object.key
        };

        const isGzip = path.extname(params.Key) === ".gz";
        let readStream = s3Client.getObject(params).createReadStream();

        readStream = isGzip ? readStream.pipe(zlib.createGunzip()) : readStream;
        writeStream = s3s.WriteStream(s3Client, { Bucket: output_bucket, Key: path.basename(params.Key, ".gz") });

        // begins the actual streaming
        readStream.pipe(writeStream);

        writeStream.on('end', () => {
            callback(null, `Handled ${JSON.stringify(params)}`);
        });
    });
};

Note that this code uses a 3rd party library for streaming bytes to S3 (which is not natively supported by the Node.JS SDK).

请注意,此代码使用第三方库将字节流式传输到S3(Node.JS SDK本身不支持)。

For that, the documentation page here, which describes how you should package your lambda before uploading it to AWS.

为此,此处的文档页面描述了在将lambda上传到AWS之前应如何打包它。

You can set a S3 event to trigger your lambda whenever a new file is put to your source bucket:

每当将新文件放入源存储桶时,您可以设置S3事件以触发lambda:


分享到: