阅读背景:

如何读取压缩的bz2 (bzip2)维基百科转储到流xml记录阅读器中,以便hadoop map减少?

来源:互联网 

I am working on using Hadoop Map Reduce to do research on the wikipedia data dumps (compressed in bz2 format). Since these dumps are so big (5 T), I can't decompress the xml data into HDFS and just use the StreamXmlRecordReader that hadoop provides. Hadoop does support uncompressing bz2 files, but it splits the pages arbitrarily and sends those to the mapper. Because this is xml, we need the splits to be a tags. Is there anyway to use the built in bz2 decompression and stream xml record reader provided by hadoop together?I am working on using Hadoop Map Reduce to do r




你的当前访问异常,请进行认证后继续阅读剩余内容。

分享到: