阅读背景:

使用Python进行Hadoop流式传输:手动拆分输入文件

来源:互联网 

I am new to Hadoop and am trying to use its streaming feature with Python written mapper and reducer. The problem is that my original input file will contain sequences of lines which are to be identified by a mapper. If I let Hadoop split the input file, it might do it in the middle of a sequence and, thus, that sequence will not be detected. So, I was thinking about splitting the files manually. This will also break some sequences, therefore, in addition to that I would also provide and alternative split that would create files overlapping the "first" split. That way I will not loose any sequences. I am new to Hadoop and am trying to use its str




你的当前访问异常,请进行认证后继续阅读剩余内容。

分享到: