阅读背景:

Spark 2.0.0用可变模式读取json数据

来源:互联网 

I am trying to process a month's worth of website traffic, which is stored in an S3 bucket as json (one json object per line/website traffic hit). The amount of data is big enough that I can't ask Spark to infer the schema (OOM errors). If I specify the schema it loads fine obviously. But, the issue is that the fields contained in each json object differ, so even if I build a schema using one day's worth of traffic, the monthly schema will be different (more fields) and so my Spark job fails.I am trying to process a month's worth of websi




你的当前访问异常,请进行认证后继续阅读剩余内容。

分享到: