/
dim81
/
learning-spark
Обзор
Документация
Войти
/
dim81
/
learning-spark
Код
Запросы
0
Задачи
Вики
Пакеты
0
Релизы
0
Аналитика
Безопасность
master
src/python/MakeParquetFile.py
22 строки
808 B
Holden Karau
Add something to make a parquet file and query it with spark sql
09 сен 2014, 23:15
09 сен 2014, 23:15
e42eed5
Код
Авторство
О чём код?
# Createas a parquet file and loads an input file into it # For input you can use files/favourite_animal.csv as the iput from pyspark import SparkContext from pyspark.sql import SQLContext import json import sys if __name__ == "__main__": if len(sys.argv) != 4: print "Error usage: LoadHive [sparkmaster] [inputFile] [parquetfile]" sys.exit(-1) master = sys.argv[1] inputFile = sys.argv[2] parquetFile = sys.argv[3] sc = SparkContext(master, "MakeParquetFile") sqlCtx = SQLContext(sc) # Load some data into an RDD rdd = sc.textFile(inputFile).map(lambda l: l.split(",")) namedRdd = rdd.map(lambda r: {"name": r[0], "favouriteAnimal": r[1]}) schemaNamedRdd = sqlCtx.inferSchema(namedRdd) # Save it schemaNamedRdd.saveAsParquetFile(parquetFile)