Skip to content

S
SparkStudy
Commit 16f1d35c authored by 박준형's avatar 박준형
Browse files
Options

Update code_perf.md

parent bbc24192
Hide whitespace changes
Inline Side-by-side
Showing with 2 additions and 1 deletion
code_perf.md
View file @ 16f1d35c
CSV 파일 읽기 개선
spark.conf.set("spark.sql.files.maxPartitionBytes", 1024 * (1024 * 256)) spark.conf.set("spark.sql.files.maxPartitionBytes", 1024 * (1024 * 256))
val df = spark.read.option("header", "true") val df = spark.read.option("header", "true")
.option("io.compression.codecs", "nl.basjes.hadoop.io.compress.SplittableGzipCodec") .option("io.compression.codecs", "nl.basjes.hadoop.io.compress.SplittableGzipCodec")
.schema(schema).csv(fileName) .schema(schema).csv(fileName)
\ No newline at end of file
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment