/
githubmirror
/
spark
Обзор
Документация
Войти
/
githubmirror
/
spark
Код
Запросы
0
Пакеты
0
Релизы
0
Аналитика
Безопасность
master
sql/core/benchmarks/ArrowCacheBenchmark-results.txt
82 строки
7 KB
Dongjoon Hyun
[SPARK-58415][INFRA][TEST] Regenerate benchmark results
29 июл 2026, 10:51
29 июл 2026, 10:51
6b793ce
Код
Авторство
О чём код?
================================================================================================ Arrow Cache vs Default Cache ================================================================================================ ================================================================================================ Cache primitive types ================================================================================================ OpenJDK 64-Bit Server VM 17.0.20+8-LTS on Linux 6.17.0-1020-azure AMD EPYC 9V74 80-Core Processor Cache 5M rows with primitives: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative --------------------------------------------------------------------------------------------------------------------------- Default cache - write + read 1986 2009 33 2.5 397.1 1.0X Default cache - write + read (uncompressed) 1238 1260 30 4.0 247.7 1.6X Arrow cache - write + read 1397 1402 8 3.6 279.3 1.4X Arrow cache - write + read (zstd level -1) 1838 1855 23 2.7 367.7 1.1X Arrow cache - write + read (zstd level 1) 1922 1946 33 2.6 384.5 1.0X Arrow cache - write + read (zstd level 3) 2020 2024 5 2.5 404.0 1.0X ================================================================================================ Cache then filter ================================================================================================ OpenJDK 64-Bit Server VM 17.0.20+8-LTS on Linux 6.17.0-1020-azure AMD EPYC 9V74 80-Core Processor Cache 5M rows, then filter: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative ------------------------------------------------------------------------------------------------------------------------ Default cache - filter 1698 1708 14 2.9 339.6 1.0X Default cache - filter (uncompressed) 1429 1446 24 3.5 285.9 1.2X Arrow cache - filter 1548 1555 10 3.2 309.6 1.1X Arrow cache - filter (zstd level -1) 1961 1969 11 2.5 392.3 0.9X Arrow cache - filter (zstd level 1) 1981 2004 32 2.5 396.3 0.9X Arrow cache - filter (zstd level 3) 2063 2067 5 2.4 412.6 0.8X ================================================================================================ Cache columnar input (Parquet) ================================================================================================ OpenJDK 64-Bit Server VM 17.0.20+8-LTS on Linux 6.17.0-1020-azure AMD EPYC 9V74 80-Core Processor Cache 2M rows from Parquet: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative ----------------------------------------------------------------------------------------------------------------------------- Default cache - columnar input 1648 1652 6 1.2 824.1 1.0X Default cache - columnar input (uncompressed) 1458 1479 30 1.4 728.9 1.1X Arrow cache - columnar input 1482 1501 26 1.3 741.2 1.1X Arrow cache - columnar input (zstd level -1) 1722 1724 2 1.2 861.0 1.0X Arrow cache - columnar input (zstd level 1) 1768 1772 6 1.1 883.8 0.9X Arrow cache - columnar input (zstd level 3) 1788 1811 33 1.1 893.9 0.9X ================================================================================================ Re-cache Arrow cached data (zero-copy test) ================================================================================================ OpenJDK 64-Bit Server VM 17.0.20+8-LTS on Linux 6.17.0-1020-azure AMD EPYC 9V74 80-Core Processor Re-cache 2M rows (zero-copy): Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative -------------------------------------------------------------------------------------------------------------------------------- Default cache - cache a cached DF 411 425 11 4.9 205.4 1.0X Default cache - cache a cached DF (uncompressed) 211 228 31 9.5 105.3 2.0X Arrow cache - cache a cached DF (zero-copy) 135 147 16 14.8 67.5 3.0X Arrow cache - cache a cached DF (zstd level -1) 332 339 6 6.0 166.2 1.2X Arrow cache - cache a cached DF (zstd level 1) 346 351 4 5.8 172.8 1.2X Arrow cache - cache a cached DF (zstd level 3) 355 372 24 5.6 177.3 1.2X ================================================================================================ Cache with column pruning (select 1 of 20 columns) ================================================================================================ OpenJDK 64-Bit Server VM 17.0.20+8-LTS on Linux 6.17.0-1020-azure AMD EPYC 9V74 80-Core Processor Cache 5M rows, select 1 column: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative ----------------------------------------------------------------------------------------------------------------------------- Default cache - select 1 of 20 columns 11451 11500 69 0.4 2290.2 1.0X Default cache - select 1 of 20 (uncompressed) 4335 4510 248 1.2 867.0 2.6X Arrow cache - select 1 of 20 5185 5241 80 1.0 1036.9 2.2X Arrow cache - select 1 of 20 (zstd level -1) 8336 8383 67 0.6 1667.2 1.4X Arrow cache - select 1 of 20 (zstd level 1) 8353 8376 33 0.6 1670.5 1.4X Arrow cache - select 1 of 20 (zstd level 3) 8590 8671 114 0.6 1718.0 1.3X