以下是在各个数据量级针对同个查询语句的消耗时间
select type,count(*) as count from test group by type order by count desc;
mysql 600W 3s
sparksql 550W 5s
mysql 1000W 5.4s
sparksql 1100W 6.3s
mysql 1900W 9.9s
sparksql 2000W 8.7s
可以看得出,当数据量比较大的时候,spark的优势就体现出来了
备注:spark集群服务器配置为双核4G内存。集群配置为4CPU和4G内存
另附几组数据
mysql 2700W 14.4s
mysql 4885W 25.2s
mysql 7900W 40.8s
mysql 1E 52.2s
spark 2000W 12.1s
spark 4185W 15.3s
spark 6278W 20.4s
spark 8370W 24.3s
spark 1E 28.1s