BlockManager源码分析
- key-value形式存储的数据块,用block ID表示
- 充当在Spark集群中的每个节点上运行的本地缓存(driver and executors)
SparkEnv实例化一个BlockManager对象blockManager
// NB: blockManager is not valid until initialize() is called later.
val blockManager = new BlockManager(
executorId,
rpcEnv,
blockManagerMaster,
serializerManager,
conf,
memoryManager,
mapOutputTracker,
shuffleManager,
blockTransferService,
securityManager,
numUsableCores)
调用BlockManager的主构造器(主构造器会执行类定义中的所有语句)
// 定义变量blockManagerId,初始值为null
var blockManagerId: BlockManagerId = _
......
blockManagerId =
if (idFromMaster != null)
idFromMaster
else id