Spark Source code reading (draft)

Table of contents

1. Preface
2. Core Spark Workflow
   • How are DataFrames converted to RDD?
   • How are RDDs converted to Job?
   • How are jobs and stages scheduled?
   • How are tasks scheduled?
   • How does executor execute tasks?
   • How are the results sent back to the application?
3. Standalone Cluster
   • How does master manage worker & executor? (in progress)
   • How does driver programs communicate with master? (TODO)
   • How does master communicate with worker? (TODO)
4. Important Internal Infrastructure
   • What does BlockManager do? (TODO)
   • How shuffle works between stages? (TODO)
   • What is inside the cache? (TODO)
   • How are the caches managed? (TODO)
   • How does the recovery work? (TODO)
   • How does broadcast work? (TODO)
   • How does accumulator work? (TODO)
   • How is the object serialized? (TODO)
   • RPC
   • Task Pool
5. Pefromance Optimization
   • How does Spark respect locality? (TODO)
   • What are the physical plan and logical plan? (TODO)
   • What does the query optimizer do? (TODO)
   • How does code generation work? (TODO)
   • How does speculative task work? (TODO)
   • How does out-heap memory work?(TODO)
6. Streaming/Machine Learning (not planned)
7. Appendix A Minor concepts