Profiling of DS2 on paddle cloud

[wanghaoshuang]

DeepSpeech2 on PaddleCloud

Paddle Cloud GPU

Tesla P40

Experiments

cudnn

id	parallelism	GPU	batch_size	time(sec)	reader threads
1	1	1	256	OOM error	8
2	1	1	256	39524	16
3	1	4	256	9770	8
4	1	4	256	12481	16
5	2	4	256	5217	8
6	2	4	128	6003	8
7	2	4	128	6978	16

• id: experiment id
• parallelism: number of machines(k8s ins?) using for trainning
• GPU: number GPU using in one k8s instance
• batch_size: batch size in each k8s instance
• time: time per pass
• reader threads: thread num of reading and processing data in xmap

[wanghaoshuang]

@Yancey1989 Can you help me explain the meaning of parallelism option in paddle cloud submit cmd?k8s instance or physical machine?

[Yancey0623]

Sure, parallelism is the trainer process count.