[24.1] Remove defaults channel for conda usage #29886

Job	Run time
build-client / build-client	9m 51s
setup-selenium / setup_chromedriver	5s
Test (3.8, 0)	44m 39s
Test (3.8, 1)	58m 21s
Test (3.8, 2)	55m 10s
	2h 48m 6s

Provide feedback