Tests are failing on Windows

Windows tests added in https://github.com/jupyter-server/jupyter_server/commit/79fbf801c5908f4d1d9bc90004b74cfaaeeed2df are failing. I would think fixing this might be warranted given that https://github.com/advisories/GHSA-hrw6-wg82-cm62 pertains Windows specifically?