Since writing this I realized there already is a tool for this https://github.com/aedocw/epub2tts that will handle epub files, mine only handles txt files.

git clone https://github.com/tom-huntington/unofficial-openai-tts-cli.git
cd unoffical_openai_tts_cli
pip install .
tts_openai -h