Extracted path tokenization logic to an independent function so it can
be tested directly and reduce duplicate regex code in another function.