I've found that if an exception is thrown in the setUpClass or setTestName methods (annotated with @BeforeClass and @BeforeMethod respectively) of UITestRunner, then the subsequent test(s) are skipped, and so the afterMethod method doesn't execute. This meant that the Chrome session didn't get killed.
Adding alwaysRun=true to the @AfterMethod annotation seems to have resolved this issue.
Is there any downside to that which I might be missing? If not, I'm happy to submit a PR with that change in.