`test_time_series_transformer` occasionally fails

This check is not reliable enough:

```py3
# Test non-permutation invariant
assert not np.allclose(out, out_perm, atol=1e-5)
```

Turning down `atol` could resolve this, but I'm open to more constructive fixes. Could also be worth investigating if this is not a deeper problem with the `TimeSeriesTransformer`.