Cannot scan binary keys with Jedis

[n12c]

In org.springframework.data:spring-data-redis:2.4.5, in org.springframework.data.redis.connection.jedis.JedisKeyCommands#scan(long, org.springframework.data.redis.core.ScanOptions)[1] we round trip keys through String, corrupting them. This makes it impossible scan top level keys using Jedis.

Specifically, we call redis.clients.jedis.Jedis#scan(java.lang.String, redis.clients.jedis.ScanParams) which decodes keys as strings, interpreting the binary data as UTF-8. On the next line, we encode the strings back into byte arrays using JedisConverters.stringListToByteList(), which renders UTF-8 again. This generally results in any non-UTF-8 sequences in a binary key being replaced with efbfbd (the UTF-8 serialization of the Unicode replacement character).

Instead, we should call redis.clients.jedis.BinaryJedis#scan(byte[], redis.clients.jedis.ScanParams) and remove the second conversion using JedisConverters.stringListToByteList().

[1] https://github.com/spring-projects/spring-data-redis/blob/master/src/main/java/org/springframework/data/redis/connection/jedis/JedisKeyCommands.java#L159

[mp911de]

SCAN is somewhat contradicting the idea of non-string keys. Any format other than UTF-8 strings can potentially contain an asterisk and that's why we went for accepting String as scan input parameter. We cannot accept a typed value (e.g. ScanOptions<K>) because there are no means to express a prefix/suffix/contains match clause.

We could switch pattern to a byte array internally and call the binary SCAN method to avoid key name to UTF-8 conversion. I think we need to do the same for Lettuce as Lettuce's ScanArgs use a string to express the match clause.

[n12c]

Interesting, I hadn't considered the globbing implications. How does that comport with the implementation of SSCAN[1] though? It doesn't do the string conversion and handles binary set values without problems; the implementation there looks exactly like what (I thought) the SCAN implementation should. But, the implementation does pass through the MATCH subcommand as the SSCAN command[2] does support it. Is SSCAN matching something that only circumstantially works when the binary data involved is actually UTF-8 data?

[1] https://github.com/spring-projects/spring-data-redis/blob/master/src/main/java/org/springframework/data/redis/connection/jedis/JedisSetCommands.java#L275
[2] https://redis.io/commands/sscan

[mp911de]

For SSCAN, we already obtain the key name in its binary form. String matching happens on the set member level so prefixes of binary set members may get encoded improperly. The sscan method we invoke returns its results as byte[].