Release version 0.4.0

[allanbank]

Currently for the 0.4.0 release the following items have already been merged:

• [hbase] Separate HBase v0.9x and v1.x bindings #392 - Separate HBase v0.9x and v1.x bindings.
• [core] Make hdrhistogram default measurement type #371 - Make hdrhistogram default measurement type.

Of particular note is that #371 introduces two incompatibilities with previous releases:

• Default measurement changed from histogram to hdrhistogram
  Users who want previous behavior can set the 'measurementtype' property to 'histogram'
• Reported 95th and 99th percentile latencies now in microseconds (previously in milliseconds)

After the RC0 release is created we need to verify that both of those changes are working as intended. Once both items are tested I will create a RC1.

Once the RC1 release has been created I will ask for everyone to test as many of the binding as possible. I think the minimum set should be (as they were tested in 0.3.0 or saw a recent update):

• Apache Accumulo 1.6.0 via the accumulo binding.
• Apache Cassandra 2.0.15 via the cassandra-cql and cassandra-10 bindings.
• Apache HBase 1.0.0 via the hbase10 binding.
• Apache HBase 0.98.x via the hbase098 binding.
• Apache HBase 0.94.x via the hbase094 binding.
• MongoDB via the mongodb and mongodb-async bindings.
• Tarantool 1.6.1 via the tarantool binding.
• Aerospike 3.5.15 via the aerospike binding.

While I will not pester anyone about testing the bindings until after the RC1 I want to encourage everyone to test as soon as RC0 is released.

In the interest of getting the release out quickly I am going to try and avoid patches unless absolutely necessary. The flip side of that is I want to give folks a chance to get more changes into the baseline so I won't do the initial RC0 build until September 21st. I will plan to do the RC1 build about a week later on September 28th and hope for a final release before October 8th. To summarize:

• RC0 - September 21st
• RC1 - September 28th
• 0.4.0 - before October 8th.

[busbey]

Sounds like a good plan.

As things currently stand, #392 also introduces an incompatible change because it replaces the behavior of the 'hbase' binding with 'hbase098'.

[busbey]

Just realized #392 also broke compat by renaming 'hbase-10' to 'hbase10'.

As an aside, back in 0.3.0 only hbase against version 1.0.0 was tested. AFAIK after we determined 0.98 and 0.94 didn't work back in 0.2.0 we didn't retest due to time.

[cmccoy]

@allanbank - thanks for doing this!

As an aside, back in 0.3.0 only hbase against version 1.0.0 was tested. AFAIK after we determined 0.98 and 0.94 didn't work back in 0.2.0 we didn't retest due to time.

That's right. Regarding the version incompatibilities: I renamed all bindings to be explicit about which version they target. Since bin/ycsb selects a binding based on the component of the database name prior to a - character, I renamed hbase-10 as well. I'd be happy to try and accommodate other suggestions prior to v0.4.

[allanbank]

0.4.0-RC1 has been cut. Let the testing begin.

For this first phase of testing please focus on the following issues:

• [hbase] Separate HBase v0.9x and v1.x bindings #392 - Separate HBase v0.9x and v1.x bindings.
• [core] Make hdrhistogram default measurement type #371 - Make hdrhistogram default measurement type.
• Race condition in workload D? #327 - Race condition between insert and read/update operations. (via PR Proposed fix for #327. #364)

I am personally going to start with #327.

Thanks,
Rob

[allanbank]

To verify #327 I ran workload D adjusted to 10,000,000 operations and 50% read / 50% insert using the mongodb-async binding and saw no return=1 values. This also used the new default hdrhistogram.

I am going to run the workload again with the measurementtype=histogram reporting to make sure that works too.

Rob.

[allanbank]

measurementtype=histogram works as expected.

That I think leaves #392.

@cmccoy @busbey - Will you have some time in the next week or so to try the different HBase bindings? Do you have access to v1.0, v0.98 and v0.94 clusters? I have a small VM farm I can throw a small cluster onto but need to know which version I should target.

Thanks,
Rob

[cmccoy]

I'm sorry - I'm just getting back from sick leave, and am a bit swamped.
FWIW, I tested against #392 against HBase 1.0.1.1, HBase 0.98.14 and 0.94.27 prior to merging, although that missed any subsequent commits to master.

[gkamat]

I'll sign up to test RC1 against the different HBase bindings. Will try to have results by mid-next-week.

[busbey]

I'm planning to do windows and accumulo testing tomorrow evening. Given the work I already did on #407 I think I can test JDBC against MySQL as well this weekend.

[busbey]

@tlopatic could we trouble you to test Aerospike again?

@bigbes how's your schedule looking for Tarantool?

[uncle-betty]

Sure! Just tested, everything as expected on Aerospike with RC1. And I like the new measurement type default.

[Thomas-Yang]

I am testing with Cassandra 2.0.15 on a local machine. It gives me the following errors:
com.yahoo.ycsb.DBException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: localhost/127.0.0.1 ([localhost/127.0.0.1] Unexpected error during transport initialization (com.datastax.driver.core.TransportException: [localhost/127.0.0.1] Channel has been closed)))
at com.yahoo.ycsb.db.CassandraCQLClient.init(CassandraCQLClient.java:158)
at com.yahoo.ycsb.DBWrapper.init(DBWrapper.java:63)
at com.yahoo.ycsb.ClientThread.run(Client.java:289)

[busbey]

I ran through the recommended workloads with Accumulo 1.6:

• Ran into known issue Accumulo binding errors during cleanup #340 again
• Additional incompatible change in hdrhistogram, following measurements are exported as floating instead of as integers:
  • number of operations
  • min latency
  • max latency
  • 95p latency
  • 99p latency
• workload d fails with

java.lang.RuntimeException: Too many unacknowledged insertion keys.
        at com.yahoo.ycsb.generator.AcknowledgedCounterGenerator.acknowledge(AcknowledgedCounterGenerator.java:44)
        at com.yahoo.ycsb.workloads.CoreWorkload.doTransactionInsert(CoreWorkload.java:769)
        at com.yahoo.ycsb.workloads.CoreWorkload.doTransaction(CoreWorkload.java:575)
        at com.yahoo.ycsb.ClientThread.run(Client.java:329)

this workload performed correctly in 0.2.0 and 0.3.0.

• noticed workload e doesn't export a measurement of SCAN operations. checked and it didn't do this in 0.3.0 or 0.2.0.

[uncle-betty]

Interesting. The exception in workload D is really odd. It's part of the fix for "inserted records are read before they've actually been inserted", #327.

How exactly (# of threads, # of records, etc.) did you run workload D to produce the exception?

A bit of background. The fix works as follows: As before, doTransactionInsert() invokes nextInt() to obtain a new key. That key would be instantaneously passed over to lastInt(), even before the insert completed. Now, with the fix, that key is only passed over to lastInt() after doTransactionInsert() calls acknowledge(), which it only does after the insert completed. So, now we never try to read keys generated by nextInt(), whose insert is still pending.

The semantics of lastInt() are "keys up to X have been inserted into the database", X being the value reported by lastInt(). What complicates the fix a little is that acknowledgements can arrive out of order. Suppose that we start three insert operations with keys 1000, 1001, and 1002. Suppose that they complete in the order 1002, 1001, 1000. Ultimately, we want lastInt() to report 1002. However, we want it to only report 1002, after 1001 and 1000 have been inserted as well.

What the fix does is to keep track of the acknowledged keys in a sliding window. That window has 1,000,000 entries. Whenever there's a contiguous chunk of N keys at the beginning of this window, we increase the value reported by lastInt() by N and also shift the window by N.

What probably happened in your case is that the window cannot be moved, because a key is missing. A thread probably invoked nextInt() and then got stuck somewhere. In the meantime, the other threads happily invoked nextInt() / acknowledge(), up to 1,000,000 times, filling up the window. This then caused the exception.

Theoretically, it could just be a thread starvation issue, where one thread invokes nextInt(), then never sees the CPU again, and thus keeps the sliding window from sliding. But that would mean that the other threads got to do 1,000,000 insertions without the starved thread getting to complete its insertion. This is definitely possible, but maybe not the most likely explanation.

I'd like to first rule out something else. Could there be an issue with the Accumulo driver? Something that locks up one of the YCSB threads, which then blocks the sliding window. It's something that we might not necessarily have noticed before, as a single blocked thread didn't have that huge an impact on a YCSB run before.

Thomas