Hide stripped test implementations by default on the results web site

[bhauer]

For context, we have an classification attribute named "Implementation Approach," and the two options are Realistic and Stripped. This attribute is an attempt to distinguish implementations that are representative of the real-world (realistic) from implementations that have been purposefully tuned to the particulars of our benchmark (stripped).

Judging the implementation approach is, by its nature, a matter with some nuance. There is considerable gray area between a realistic test and a test that has been carefully hand-tuned to our test cases. For example, for Framework X, is a routing trie considered realistic or stripped? We expect classification to be determined on a case-by-case basis, and in most cases in reaction to analysis of the implementation by the community.

Despite the inevitable ambiguity, we have already identified some implementations that we believe qualify as "stripped." (And one more we plan to switch imminently.)

There have been conversations since the addition of the Implementation Approach attribute that ask a valid question: "Why even include stripped tests? What is the value in doing so?"

My general demeanor in accepting test implementations has been very tolerant. To date, we've only rejected a small number of test implementations. I've been of the opinion that as long as the data can be filtered to the options that are relevant to the reader, more data is not a bad thing. That said, we want the results to represent a high-water mark for production-grade deployments, and a stripped test is probably not production-grade. So the argument for not showing this particular portion of the results data is quite compelling.

My current thinking is that we should hide stripped implementations by default within the results web site. The data remain available if the reader elects to unhide stripped tests, but they won't clutter the default view.

I'm going to leave this open for a few days before I make any changes just on the off chance there are strong opposing opinions here.

[bhauer]

This change has been deployed to the results web site.

[zhong-j-yu]

@bhauer - my framework bayou is marked as stripped. I think that is a mistake. Is there any room for discussion, or it is entirely your discretion?

Here's my argument if it matters. As far as I can see, your decision is probably based on the two config parameters I set:

https://github.com/TechEmpower/FrameworkBenchmarks/blob/master/frameworks/Java/bayou/src/main/java/bayou/BayouServer.java

    System.setProperty("bayou.http.server.pipeline", "true" ); // favor pipelined requests
    System.setProperty("bayou.http.server.fiber",    "false"); // fiber not needed in this app

However, they are entirely reasonable.

The 2nd parameter fiber is a high level concept that is not present in comparable frameworks, and it is probably not needed in most applications. While it's enabled by default, it should be disabled in this situation.

The 1st parameter pipeline determines whether to favor processing pipelined requests in the same connection, or to favor processing other requests in other connections. There is no definitive answer to that; by default my framework prefers the latter; many other frameworks prefer pipelined requests. I don't think there is any "cheating" if I set this option in line with comparable frameworks.