Initial plan

[tylergu]

It is hard to come up with the list of operators we want to study, so we decided to study one operator(spark-operator) at first. By studying the first operator, we hope to:

• Gain experience in finding bug reports that are within our interest, e.g. what key words to use to query Github or JIRA
• Have a better idea of what operators we are interested in
• Have a better idea of how to profile each bug report, e.g. root cause, triggering

After studying one or two operators, we should be able to

1. finalize our list of operators to study
2. construct a pool of bug reports
3. randomly sample from the pool and dig into each bug report

[tianyin]

Sounds good.

On the other hand, I believe that this is based on the assumption that there are typically only 10+ bug related issues for an operator so you will study all of the bugs.

[tylergu]

@tianyin , In the case of the spark-operator, there are only 12 closed issues with the bug tag. But I realized that the developers are not carefully tagging the issues, i.e. some issues are bugs and within our interest, but not tagged with the "Bug" tag. I glanced and already found several of them on the first page of the issues (e.g. this failure happened in production environment). So I think there are much more than 12 bugs in this spark-operator.

But I will first study the 12 tagged bug reports (we can discard those outside of our interest), as they are low-hanging fruit. Then we will have much clearer vision.

On the other hand, these operator-repos varies largely in number of bugs, typically a popular one would have ~40 issues tagged with 'Bug'. We will have much smaller pool than the x-system failure study, but still can not study all the bugs.

[tianyin]

Your goal is to study a representative number of issues instead of all the cases :) So, be strategic about how many cases you want to study for each operator.

some issues are bugs and within our interest, but not tagged with the "Bug" tag

I see. It's hard to know how many are like that. If they are following some rigorous SE process, the number could be small; otherwise, the number could be even larger than 12.

so we decided to study one operator(spark-operator) at first

It's always good to start from something you are very familiar with, such as an operator tested in the Sieve project.