Exclusions Related Discussion: Edge Cases

[ppratikcr7]

Version where bug was found:
e.g "5.1"

Describe the bug
Q1. Should we consider Segments related exclusions or Consistency related?

Q2. Should we remove the previous user “user1” from individualEnrolment? If Yes, this will impact the Individual Consistency. If we don't remove them then we would have the data collected from their logs, which we don’t want to collect.

Q3. When should we remove those old enrolled users? During the time of updating the segment or during the time of updating the experiment where we add the exclusion segment?

Q4. If we remove the users for that particular segment group & experiment “Experiment1” during updating of segments, we need to remove users for the same segment group that were enrolled in other experiments that have the same segment group “school1” on exclusion list.

To Reproduce
Steps to reproduce the behavior:

• We have a group called “school1”.
• We enroll a user “user1” having a group & workingGroup as “school1”.
• So, we would store groupEnrolment & individualEnrolment documents in the DB.
• We now add the group segment “school1” in the exclusion list of “Experiment1”.
• We mark a user “user2” having a group & workingGroup as “school1”.
• So, we would store groupExclusion & individualExclusion documents in the DB. Also, we would delete the old groupEnrolment document.
• Also, if “user1” marks again after the group segment “school1” is on the exclusion list, we do store the individualExclusion document but don't remove the users’ individualEnrolment document. So the user seems to be both enrolled and excluded.

Expected behavior
We need more discussion on this.

[VivekFitkariwala]

Currently, when a user is excluded by a segment rule, we generate an Individual Exclusion document for that user. Additionally, in the case of a Group Experiment, if a user is excluded because they belong to the same group as the Group Experiment, we also create an exclusion document for the group. But might create a problem with the consistency rule.

cc @amurphy-cl

[amurphy-cl]

Here’s @SritterCL 's take on the issue of dealing with changing segmentation rules:

Scenario 1:

• Student 1 is in class A.
• An experiment starts, which uses group random assignment by class.
• Student 1 reaches the decision point and is given condition A.
• Then class A is excluded from the experiment.
• Case 1: student 1 reaches the decision point again. Should the student receive condition A or default?
  • If consistency is set to individual, then we prioritize keeping student 1’s experience consistent, so we give student 1 condition A
  • If consistency is set to group, we prioritize giving the group the same experience. Since we know everyone else in the class will get default, we give student 1 default.
• Case 2: A different student in class A (student 2) reaches the decision point. What happens?
  • Since the class is excluded from the experiment, student 2 gets default.

Scenario 2:

• Student 1 is in class A.
• An experiment starts, which uses group random assignment by class.
• Student 1 reaches the decision point and is given condition A.
• Then student 1 is individually excluded from the experiment.
• Case 1: student 1 reaches the decision point again. Should the student receive condition A or default?
  • In this case, since the student is individually excluded, we ignore the consistency rule and give the student the default
• Case 2: A different student in class A (student 2) reaches the decision point. What happens?
  • If the consistency rule is group, then student 2 (and all students in the class) should be excluded from the experiment. Since student 1 is getting default, all students in the class must get default.
  • If the consistency rule is individual, then student 2 gets condition A.

[amurphy-cl]

Will review on 09/26 how much time it will take for the last bit of this to be finished.