Using teams in RAMP

[rth]

This is an issue to start the discussion on what would be necessary to use teams in RAMP, following a discussion with @kegl .
Having teams would be also quite useful for teaching, where students are often working in pairs.

Current situation

A short overview, I might have missed something, if so please comment.

Users sign up and are approved individually. They get allocated to teams with just them automatically. Then this 1 person team, requests to join an event, makes submissions and is displayed in the leaderboard.

Generally teams support is partially implemented, and because so far 1 person team is more or less the same as a user, team and users can sometimes be used interchangeably across the API.

DB wise, we have following tables,

• users : general user information
• teams : defines the team name and the admin user
• event_teams : defines team / event membership.

I couldn't find a table mapping users to teams. There is something called initiator_id, acceptor_id in the teams table but that would be enough for teams of 2 at most.

Option 1

If we wanted to make it work with minimal changes. We could,

1. Add UI to create groups (with approval from all members) before joining events
2. When signing up give the choice to sign up as a 1 person teams or as any other team one is part of.

Drawbacks

• The main limitation I see with this approach is that we loose all information about who make which submissions within a group. That's not great particularly for teaching.
• In terms of UI having to setup teams before having access to the event description is maybe not great for people discovering RAMP.
• If one changes teams in the middle of the event, all earlier submissions will still be shown as made by a 1 person team.
• Legally it's also not OK, because since we don't know who contributed what code in a team, we won't be able to remove it if a team member deletes their account. Unless we delete all submissions by a team, which not very user friendly.

Backward compatibility

• This would likely require minimal changes in the DB, possibly just adding a user_teams correspondence table would be enough.

Option 2

That I find conceptually easier, but that might require some (possibly repetitive) changes: all actions on RAMP are made by users not teams, including even sign-up, submissions etc.

Once in the event, users can create a team that's valid only for the current event. It impacts only how submissions are labeled on the leaderboard. There is an option to switch between individual and team view on the leaderboard. If a user deletes their account their submission are removed, which might impact the best score of the team they were in.

Drawbacks

• code replacements to use users not teams everywhere
• possibly require a bit more work from admins to validate N users than N/<team_size> teams.
• DB migration (cf below)

Backward compatibility

• main issue here is DB migration. As now submissions and event participation are based on teams not users, the corresponding fields would need renaming and updating. Rename table event_teams to event_users, etc.

Still I feel option 2 would be preferable, aside from the DB migration, I find that it's easier and there are fewer things that could go wrong.

[agramfort]

do you have the bandwidth to do this @rth ?

…

[kegl]

Thanks @rth! Lot's of thoughts, not very structured yet.

1. My original idea of initiator/acceptor was an asymmetric UI where one team (initiator) asked another (acceptor) to join. It allows for more than 2 members by recursion. The top acceptor would be the "head" of the team. It works for the DB structure, the problem was that the recursive links made some simple functions (like n_users) slow. To be checked, because other than that it's a nice solution.
2. Some requirements: I think the challenge description is available without signing up, no? In this case teams can decide their composition beforehand, and submit as a team.
3. For us it doesn't matter who in the team submits, but maybe it does for teaching.
4. On forming on teams of existing teams after signing up, what we should think through is how to design the rules. We usually limit the number of submission by defining a time out. We could allow team formation when the team itself doesn't yet have had more than one submission a day, or instead (or besides: timeout is also useful for limiting server load) of time out limit the number of submissions.

It's an exciting design problem :).

[kegl]

Some comments since it seems that we'll have the green light soon for the permanent Huawei RAMP site:

1. I think we should not have a "user view", the meaning of the team is that they work together, so all submissions belong to the full team. Kaggle has the same policy.
2. I think we should forget about the old initiator/acceptor idea, and add a users_team table. Migration should be relatively painless for the existing DBs with individual user teams, though some manual migration coding may be necessary.
3. I'm undecided on the team formation protocol. One possibility is to have a period before the challenge to form teams, possibly when they sign up for the event. We could allow team-reforming (uniting two teams). If it happens before any of the teams (including individual user teams) submit, we could simply delete the old team. Kaggle allows forming teams during the challenge, in which case we would need to verify that the unified team doesn't exceed the number of submissions and/or allotted total training time. It's 1) complicated, and 2) allows team formation tactics towards the end that I don't like that much. So I'm leaning towards making a simple rule that teams can be formed only by teams (including unique users) which have not yet submitted to the challenge.
4. For Saclay student data camps I would simply limit the size of the teams (set in the event admin UI) to 1. That also could be the default, so you would see no change in your data camps (team formation UI would not show up in the menu if 1) the team size is already at the limit or 2) the team has already submitted).

[agramfort]

if you have the engineering ressources to do this no objection

…

[rth]

I think we should not have a "user view", the meaning of the team is that they work together, so all submissions belong to the full team. Kaggle has the same policy.

OK, that would work, and that already what's happening in some parts of the code except that we don't have a good user <-> teams mapping. Assuming that Terms and Conditions of Use specify that they contribute code to the team/RAMP and it's not considered private information. I guess it's the same on Github, one can remove ones account but comments and code contributions will not be removed.

I think we should forget about the old initiator/acceptor idea, and add a users_team table. Migration should be relatively painless for the existing DBs with individual user teams,

+1 and it would be indeed the simplest migration wise.

I'm undecided on the team formation protocol.
We could allow team-reforming (uniting two teams)

Yes, this would require more thoughts. The easiest would be IMO just allow individuals to create teams no do an hierarchy of teams (while still keeping their 1 person team there for other events). Also in practical terms suppose there is a team of 3 and another team of 3, it would seem surprising to me that a single member can merge those two teams without asking anything from other team members (and without possibility to go back). Users would then be contributing on behalf of the team without having given their consent to be part of that team which is not ideal. It would easier of each participants just asks to join a team and existing team to approve I think.

we could simply delete the old team

Well but if it's an individual teams it might have submissions in other events, and/or user might want to do submissions in the future so it's not that straightforward. Might be better to keep the individual teams around.

So I'm leaning towards making a simple rule that teams can be formed only by teams (including unique users) which have not yet submitted to the challenge.

This could probably be an event option.

[kegl]

We are forming EventTeams, so the team composition is only for the particulat event. They can reuse the same team name and composition for another event, but this will not be reflected in the DB (those are two different EventTeams).

Here is an initial protocol pls comment. An eligible User will be able to sign up to an Event. Once signed up, he/she will automatically form an EventTeam containing himself/herself only. Teams can merge and leave the Eventas long as they haven’t made a Submission. Team mergers will need to be approved by all the members of the merging EventTeams. Once an EventTeam makes a Submission, it becomes immutable (no merge, joining, or leaving it).

Eligibility will be decided based on student status, year of graduation and the country of the school the student belongs to.

[rth]

I have been looking into this and here is the proposed team mechanism that minimizes changes to the DB schema, DB migrations and the current submission workflow is below.

UX

1. User signs up to an event and is approved (well technically the 1 person team does as it is now)
2. User can only have 1 current team under which they make submissions. This can be,
   • the 1 person team (as it is now) which is immutable
   • a team with multiple users
3. They can do team management in the "My teams" menu specific to each event,
   
4. The team management page for now looks as follows events/iris_test/my_teams/

5. By default, the users is part of their 1 person team (that is shared across events). They cannot leave such team, invite other users there or modify its name. It is immutable. We keep such 1 person teams as a special case, as otherwise the DB migration is daunting and it would require major changes to the workflow and code base. This special case could potentially be removed in a second iteration though. The UI of my_teams/ would be somewhat customized in the case of 1 person team.

Now under the team management the user can either create a new team, or accept invitation to a different team.

6. Once the user creates a new team or accept an invite they can,

   • Change its name

   • Invite other users to the team

   • Leave the team, in which case they fall back to their 1 person team.

     With a config option, leaving the team or joining other teams can be disallowed once the team made at least 1 submission.

7. Submission are made on behalf of their currently active team.

Note: in the end it's a somewhat different approach that in #505 (comment) because if team mergers are allowed it's not trivial how to decide,

• who and how approves team mergers
• how to reverse such mergers if there was a mistake

The current workflow is more similar to how invites to private Github repos work, which should be familiar to users.

DB schema

We keep the current DB schema, where in particular following tables exist,

• Team table: where we remove the initiator and acceptor team fields.
• EventTeam table: that does a many 2 many mapping between team and events. We still need many2many because 1 person teams are associated to many events. Regular teams can only be used in 1 event.

We add,

• UserTeam table: that has a many 2 many relationship between users and teams

This approach minimizes changes to the DB.

The alternative, that could be done in a second iteration could be to merge EventTeam and Team tables into 1, where a team is only associated to an event at the DB level. However that would mean, removing 1 person teams, which means,

• during the migration one need to correctly re-generate new teams for each signed up users for each event. More generally there is a risk of breaking something is we do such invasive changes.
• if there are no special 1 person teams, reversibility is a bit more problematic. Say one starts alone in a team A invites another user, then decides it's a mistake (before making submissions) and leaves the team. Then if there are no special cased 1 person teams to fall back to, it's unclear which team the user should go to.

[kegl]

Perfect, thanks @rth , I like it even more than my proposal :). I think it's ok to keep the one-person team as special case even in long term since it's likely that the Saclay challenges will not use the team feature (config?). @agramfort ?

I'll try to think about edge cases to cover all the possibilities, but it seems to me that this is bulletproof.

[agramfort]

if @rth does it with migration tests on a staging server before making a release and updating the ramp.studio web site I am happy :)

[rth]

Thanks for the feedback!

Yes, that works. We should anyway first make a 0.18.0 release with the current version of master. This could be included in the next release after that 0.19.0