Add venue profile validation for invite assignment #2572

enrubio · 2025-04-24T13:55:58Z

This PR adds a new setting for the venue group content (invited_reviewer_profile_minimum_requirements) that allows venues to define profile requirements when inviting reviewers. Validation is added to the check_new_profiles cron job and to the invite assignment preprocess. Example of setting:

'invited_reviewer_profile_minimum_requirements': { 
    'value': {
        'content.relations': 1,
        'content.dblp': True,
        'active': True
    } 
}

The keys in the value should be the path to the field in the profile. So if it's content.relations we'll look at profile.content.relations and if it's active we'll look at profile.active.

Currently we only expect the values to be an integer (to check for number of entries in history, relations, publications etc.) or a boolean (check if certain links exist in the profile).

If a user was invited, is in Pending Sign Up, and still has an incomplete profile, that user will stay in Pending Sign Up and receive email reminders each day until they complete their profile.

melisabok · 2025-04-24T20:01:25Z

openreview/venue/process/invite_assignment_pre_process.js

@@ -27,6 +28,53 @@ async function process(client, edge, invitation) {
  const profiles = await client.tools.getProfiles([edge.tail], true)
  const userProfile = profiles[0]

+  // Check for complete profile, if no profile then go to pending sign up
+  if (profileReqs && !userProfile.id.includes('@')) {


I wonder if we can move this code to the tools.py so it can be used from the preprocess and the cron job.

Ok! I can move it. To allow this JS preprocess to use it, should it get added here?: https://github.com/openreview/openreview-js/blob/8c7cd7f370459391a9cb4cc416223a386c287c6b/packages/client/src/tools.js

you check by profile id instead of email address like userProfile.id.startswith("~")

let's move this to the tools.py so the process function is not so large.

But this is a JS preprocess so it would need to go in the JS version. What do you think?

yes, let's do that

This has been moved to tools.js: openreview/openreview-js#72

melisabok · 2025-04-24T20:02:18Z

openreview/venue/process/invite_assignment_pre_process.js

+
+      // If checking publications, filter notes for pdate
+      if (pathItems[pathItems.length - 1] === 'publications') {
+        actualValue = actualValue.filter(pub => pub.hasOwnProperty('pdate'));


do you need to check if the publications are public too?

We can, like the expertise does. When we compute conflicts do/should we only look at public published publications? I was thinking we add a filter based on what info we use for conflicts.

Talked with Melisa - decided to remove filter on publications and rely on client permissions.

Copilot

Pull Request Overview

This PR adds and enforces a new setting (invited_reviewer_profile_minimum_requirements) to validate reviewer profiles before invitation assignment, ensuring that only profiles meeting venue‐specific minimum criteria are accepted.

Updated tests in test_cvpr_conference_v2.py to verify profile validation logic in invite assignments.
Modified openreview/venue/venue.py and openreview/venue/process/invite_assignment_pre_process.js to incorporate profile completeness checks and send notifications accordingly.
Propagated the new setting into group and conference helper files to support its configuration and use.

Reviewed Changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
tests/test_cvpr_conference_v2.py	Added new test for invite assignment profile requirements and validations.
openreview/venue/venue.py	Updated check_new_profiles to validate profiles based on new requirements and send notifications.
openreview/venue/process/invite_assignment_pre_process.js	Integrated profile completeness check in invite assignment preprocess.
openreview/venue/group.py	Propagated the new invited_reviewer_profile_minimum_requirements setting into group content.
openreview/conference/helpers.py	Updated venue configuration to load the new profile requirements setting.

Comments suppressed due to low confidence (1)

tests/test_cvpr_conference_v2.py:1524

Consider replacing the use of time.sleep(5) with a more robust waiting mechanism (e.g., using helpers.await_queue_edit) to reliably wait for asynchronous updates and reduce test flakiness.

time.sleep(5)

Copilot · 2025-05-06T13:32:59Z

openreview/venue/venue.py

+                                                        is_incomplete = True
+                                                        break
+                                                    else:
+                                                        print(f'Invalid path: {profile_path}')


Instead of simply printing 'Invalid path', consider using a logging framework or raising a controlled warning so that unexpected data types in the profile requirements are properly tracked and can be handled or debugged more systematically.

melisabok · 2025-05-06T19:51:34Z

openreview/venue/process/invite_assignment_pre_process.js

@@ -27,6 +28,11 @@ async function process(client, edge, invitation) {
  const profiles = await client.tools.getProfiles([edge.tail], true)
  const userProfile = profiles[0]

+  // Check for complete profile, if no profile then go to pending sign up
+  if (!client.tools.isProfileComplete(userProfile, profileReqs)) {


what happens if profileReqs is empty?

melisabok · 2025-05-06T19:51:53Z

openreview/venue/venue.py

@@ -1474,29 +1488,35 @@ def mark_as_accepted(venue_group, edge, submission, user_profile, invite_assignm
        active_venues = client.get_group('active_venues').members

        for venue_id in active_venues:
+            # Create new client for each venue


melisabok · 2025-05-06T19:52:20Z

openreview/venue/venue.py

+                                                venue_client.post_edge(invitation_edge)
+
+                                            ## Check venue profile requirements
+                                            min_requirements = venue_group.content.get('invited_reviewer_profile_minimum_requirements', {}).get('value')


can you move this code to tools.py?

enrubio added 2 commits April 24, 2025 09:31

Add venue profile validation for invite assignment

3616db6

Add filter for pdate if requiring publications

fac00fc

melisabok reviewed Apr 24, 2025

View reviewed changes

melisabok and others added 2 commits May 1, 2025 11:09

Merge branch 'master' into feature/add-venue-profile-check

81bbf7b

Remove filter on pubs, impersonate active venues

a5cee13

enrubio mentioned this pull request May 6, 2025

Add profile completion check in tools openreview/openreview-js#72

Open

Move profile completion logic to tools.js

147daa1

enrubio marked this pull request as ready for review May 6, 2025 13:31

enrubio requested review from Copilot, melisabok and celestemartinez May 6, 2025 13:32

Copilot AI reviewed May 6, 2025

View reviewed changes

melisabok reviewed May 6, 2025

View reviewed changes

Merge branch 'master' into feature/add-venue-profile-check

ce98253

Add venue profile validation for invite assignment #2572

Are you sure you want to change the base?

Add venue profile validation for invite assignment #2572

Uh oh!

Conversation

enrubio commented Apr 24, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Copilot AI May 6, 2025

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!