Groovy connector samples: the REST/CREST sync scripts replay the whole changelog after every empty poll, and other sample scripts repeat work on every call #164

Summary

src/assemble/samples.xml (L39-L47) packages src/test/resources into the shipped groovy-connector-*-samples.zip, and users start their connectors from these scripts. Several of them do far more work than they need to:

• The REST and CREST sync samples store the token "0" after a poll that finds no changes. The next poll then replays the whole changelog, with one extra GET per entry. This works out to roughly every second poll being a full replay.
• The other samples recompile templates on every search, rebuild the whole result set for every page, ignore filters and paging, or keep calling the handler after it asked them to stop.

Line references are to 1abfe74. The template timing, the MissingPropertyException and the GroovyCastException come from a throwaway harness; everything else comes from reading the code.

1. REST and CREST sync samples replay the whole changelog after an empty poll

rest_sample/SyncScript.groovy starts each pass with def lastToken = "0" and returns new SyncToken(lastToken) (users: L86, L148; groups: L166, L223). ScriptedConnectorBase.sync passes that token to handleResult (L481-L485), so it becomes the stored token.

• A poll that finds no changes stores "0".
• The next poll asks for _id gt "0" and gets the whole retained changelog.
• Every entry is sent again, and each one costs an extra GET of the current object (L111, L191): N+1 requests.
• The changelog query has no page size, so HTTPBuilder parses the whole response before the first delta is delivered.
• GET_LATEST_SYNC_TOKEN asks for _pageSize=1 without a sort key (L50-L66). Unless the server returns the changelog newest first, that is the oldest entry, so the first poll after it also replays nearly the whole changelog.

With frequent polling, most polls find nothing. So roughly every second poll is a full replay: empty poll, token "0", replay, token set to the last id, empty poll again.

crest_sample/SyncDJScript.groovy has the same token logic (L74, L166) and the same per-entry read (L110). As shipped, it never reaches a request. QueryFilter is not imported (L25-L44), so the first QueryFilter call (L64, and L177 in GET_LATEST_SYNC_TOKEN) throws MissingPropertyException. With the import added, [...] as AbstractRemoteConnection.QueryResultResponseHandler (L161, L195) throws GroovyCastException. The target is a concrete inner class whose only constructor takes (AbstractRemoteConnection, QueryResourceHandler), and Groovy can coerce a map only onto an interface or onto a class with a constructor it can call. query expects a QueryResourceHandler in any case, as used in crest/SearchScript.groovy (L160-L196). That script has the same missing import, so its search without a filter fails the same way (L117). Once the cast is fixed, the read at L110 runs inside handleResource on the HTTP I/O thread and can hang (see #163).

Fix: when nothing was read, return the incoming token unchanged. Page the changelog query.

2. The sample cookie paging rebuilds and re-sorts the whole store for every page

groovy/SearchScript.groovy (marked "Sample script for IDME-178") calls ObjectCacheLibrary.search for every page (L55). That method builds a new TreeSet of every matching object (ObjectCacheLibrary.groovy#L160-L168). The script then walks the set from the start until it finds the cookie (L72-L80).

Each page costs O(N log N) to build plus a scan up to the cookie. Paging through N objects at page size P is therefore O((N/P) · N log N). This is the same quadratic shape as reading a file once per row.

3. SQL search samples compile a Groovy template on every search

• sql/SearchScript.groovy creates a SimpleTemplateEngine and compiles a template whenever a filter is set (L93, L108). That includes every getObject by __UID__.
• sql_sample/SearchScript.groovy does it on every search (L147).

Each call parses and compiles a script class: 2.4–3.6 ms per call in a loop of 3,000 across our runs (JDK 26). With default GC settings the classes stay loaded. All of them unload once soft references are cleared (-XX:SoftRefLRUPolicyMSPerMB=0 or a small heap), so this is per-call CPU and Metaspace churn, not a leak.

Fix: build the clause once, or with bind parameters, instead of compiling a template per call.

4. SQL sync samples resend rows and scan the whole table on every poll

sql/SyncScript.groovy (L85, L107, L123) and sql_sample/SyncScript.groovy (L65, L84, L100) select timestamp > token without ORDER BY. Each delta carries its own row's timestamp as its token, and SYNC returns no final token. The client therefore keeps the token of the last delta it received: the timestamp of the last row in result order, not the newest one. Any changed row newer than that is sent again on every poll, until the result happens to end with the newest row.

Neither DDL indexes timestamp:

• sample_HR_DB.ddl#L17 (likewise Groups and Organizations);
• testDatabase.ddl#L32.

The sample tables are tiny, but once this schema is copied to a real table, every poll and every order by timestamp desc in GET_LATEST_SYNC_TOKEN scans it.

As shipped, both scripts also fail on the first changed row:

• Inside the closure passed to handler(...), it is that closure's own parameter, which is null, not the row, so it.timestamp throws.
• Once that is fixed, row is undefined (sql L91, sql_sample L71).

5. Smaller items

• arm/SearchScript.groovy serves an equality filter on __UID__ with one read, but ignores every other filter (L136-L139) and lists everything. For groups it makes one GET per group (L154-L158), so a search by __NAME__ or any other attribute costs N+1 requests.
• Paging options are ignored:
  • sql/SearchScript.groovy never reads pageSize or pagedResultsCookie (L113-L153);
  • rest_sample/SearchScript.groovy sends only _queryFilter (L94-L98).
  • Every page request returns the full result set.
  • sql_sample/SearchScript.groovy does read the page size, but builds invalid SQL with it (L116-L133): LIMIT lands before WHERE on the first page, and ORDER BY is appended without a leading space.
• false from the handler is ignored. The loop keeps reading and calling the handler after the caller has asked it to stop:
  • sql/SearchScript.groovy (L115-L116);
  • rest_sample/SearchScript.groovy (L101-L102);
  • rest_sample/SyncScript.groovy (L87-L93).
• azure_ad/shared/AzureADOAuth2HttpClientFactory.groovy takes one lock around every Graph API request and refreshes the token while holding it (L137-L152). The token client is built without timeouts (L124-L126), and the Graph client reuses the same builder (L207). While a refresh is in flight, a hung token endpoint blocks every Azure request behind the lock.