Merge develop into mls (#3597)

* Fix bug: federatorInternal host not set for background-worker (#3516)

* WPB-3916: Filtering out duplicate members when sending defederation notifications (#3515)

* integration: Add test to verify behaviour with offline backends (#3501)

* background-worker: Make push backoff times configurable

* brig/getFederationStatus: Always return NonConnectedBackends as empty when fed policy is AllowAll

* integration: Use separate vHosts for backendA and B.

* integration/RunServices: Add hack to make federation work

* integration: Add test to verify behaviour with offline backends

* helm-var-integration: Workaround bug with federation

* integration-test.sh: Run new integration test suite first

---------

Co-authored-by: Marko Dimjašević <marko.dimjasevic@wire.com>

* Distinguish between update and upsert cassandra commands (#3513)

* Remove billing-team-member-backfill tool (#3520)

* dockerephemeral: Increase nofile ulimits for ES and Fake DynamoDB (#3521)

* [WPB 3842] Federation completeness check (#3514)

* WPB-3842: Improving checks for adding users to a conversation.

Added a check to `ensureAllowed` that checks for full federation
connections for domains in a conversation, including the domains for new
users.

* WPB-3842: Adding the changelog

* WPB-3842: Moving where the extra domain checks are being performed.

Updating integration tests to reflect the updated semantics of
conversation join semantics. Many of them weren't expecting errors
relating to unreachable domains, and had to be updated to reflect this.

* Fix asserted domains in an integration test

* Integration test: assert on non-federating domains

* WPB-3842: Changing parallel testing to sequential testing

---------

Co-authored-by: Marko Dimjašević <marko.dimjasevic@wire.com>

* WPB-3798 incorrect json field names (#3518)

* WPB-3798: Updating code and tests after renaming fields

* WPB-3798: More updates to names after finding more JSON prefix mangling

* WPB-3798: Fixing schema instances for SAML data

* WPB-3798: Fixing instances that had errors, found by tests

* WPB-3798: Adding changelogs

* WPB-3798: PR feedback.

* WPB-3798: Fixing an error with a field called `data'`

The trailing ' would end up in the JSON representation. I've changed it
to use a leading `_` like other structures, and wrote a newtype to
handle the minimal prefix stripping.

Also cleaning up the diff in regards to imports.

* WPB-3798: Cleaning up imports to minimise the diff

* nit-picks (#3519)

* Remove unneeded -Wwarn (re-enabeling -Werror in those modules).

* Makefile: fix hspec_options overloading in .envrc.local.

* integration: Fix testAddingUserNonFullyConnectedFederation and testNotificationsForOfflineBackends (#3529)

* integration: Fix testAddingUserNonFullyConnectedFederation

* integration: Don't allow adding users to conv when one of the pariticipating backends is down

* integration: Add retries to get around problem of federation domain sync threads

* Introduce API v5 (#3527)

* Introduce development version 5

* Specialise API to a specific version

* Use versioned swagger for galley

* Use version swagger for all other services

* Collect all service Swaggers into a typeclass

* Fix swagger integration tests

* Revert any changes to API versions before 5

* Remove promotion of isDevelopmentVersion

* Add CHANGELOG entry

* stern: Optimize RAM usage of /i/users/meta-info (#3522)

* stern: Fetch only the notifications that are needed

* stern: Fetch only the conversations that are needed

* Integration tests: use static ports (#3536)

* [WPB-3799] cannot fetch conversation details after connection request (#3538)

* brig-integration: Fix flaky tests for API.Federation (#3539)

* brig-integration: Don't assume only 1 result in search by display name

Display names are random strings from 2 to 128 characters. If a 2 string name gets generated it is likely that it matches some name generated in another test.

* brig-integration: Mark test not flaky

It didn't fail after runnning it 1000 times.

* Integration suite: Fix bug in local setup: wrong port for nginz http2 (#3543)

* [WPB-662] servantify brig provider bot api (#3540)

* Fix broken "we are hiring" link (#3549)

* Multi-ingress guest links (#3546)

* Check validity of notification IDs (#3550)

* Check validity of notification IDs

* Add CHANGELOG entry

* fixup! Add CHANGELOG entry

* fixup! fixup! Add CHANGELOG entry

* WPB-633 Servantify Brig/Provider.Service API (#3554)

* WPB-1214: Servantify Brig/Provider.Service API

- Moving the routes over to servant, and removing the old routing code.
- Adding new instances to types that needed them for servant.

* WPB-663: Removing a redundant TODO comment, adding changelog

* Fix ES migration script. (#3558)

* Revert "WPB-633 Servantify Brig/Provider.Service API (#3554)"

This reverts commit 3653d56.

* Integration tests: delete all rabbitmq queues during dynamic backends setup phase (#3523)

* [WPB-4406] federator improve logging (#3556)

* Makefile: Avoid executing the hint (#3564)

Backticks execute the command even when they are in quotes.

* Finalise v4 (#3545)

* Remove MLS endpoints from the API

They will be reintroduced when merging the mls branch. These endpoints
are not currently functional on develop, so removing them from here will
reduce the amount of conflicts.

* Finalise v4

* Add CHANGELOG entry

* Add pregenerated swagger for v4

* Delete MLS tests in brig

* Remove more MLS endpoints from v4

* Set default API version to 5 in integration tests

* Update the documentation on API versioning

---------

Co-authored-by: Marko Dimjašević <marko.dimjasevic@wire.com>

* Fix: SCIM user lookup after changing IdP issuer ID (#3473)

* doc: document webapp configuration for multi-ingress environments (#3569)


---------

Co-authored-by: Sven Tennie <sven.tennie@gmail.com>

* [WPB-4361] upgrade jwt-tools (#3559)

* cassandra: Add column and table names in parsing error messages (#3555)

* s/CORS/CSP/ as mentionned by Sven in WPB-2912

* Replace broken integrations with links

see WPB-3599

* replace all instances of example.com with wire.example as per wpb-2621, in charts only

* change back from wire.example to example.com as this was mistakenly commit to develop instead of to the proper branch

* add documentation on creating a first user

* reverting previous commit as sent to wrong branch

* Update sftd docs: include uri scheme in allowOrigin (#3584)

* Update sftd docs: include uri scheme in allowOrigin

* fixup

* WPB-4629 impossible to add users to a conversation if one of the members is from an offline backend (#3585)

* fake-aws-s3 chart: Upgrade to minio 5.0.13 (#3565)

* Disable de-federation to avoid running into a scalability issue (#3582)

https://wearezeta.atlassian.net/browse/WPB-4668

Co-authored-by: Akshay Mankar <akshay@wire.com>

* [WPB-3664] Bug fix: Notify remote backends of their users removed from conversation when reachable again (#3537)

* Formatting

* Test utilities for changing a conv name

* Add a test confirming the bug report

* An action to enqueue notifications concurrently

* Enqueue member removal notification for remotes

* Add a changelog

* Test case formatting

* Migrate test roleUpdateWithRemotesUnavailable

* Migrate test putReceiptModeWithRemotesOk

* Migrate test putReceiptModeWithRemotesUnavailable

* Migrate test testRoleUpdateWithRemotesOk

* Migrate test roleUpdateRemoteMember

* Migrate test putQualifiedConvRenameWithRemotesUnavailable

This one is already covered by testSynchroniseUserRemovalNotification

* Migrate test putQualifiedConvRenameWithRemotesOk

* Migrate test deleteLocalMemberConvLocalQualifiedOk

* Migrate test deleteRemoteMemberConvLocalQualifiedOk

* Migrate test deleteUnavailableRemoteMemberConvLocalQualifiedOk

* Add the copyright header to a test module

* Move a test utility (allPreds)

* Test utility: create a team with members

* Migrate test testAccessUpdateGuestRemoved

* Migrate test messageTimerChangeWithRemotes

* Migrate test messageTimerUnavailableRemotes

* Migrate test testAccessUpdateGuestRemovedRemotesUnavailable

* Migrate test accessUpdateWithRemotes

* Migrate test testAddRemoteMember

* Migrate test testDeleteTeamConversationWithRemoteMembers

* Migrate test testDeleteTeamConversationWithUnavailableRemoteMembers

* Move a test utility (assertLeaveNotification)

* Migrate test "POST /federation/leave-conversation : Success"

* Migrate test "POST /federation/on-user-deleted-conversations : Remove deleted remote user from local conversations"

* Migrate test updateConversationByRemoteAdmin

* Tests: support giving a role when adding

* Use cannon API for notifications when possible

* Use startDynamicBackends when possible

* Fix assertion

* Migrate test testAddRemoteUsersToLocalConv

* Test add member endpoint at version 1

* Add return value to enqueueNotification

* Use cannon assertions in offline backends test

* Check that remote notifications are received

* Test removal of users from unreachable backends

* Use correct domains for default backends

Taking the domains in the `backendA` and `backendB` resources only works
locally.

* fixup! Use cannon assertions in offline backends test

---------

Co-authored-by: Paolo Capriotti <paolo@capriotti.io>
Co-authored-by: Akshay Mankar <akshay@wire.com>

* WPB-4240: Migrate from swagger2 to openapi3 (#3570)


---------

Co-authored-by: Igor Ranieri Elland <54423+elland@users.noreply.github.com>
Co-authored-by: Igor Ranieri <igor@elland.me>

* Remove mocked MLS member add test

* Resolve conflict in pregenerated swagger

* Remove MLS end2end tests

---------

Co-authored-by: Stefan Matting <smatting@users.noreply.github.com>
Co-authored-by: Owen Harvey <owenlharvey@gmail.com>
Co-authored-by: Akshay Mankar <akshay@wire.com>
Co-authored-by: Marko Dimjašević <marko.dimjasevic@wire.com>
Co-authored-by: fisx <mf@zerobuzz.net>
Co-authored-by: Igor Ranieri Elland <54423+elland@users.noreply.github.com>
Co-authored-by: Leif Battermann <leif.battermann@wire.com>
Co-authored-by: Jappie Klooster <jappieklooster@hotmail.com>
Co-authored-by: Leif Battermann <leifbattermann@gmail.com>
Co-authored-by: Thomas Belin <thomasbelin4@gmail.com>
Co-authored-by: Sven Tennie <sven.tennie@gmail.com>
Co-authored-by: Arthur Wolf <wolf.arthur@gmail.com>
Co-authored-by: Igor Ranieri <igor@elland.me>

changelog.d/1-api-changes/WPB-4668-disable-defederation

@@ -0,0 +1 @@
+Remove de-federation (to avoid a scalability issue).

changelog.d/3-bug-fixes/WPB-4629

@@ -0,0 +1 @@
+Fixed add user to conversation when one of the other participating backends is offline

changelog.d/3-bug-fixes/remote-member-removal-notification

@@ -0,0 +1 @@
+This fixes a bug where a remote member is removed from a conversation while their backend is unreachable, and the backend does not receive the removal notification once it is reachable again.

changelog.d/4-docs/WPB-4240

@@ -0,0 +1 @@
+Updating the route documentation from Swagger 2 to OpenAPI 3.

changelog.d/5-internal/WPB-4240

@@ -0,0 +1,4 @@
+Updating the route documentation library from swagger2 to openapi3.
+
+This also introduced a breaking change in how we track what federation calls each route makes.
+The openapi3 library doesn't support extension fields, and as such tags are being used instead in a similar way.

charts/fake-aws-s3/requirements.yaml

@@ -1,4 +1,4 @@
 dependencies:
 - name: minio
-  version: 3.2.0
+  version: 5.0.13
   repository: https://charts.min.io/

charts/fake-aws-s3/values.yaml

@@ -1,9 +1,5 @@
-# See defaults in https://github.com/helm/charts/tree/master/stable/minio
+# See defaults in https://github.com/minio/minio/blob/RELEASE.2023-07-07T07-13-57Z/helm/minio/values.yaml
 minio:
-  mcImage:
-    repository: quay.io/minio/mc
-    tag: RELEASE.2021-10-07T04-19-58Z
-    pullPolicy: IfNotPresent
   fullnameOverride: fake-aws-s3
   service:
     port: "9000"

deploy/dockerephemeral/docker-compose.yaml

@@ -57,8 +57,7 @@ services:
 
   fake_s3:
     container_name: demo_wire_s3
-#    image: minio/minio:RELEASE.2018-05-25T19-49-13Z
-    image: julialongtin/minio:0.0.9
+    image: minio/minio:RELEASE.2023-07-07T07-13-57Z
     ports:
      - "127.0.0.1:4570:9000"
     environment:

docs/src/how-to/install/sft.md

@@ -18,7 +18,7 @@ tags:
 
 sftd:
   host: sftd.example.com # Replace example.com with your domain
-  allowOrigin: webapp.example.com # Should be the address you used for the webapp deployment
+  allowOrigin: https://webapp.example.com # Should be the address you used for the webapp deployment (Note: you must include the uri scheme "https://")
 ```
 
 In your `secrets.yaml` you should set the TLS keys for sftd domain:

docs/src/understand/configure-federation.md

@@ -457,13 +457,8 @@ the sysadmin:
 
 * [`PUT`](https://staging-nginz-https.zinfra.io/api-internal/swagger-ui/brig/#/brig/put_i_federation_remotes__domain_)
 
-* [`DELETE`](https://staging-nginz-https.zinfra.io/api-internal/swagger-ui/brig/#/brig/delete_i_federation_remotes__domain_)
-  - **WARNING:** If you delete a connection, all users from that
-   remote will be removed from local conversations, and all
-   conversations hosted by that remote will be removed from the local
-   backend.  Connections between local and remote users that are
-   removed will be archived, and can be re-established should you
-   decide to add the same backend later.
+* **NOTE:** De-federating (`DELETE`) has been removed from the API to
+   avoid a scalability issue.  Watch out for a fix in the changelog!
 
 The `remotes` list looks like this:
 

integration/default.nix

@@ -15,8 +15,11 @@
 , Cabal
 , case-insensitive
 , containers
+, cql
+, cql-io
 , cryptonite
 , data-default
+, data-timeout
 , directory
 , errors
 , exceptions
@@ -80,8 +83,11 @@ mkDerivation {
     bytestring-conversion
     case-insensitive
     containers
+    cql
+    cql-io
     cryptonite
     data-default
+    data-timeout
     directory
     errors
     exceptions

integration/integration.cabal

@@ -100,21 +100,23 @@ library
     Notifications
     RunAllTests
     SetupHelpers
+    Test.AccessUpdate
     Test.AssetDownload
     Test.B2B
     Test.Brig
     Test.Client
     Test.Conversation
-    Test.Defederation
     Test.Demo
     Test.Federation
     Test.Federator
+    Test.MessageTimer
     Test.MLS
     Test.MLS.KeyPackage
     Test.MLS.One2One
     Test.MLS.SubConversation
     Test.Notifications
     Test.Presence
+    Test.Roles
     Test.User
     Testlib.App
     Testlib.Assertions
@@ -147,8 +149,11 @@ library
     , bytestring-conversion
     , case-insensitive
     , containers
+    , cql
+    , cql-io
     , cryptonite
     , data-default
+    , data-timeout
     , directory
     , errors
     , exceptions

integration/test/API/BrigInternal.hs

@@ -99,25 +99,6 @@ updateFedConn' owndom dom fedConn = do
   conn <- make fedConn
   submit "PUT" $ addJSON conn req
 
-deleteFedConn :: (HasCallStack, MakesValue owndom) => owndom -> String -> App Response
-deleteFedConn owndom dom = do
-  bindResponse (deleteFedConn' owndom dom) $ \res -> do
-    res.status `shouldMatchRange` (200, 299)
-    pure res
-
-deleteFedConn' :: (HasCallStack, MakesValue owndom) => owndom -> String -> App Response
-deleteFedConn' owndom dom = do
-  req <- rawBaseRequest owndom Brig Unversioned ("/i/federation/remotes/" <> dom)
-  submit "DELETE" req
-
-deleteAllFedConns :: (HasCallStack, MakesValue dom) => dom -> App ()
-deleteAllFedConns dom = do
-  readFedConns dom >>= \resp ->
-    resp.json %. "remotes"
-      & asList
-      >>= traverse (\v -> v %. "domain" & asString)
-      >>= mapM_ (deleteFedConn dom)
-
 registerOAuthClient :: (HasCallStack, MakesValue user, MakesValue name, MakesValue url) => user -> name -> url -> App Response
 registerOAuthClient user name url = do
   req <- baseRequest user Brig Unversioned "i/oauth/clients"

integration/test/API/Galley.hs

@@ -5,6 +5,7 @@ module API.Galley where
 import Control.Lens hiding ((.=))
 import Control.Monad.Reader
 import Data.Aeson qualified as Aeson
+import Data.Aeson.Types qualified as Aeson
 import Data.ByteString.Base64 qualified as B64
 import Data.ByteString.Base64.URL qualified as B64U
 import Data.ByteString.Char8 qualified as BS
@@ -83,6 +84,21 @@ postConversation user cc = do
   ccv <- make cc
   submit "POST" $ req & addJSON ccv
 
+deleteTeamConversation ::
+  ( HasCallStack,
+    MakesValue user,
+    MakesValue conv
+  ) =>
+  String ->
+  conv ->
+  user ->
+  App Response
+deleteTeamConversation tid qcnv user = do
+  cnv <- snd <$> objQid qcnv
+  let path = joinHttpPath ["teams", tid, "conversations", cnv]
+  req <- baseRequest user Galley Versioned path
+  submit "DELETE" req
+
 putConversationProtocol ::
   ( HasCallStack,
     MakesValue user,
@@ -296,12 +312,39 @@ getGroupClients user groupId = do
       (joinHttpPath ["i", "group", BS.unpack . B64U.encodeUnpadded . B64.decodeLenient $ BS.pack groupId])
   submit "GET" req
 
-addMembers :: (HasCallStack, MakesValue user, MakesValue conv) => user -> conv -> [Value] -> App Response
-addMembers usr qcnv newMembers = do
+data AddMembers = AddMembers
+  { users :: [Value],
+    role :: Maybe String,
+    version :: Maybe Int
+  }
+
+instance Default AddMembers where
+  def = AddMembers {users = [], role = Nothing, version = Nothing}
+
+addMembers ::
+  (HasCallStack, MakesValue user, MakesValue conv) =>
+  user ->
+  conv ->
+  AddMembers ->
+  App Response
+addMembers usr qcnv opts = do
   (convDomain, convId) <- objQid qcnv
-  qUsers <- mapM objQidObject newMembers
-  req <- baseRequest usr Galley Versioned (joinHttpPath ["conversations", convDomain, convId, "members"])
-  submit "POST" (req & addJSONObject ["qualified_users" .= qUsers])
+  qUsers <- mapM objQidObject opts.users
+  let path = case opts.version of
+        Just v | v <= 1 -> ["conversations", convId, "members", "v2"]
+        _ -> ["conversations", convDomain, convId, "members"]
+  req <-
+    baseRequest
+      usr
+      Galley
+      (maybe Versioned ExplicitVersion opts.version)
+      (joinHttpPath path)
+  submit "POST" $
+    req
+      & addJSONObject
+        ( ["qualified_users" .= qUsers]
+            <> ["conversation_role" .= r | r <- toList opts.role]
+        )
 
 removeMember :: (HasCallStack, MakesValue remover, MakesValue conv, MakesValue removed) => remover -> conv -> removed -> App Response
 removeMember remover qcnv removed = do
@@ -342,3 +385,89 @@ getConversationCode user conv mbZHost = do
         & addQueryParams [("cnv", convId)]
         & maybe id zHost mbZHost
     )
+
+changeConversationName ::
+  (HasCallStack, MakesValue user, MakesValue conv, MakesValue name) =>
+  user ->
+  conv ->
+  name ->
+  App Response
+changeConversationName user qcnv name = do
+  (convDomain, convId) <- objQid qcnv
+  let path = joinHttpPath ["conversations", convDomain, convId, "name"]
+  nameReq <- make name
+  req <- baseRequest user Galley Versioned path
+  submit "PUT" (req & addJSONObject ["name" .= nameReq])
+
+updateRole ::
+  ( HasCallStack,
+    MakesValue callerUser,
+    MakesValue targetUser,
+    MakesValue roleUpdate,
+    MakesValue qcnv
+  ) =>
+  callerUser ->
+  targetUser ->
+  roleUpdate ->
+  qcnv ->
+  App Response
+updateRole caller target role qcnv = do
+  (cnvDomain, cnvId) <- objQid qcnv
+  (tarDomain, tarId) <- objQid target
+  roleReq <- make role
+  req <-
+    baseRequest
+      caller
+      Galley
+      Versioned
+      ( joinHttpPath ["conversations", cnvDomain, cnvId, "members", tarDomain, tarId]
+      )
+  submit "PUT" (req & addJSONObject ["conversation_role" .= roleReq])
+
+updateReceiptMode ::
+  ( HasCallStack,
+    MakesValue user,
+    MakesValue conv,
+    MakesValue mode
+  ) =>
+  user ->
+  conv ->
+  mode ->
+  App Response
+updateReceiptMode user qcnv mode = do
+  (cnvDomain, cnvId) <- objQid qcnv
+  modeReq <- make mode
+  let path = joinHttpPath ["conversations", cnvDomain, cnvId, "receipt-mode"]
+  req <- baseRequest user Galley Versioned path
+  submit "PUT" (req & addJSONObject ["receipt_mode" .= modeReq])
+
+updateAccess ::
+  ( HasCallStack,
+    MakesValue user,
+    MakesValue conv
+  ) =>
+  user ->
+  conv ->
+  [Aeson.Pair] ->
+  App Response
+updateAccess user qcnv update = do
+  (cnvDomain, cnvId) <- objQid qcnv
+  let path = joinHttpPath ["conversations", cnvDomain, cnvId, "access"]
+  req <- baseRequest user Galley Versioned path
+  submit "PUT" (req & addJSONObject update)
+
+updateMessageTimer ::
+  ( HasCallStack,
+    MakesValue user,
+    MakesValue conv
+  ) =>
+  user ->
+  conv ->
+  Word64 ->
+  App Response
+updateMessageTimer user qcnv update = do
+  (cnvDomain, cnvId) <- objQid qcnv
+  updateReq <- make update
+  let path = joinHttpPath ["conversations", cnvDomain, cnvId, "message-timer"]
+  req <- baseRequest user Galley Versioned path
+  submit "PUT" (addJSONObject ["message_timer" .= updateReq] req)

integration/test/API/GalleyInternal.hs

@@ -51,12 +51,3 @@ getFederationStatus user domains =
         submit
           "GET"
           $ req & addJSONObject ["domains" .= domainList]
-
-deleteFederationDomain ::
-  ( HasCallStack
-  ) =>
-  String ->
-  App Response
-deleteFederationDomain domain = do
-  req <- rawBaseRequest OwnDomain Galley Unversioned $ joinHttpPath ["i", "federation", domain]
-  submit "DELETE" req