LTI Integration

[bhunt02]

Description

This will get a better description later
Oh god... do I really have to???

Fine...

Major Changes/Additions

• Learning-Teaching Interoperability

  • Added LTI implementation to allow LTI-compliant platforms to implement HelpMe as a tool
  • As with LMS integrations, only Canvas has an implementation. If we get more clients, it should be easy enough to extend over to other LMS's so long as they have:
    • An ID corresponding to the course identifier (like Canvas does) (used to identify course via LMS course integration via user account)
    • Supports sharing user email claims with us (used to identify user account if exists)
  • Developed a customized, limited frontend for use with the LTI tools (embedded in an iframe within the platform)
    • Currently, courses dashboard and course page + course integration management page for course staff
    • LTI context and customized header added
      • LTI tool will resize itself to the window's full size when launched
        • BECAUSE CANVAS DOESN'T ALLOW SPECIFYING FULL HEIGHT BY DEFAULT IN LAUNCH SPECIFICATION :))))
  • Developed a customized authentication pipeline - more on that later
    • This is related to why we are presenting a limited frontend to LTI tools: we have to issue SameSite=None authentication tokens when HelpMe is embedded in an iframe
      • These tokens can be a bit dangerous for us
      • More on the authentication pipeline later to explain how this danger is circumvented
  • Comes with addition of an admin dashboard - this is where the 'userRole' column of the User table can become relevant.
    • Only accessible to userRole = admin.
    • Currently only has the list of registered LTI platforms
      • Platforms can be added manually (ew don't do that, we will tell site admins to use https://<domain>/api/v1/lti/register for dynamic registration)
      • Platforms can be deleted
      • Platforms can be set to be active/inactive (accept and process requests for LTI launches or not)
      • Platforms (on our end) can be updated with new credentials if they change
      • Platforms that implement the scopes for updating and reading the OpenID registration will be able to be updated and also automatically dropped if the configuration doesn't exist on their end anymore
        • UNFORTUNATELY CANVAS DOESN'T ALLOW THESE SCOPES AND IT'S ANNOYING BECAUSE I SPENT A BIT IMPLEMENTING THIS ONLY TO FIND CANVAS DOESN'T ALLOW IT (WHY??? WE MIGHT HAVE HANGING PLATFORMS ON OUR END???)
  • Uses lti-typescript (more on that later) to perform the LTI launch authorization/pipeline

• Authentication Pipeline Expansion/Refactor

  • Secondary Authentication Pipeline, specific for LTI
    • Modifications to all frontend pages under (auth), with behaviours specific to if their pathname starts with '/lti'
      • Baby-gate LTI launches to the LTI pages as much as possible (I think I've covered all my bases)
      • To maximize re-use of code, the structure is duplicated, but the same components are imported and exported from the routes under /lti/(auth)/...
    • When the site is embedded in an iframe, we need to:
      • Issue access tokens with SameSite=None;
      • Secondary redirects to other platforms (e.g., Google SSO, UBC Shibboleth) need to occur in a new window/tab;
        • These automatically close after the authorization is complete (but don't if it fails so the user can see the error message)
        • The tokens are attached to the domain on the user's browser so it's fine this way, the iframe presents an interface to reload the embedded page, the middleware can then access the new authentication state and behave accordingly
  • Solution: extract and modularize authentication methods used in the backend application. They have also been updated.
    • auth.service.ts now contains methods which process the response, but take in several optional parameters.
    • login.service.ts was created to facilitate the login methods for both (at least, the two current versions, default and lti) with variable behaviours
      • entry is a function which maps to the old login entry behaviour that was repeated in several instances of the code (e.g., SSO auth & legacy auth had shared behaviours)
      • handleCookies performs the management of redirects with email-verification specific behaviours added, and LTI-specific ones too
    • auth.controller.ts heavily modified with a lot of behaviour extracted to auth.service.ts
    • lti-auth.controller.ts created to call auth.service.ts methods with LTI-specific parameters (e.g., cookie options, redirects)
  • Scoped Authorization Tokens
    • LTI implementation introduces this, with lti_auth_token specifically
      • Secondary type of auth_token so we can differentiate (at least slightly)
      • Implements new 'restrictPaths' property of our tokens that checks whether the API route or the frontend path being accessed is allowed for the token
        • This prevents LTI launches from accessing routes we don't want them to
      • We could apply this in the future, setting non-verified accounts to be restricted to any page outside /verify, and upon verification, allowing them through
      • We could also apply this in the future to route protection
  • SSO Authentication Changes
    • No longer uses a cookie to store the organization ID.
    • Uses the supported state query parameter to pass a unique, 32-bit identifier that is later used to authenticate the SSO login request (at the callback) and retrieve the stored organization ID.
    • AuthStateModel is used for this. It stores the state sequence as a primary key, expires in 60 seconds, and identifies the organization being logged in with.
    • This is a lot simpler than using cookies, and it's supported by most OAuth providers.

• LMS Integrations

  • LMS Access Token Generation
    • Using OAuth syntax/pipelines, LMS course integrations can now have access tokens generated on demand. This is the expected implementation from Instructure and our own UBC CTL.
    • Added an organization setting which disables API key usage (so that developers can still use API keys to authorize with Canvas)
      • Functionally on production we'll disable this option
    • Related types & token management added to user pages and within the LMS course integration upsert menu
    • Access token generation requires the LMS organization integration to have a client ID and client secret set (supplied by the platform upon creation of the developer key)
      • This might be Canvas-specific but I don't know, I've not worked with Moodle, or Blackboard, or any of the rest
      • We can address it later if it is Canvas-specific but I feel as though it's not since this is deeply intertwined with LTI protocols
    • LMS user access tokens are stored as LMSAccessTokenModel entities
      • LMSCourseIntegration now have a 0..1 relationship with these entities
      • Users have 0..* of these, but they are softly identified by the user-organizationIntegration relationship
        • A.K.A., user ID, organization ID, organization API platform
      • Internally identified by a unique serial ID so the FK with LMSCourseIntegration isn't insane
    • During the authorization pipeline (which redirects to the LMS to perform authorization to obtain the token):
      • An LMSAuthStateModel entity is created (it has some useful properties such as storing a redirect right back to the page with certain query params)
        • It can also expire so users have to authorize within 60 seconds
        • It makes sure we can authenticate that the request to generate an access token actually came from the platform (as long as a valid, matching state entity exists)
    • Users can manage their access tokens in the profile page under a submenu
      • It's implemented in such a way that if we have more 3rd party token types other than LMS in the future, those could be added easily
  • Updates to Organization Integration Creation/Update
    • Added three properties:
      • secure (default: true): boolean property indicating whether the platform is HTTP or HTTPS
        • Really only useful if you're hosting Canvas locally and need to contact an HTTP route, but I digress
        • Hence why it defaults to true
      • clientID (optional): corresponds to the tool client ID, from the LMS platform, for the developer key used to create access tokens
      • clientSecret (optional): corresponds to the tool client secret, blah blah blah
    • Extracted upsert modal to its own component
    • Organization integration management also comes with the ability to invalidate user access tokens
      • I don't know I just thought it would be a good idea since otherwise, the user's access tokens are managed by the user solely
    • Backend: slight refactor of organization upsert
  • Updates to Course Integration Creation/Update
    • When there's an access token available/selected, it uses this token to retrieve the list of user courses from Canvas
      • No more explaining to users what their Canvas course ID is
    • Can use either API key or access token (if API key is enabled in organization settings)
    • If it's not, can only use access token generation
    • When either is uploaded, the other is deleted from the database (so it's easier for us to decide what to use)
      • API key expiry is deleted alongside API key in the case when an access token replaces it
    • Various updates to the interface and properties that can be input/logic of when they're shown
    • Backend: overhaul of course integration upsert

What's an LTI-Typescript???

A library called ltijs was offered to me as an option for configuring our integration.
I started trying to use the library and then I realized it was pure Javascript.
This deeply upset me. I then embarked on a 2-week-long endeavor to fork and rebuild this library using TypeScript.
It uses TypeORM, like we do, for compatibility. Also because the original implementation used MongoDB and I wasn't about to add another DBMS and database server/container we have to use and secure.
Learned a lot about the LTI protocol and how it works. We now have a library that is strongly typed and I'll try to maintain.
You can find it here.

What else has been added?

• Configuration
  • New Environment Variables
    • server/.env
      • LTI_SECRET_KEY - the secret key used by LTI-Typescript for en/decrypting data in encrypted tables
    • server/postgres.env
      • POSTGRES_LTI_DB - the name of the database used for lti (it's lti)
  • 2 new databases: lti, lti_test (purposes are exactly what you think)
    • Includes updates to the test_createdb.sql file that is used for creating the test DBs in CI
• Frontend
  • api/server.ts file added, provides server-side version (NOT FOR MIDDLEWARE) of the api/index.ts file
    • Since it's server-side, it needs to add the authorization cookie header explicitly
      • APIClient constructor customized for this reason
  • /lti path
    • All lti-specific interfaces are stored here
  • /admin path
    • Stub definition for main admin console
    • /admin/lti - LTI platform management
  • constants.tsx - File for defining constants. Currently exports and holds the HELPME_VERSION variable
    • Hard coded still but easier to locate than it was before
• Backend
  • Added admin.controller.ts (empty, for now) and admin.service.ts (same)
  • Added lti.middleware.ts, a middleware that is registered and used to configure the LTI-Typescript instance
    • e.g., dynamic registration settings, whitelisted routes
  • dataEndec.ts, which will decrypt and encrypt values for storage in an entity that has an encrypted column
    • Make sure the key is the same when you decrypt/encrypt the values
    • Currently only used for LMSAccessTokenModel as it encrypts the token details with the client secret.
  • Various decorators for use with the LTI backend routes (currently only one)
  • AdminRoleGuard is used to guard routes only for users with userRole = admin
  • LTIGuard checks that the response.locals['token'] object is defined.
    • Mostly just a safeguard if the LTI middleware fails or is inactive, prevents accessing the LTI launch route without the IdToken.
  • Added CRON jobs which delete expired tokens for LMSAuthStateModel, AuthStateModel.
  • AbstractLMSAdapter / LMS Adapters in general
    • Static methods
      • createState - creates an LMSAuthStateModel instance with a unique state sequence
      • logoutAuth - initializes a logout (invalidation) operation of a user's access token
        • Determines which implemented adapter to use, then calls its implementation of (non-static) logoutAuth
      • redirectAuth - initializes an authorization redirect operation for access token generation.
        • Determines which implemented adapter to use, then calls its implementation of (non-static) redirectAuth
      • postAuth - accepts an authorization response for access token generation.
        • Determines which implemented adapter to use, then calls its implementation of (non-static) postAuth
    • Non-static methods
      • logoutAuth - implemented in subclasses. Used for invalidating access tokens.
      • checkAccessToken - implemented method for checking whether an access token is valid. Refreshes the access token if not.
        • Throws an exception if the access token can't be refreshed.
        • Otherwise, saves the refreshed token and returns it.
        • If the token is not expired, returns it outright.
      • getAuthorization - implemented method for getting the authorization header.
        • If API key is used for course integration, simply creates a Bearer <token> header.
        • Otherwise, uses checkAccessToken to get the (possibly refreshed) access token, and creates a <token-type> <token> header.
      • getUserCourses - implemented in subclasses, gets the courses to which a user belongs, pending they have an access token.
        • This only works with an access token as when a token is created, it provides the user's Canvas ID.
          • We do not get this ID from the API key. We could, but doesn't seem worth it if we're sunsetting that.
  • lmsIntegration.controller.ts
    • Added routes for:
      • GET org/:oid/token -> (ORG ADMIN) retrieves the tokens for the organization and (optional) platform
      • DELETE org/:oid/token/:tokenId -> (ORG ADMIN) invalidates user access token if found
      • GET oauth2/token -> retrieves the user's options for tokens (used in course integration upsert)
      • DELETE oauth2/token/:tokenId -> invalidates user access token if found and belonging to user
      • GET oauth2/authorize -> starts the pipeline for generating an access token
      • GET oauth2/response -> performs the response actions for generating an access token after redirected back
      • GET course/list/:tokenId -> gets the platform course list for the user given their token
  • lmsIntegration.service.ts
    • Added createAccessToken, destroyAccessToken methods
      • createAccessToken - takes the data from the oauth2 response, encrypts it into an LMSAccessTokenModel
      • deleteAccessToken - performs the logout auth and deletes the token if successful
  • login.service.ts
    • initLoginEnter - used in login controller and lti auth controller to try to call LoginService.enter, fails if token is invalid
    • enter - performs the old entry method with modifications
    • handleCookies - performs the redirect/cookie handling that the old entry method (and duplicates of it) would do
    • handleLTICourseInviteCookie - used in specific circumstances to check the LTI course invite without calling for a redirect
    • generateAuthToken - generates an authentication token for HelpMe with given parameters
  • lti-auth.controller.ts - implements several methods from auth.controller.ts, customized for LTI (nothing unique besides parameter differences)
  • lti.controller.ts - entrypoint for LTI functionality.
    • ALL / - entrypoint for LTI. returns a redirect to the /lti frontend route, or /lti/<courseId> frontend route if course matching API course is found.
      • Authorizes the user with LoginService.enter method if the user is found.
      • Redirects to /lti/login frontend if user is not found.
      • Performs other operations, such as generating (if the course is found but user is not) LTICourseInviteModels and cookies to add the user to a course after they've registered, if not found
      • Automatically adds the user to the course if the course exists and the user exists
      • Sets query parameters that are eventually put into session storage and used for the LTI launch session
        • Particularly, the platform and the api course ID which are used in integration management
    • Includes AdminGuard'd endpoints for platform management.
    • Has exported function in file for mapping LTI-Typescript platform to representation HelpMe uses of it
  • lti.middleware.ts - defines the configuration for the LTI-Typescript provider-as-middleware
    • Launches the configuration as a middleware, used to perform the LTI pipeline except for whitelisted routes
    • Also automatically drops platforms which no longer store a configuration for us, so long as they have the scopes necessary enabled to get that information
  • lti.service.ts
    • Provides access to the active LTI provider as an instance property
    • createCourseInvite - creates an LTI course invite
    • checkCourseInvite - checks an LTI course invite for validity, if valid, adds user to course
    • findMatchingUserAndCourse - uses the LTI launch parameters to find a) a matching user, and b) a matching course, if possible.
      • If neither are found, behaviour in the controller defines how the LTI launch acts
    • extractCourseId - uses the LTI launch parameters and finds the course ID from the platform
      • This course ID is then checked if it matches with any existing course LMS integrations. If it does, that course is used for the launch.

What else has changed?

• Common
  • 'userRole' attribute in User class in common/index.ts now maps to the enum.
  • Renamed UBCOloginParam to generic LoginParam (just for convenience, it's not even UBCO-specific - in fact, it's less UBCO-specific than the shibboleth login!!!)
• Frontend
  • Dashboard layout now shows verification page if redirect does not properly occur (bandaid)
  • Adjusted common/index.ts in general for placements, added class-transformer decorators to the LMS types/classes
  • Migrated/removed many types that are declared within typings/*.ts to common/index.ts so the backend and frontend declarations are fully aligned
  • registration/page.tsx extrapolated to optionally take in an oid URL parameter, rather than relying on the value in localStorage
    • Also, no longer deletes this value from localStorage upon successful registration
  • coursesSection.tsx modified to support the needed behaviour within the LTI pages
  • Updated some control logic related to LMS integrations
    • Also updated some descriptions
    • Also made it so if a resource is disabled the related fetch for the resource elements is not called (it's wasteful otherwise/quizzes throw an ugly error if disabled and a fetch is attempted)
  • Updated various erroneous control logic related to roles (e.g., EmailNotifications.tsx assumes 'userRole' mapped to OrganizationRole type, but it never would)
  • api/cookieApi.ts renamed to api/cookie-utils.ts and modified for LTI-specific behaviours
  • api/index.ts
    • Added the various auth routes here as a result of removing api/authApi.ts
    • profile.index -> profile.getUser, with original implementation that was in userApi.ts
      • This is for compatibility reasons
    • profile.index -> profile.fullResponse, profile.getUser but returns full response object
    • mail added, mail.resendVerificationCode added (result of removing api/mailAPI.ts)
    • Useful function for fetchUserDetails (reused to supply profile to UserInfoContext) extracted/added
  • middleware.ts
    • Updated/added public pages
    • Restricted routes from authentication token are processed
    • LTI-specific behaviours added
      • HelpMe tab that's launched from the 'Open HelpMe' button in the LTI interface will delete the lti_auth_token
      • Default redirect page is different for LTI, e.g., '/lti' vs. '/courses'
      • All pages redirected to in LTI are different, usually prepending '/lti' to the redirect URL
• Backend
  • auth.controller.ts methods extracted to auth.service.ts for use with other authentication controllers, like lti-auth.controller.ts
  • auth.service.ts now contains additional general-purpose authentication methods
  • bootstrap.ts modified for LTI usage
    • Helmet and cookieParser default settings (used elsewhere in the app) were incompatible with the LTI implementation
    • Standard Helmet and cookieParser are used app-wide, except for on the LTI routes
    • LTI routes have:
      • Helmet: { frameguard: false, contentSecurityPolicy: false } - prevents issues with displaying app in iframe
      • CookieParser: called with LTI_SECRET_KEY as the key used to decrypt the cookies (cookies written along the LTI routes will also be signed with this key)
    • Also had to make addGlobalsToApp function take test as an argument as the Regex used for the custom middlewares would be far too complicated (and unrealistic to use) for tests
      • Tests do not have the /api/v1/ prefix, complicates things, can't make that optional
  • JWTAuthGuard updated with custom handleRequest
    • Checks for restrictPaths in the 'user' (decoded and verified JWT)
      • If there's none, returns the user (they're not restricted)
      • If there is, it:
        • If restrictPaths is an array, it checks whether the path being requested is contained within it, either through Regex matching or string equality
        • If it's a string, it checks whether the path being requested is allowed, either through Regex matching or string equality
  • lmsIntegration.controller.ts
    • Moved data validation from the controller method into the service method for POST org/:oid/upsert -> upsertOrganizationLMSIntegration
    • Overhauled POST course/:courseId/upsert
    • Updated POST :courseId/test
  • lmsIntegration.service.ts
    • upsertOrganizationLMSIntegration
      • Moved data validation originally in controller to here
      • Operations now drop undefined columns
      • Invalidates user access tokens if the client ID or client secret changes
        • Removes them all after even if some operations were unsuccessful (we won't need them, but we're doing a courtesy to the platform by invalidating them outright)
    • updateCourseLMSIntegration
      • Overhauled, overwrites properties only if they're defined (safeguard)
    • testConnection
      • Now works with access tokens
    • getAPICourses
      • Satellite method to call an adapter for Adapter.getUserCourses
  • JwtStrategy
    • Now either gets the auth_token (default) or if it is null, the lti_auth_token, rather than just auth_token or null
  • login.controller.ts
    • Extracted methods to login.service.ts
    • Changed 'legacy login attempt with SSO account' HTTP status to I_AM_A_TEAPOT
      • This brings me great joy
    • logout - now tries to clear both auth_token and lti_auth_token
  • mail.controller.ts
    • Token behaviours updated. resendRegistrationToken adjusted.
  • profile.controller.ts
    • get profile returns restrictPaths as well.
  • user-token.entity.ts
    • created_at -> createdAt, CreateDateColumn with timestamp with timezone type, Date type in TS
    • expires_at -> expiresIn, bigint column that aligned with epoch millis to bigint column that represents the number of seconds until the token expires.

What's been removed?

• Environment Variables
  • Removed GOOGLE_REDIRECT_URI - we just compute this now since we have multiple redirect URIs (default, LTI) and other SSO providers will follow similar URIs
• Frontend
  • api/authApi.ts file was removed, methods extracted to central index.ts file
  • api/mailAPI.ts file removed, ...
  • api/userAPI.ts file removed, ...
  • Any usages of functions from these files have been corrected, the server-side version of the frontend API is used in server components (NOT THE MIDDLEWARE)
  • api/organizationAPI.ts almost entirely scrapped with methods extracted
    • Only getOrganization remains... might remove this in this PR as well
• Backend
  • Deprecated proxy mappings and remnants of the routes for the static admin pages
    • Deprecated admin.command.ts command file
    • Deprecated admin-user.entity.ts file

Closes #308, #389

Type of change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• This change requires a documentation update
  • Kind of already did it with environment variables
• This requires a run of yarn install
• This change requires an addition/change to the production .env variables. These changes are below:
  server/.env

LTI_SECRET_KEY=<it's a secret to everyone>

server/postgres.env

POSTGRES_LTI_DB=lti

• This change requires developers to add new .env variables. The file and variables needed are below:
  server/.env

LTI_SECRET_KEY=<it's a secret to everyone>

server/postgres.env

POSTGRES_LTI_DB=lti

• This change requires a database query to update old data on production. This query is below:

Production Environment Changes

• Shibboleth authentication URL needs to be changed
• Admin proxy routes need to be removed

How Has This Been Tested?

I wrote a lot of tests. A LOT.

Checklist:

• I have performed a self-review of my own code
• I have commented my code where needed
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• I have checked that new and existing tests pass locally with my changes
• Any work that this PR is dependent on has been merged into the main branch
• Any UI changes have been checked to work on desktop, tablet, and mobile

[AdamFipke]

holy smokes, i just finished mostly reading the PR description. I don't have any comments yet and I haven't started reviewing the code itself, but I just wanna say I appreciate you putting in so much effort and detail into it

EDIT: the description (everything above the "Type of Change" header) is almost 3500 words haha