Ability to get Resource Bookings together with Work Periods

[maxceem]

We are going to implement the next UI for admins to list ResrouceBookings (RB) together with the WorkPeriod (WP):

We have to be able to get data that is necessary for this page with abilities to filter, sort and paginate (ignoring Alerts).

[maxceem]

Context

Project Context

Topcoder TaaS API - is the main backend service for the Topcoder TaaS App which helps us to manage talent requests from the clients. Additionally to the TaaS App we are working on implementing TaaS Admin App where Topcoder admins could perform some operations.

Challenge Context

In the TaaS API we have Resource Bookings (RB) that represent a member assigned to some work for some time period from "startDate" till "endDate". We already have a model and endpoints implemented for them. We want to be able to process payments for Resource Bookings by weeks. For this purpose we already created a model WorkPeriod that represents weeks and corresponding endpoints for Work Periods (WP) in the previous challenge. There is a new design for TaaS Admin App which we have to support. It requires us to be able to get Resource Bookings together with Work Periods in one endpoint with filter and sorting abilities. As this is an admin app, we would need to support big numbers of records per page and load quite a lot of data fast, so the performance of such operations matters.

Expected Outcome

Endpoint which would retrieve, filter, sort and paginate all the data required for this admin page.

Challenge Details

Technology Stack

• Node.js
• PostgreSQL
• ElasticSearch

Code access

The work in this challenge would be done in 2 repositories:

• https://github.com/topcoder-platform/taas-apis branch feature/rb-with-wp
• https://github.com/topcoder-platform/taas-es-processor branch feature/rb-with-wp

Environment variables necessary to run this service would be shared on the forum. Or you would see them after registering to this challenge under "Registered User Additional Information" below.

Individual requirements

Main Requirement

We have to be able to get data that is necessary for this page with abilities to filter, sort and paginate (Alerts are out of scope for now).

• Update endpoint "GET /resourceBookings" so it could return ResourceBookings together with WorkPeriods by specifying fields query param like this "GET /resourceBookings?fields=id,projectId,workPeriods&projectId=&workPeriods.endDate=&workPeriods.paymentSatus=&sortBy=workPeriods.memberHandle" by following "fields" query param logic.
• Endpoint must support filtering by "projectId", "workPeriods.endDate", "workPeriods.startDate", "workPeriods.paymentSatus" (single filter and multiple combined together)
• Endpoint must support sorting by "startDate", "endDate", "workPeriods.memberHandle", "workPeriods.customerRate", "workPeriods.paymentStatus", "workPeriods.daysWorked"
• Endpoint should support filtering and sorting by all other fields which we now support for Resource Booking and WorkPeriods endpoints. If something is very hard to support, please confirm on the forum if we can drop it.
• If we try to filter or sort by some field which should not be included as per rules of fields param, then return error.
• Add support for the "fields" query param to the endpoint "GET /resourceBookings/:id" too for consistency.

Proposed Solution

Our API follows the next logic: first, it tries to get data from ElasticSearch and if it fails, then get data from DB. We have to support retrieving all the data from both DB and ES. Querying such data from DB is straightforward to implement. Current ElasticSearch index structure looks like doesn't allow us to get such data as we keep Resource Bookings and Work Periods in separate indexes. The next approach is suggested, but if you have other ideas, please share them on the forum to confirm:

• remove separate index for WorkPeriods
• put Work Periods as nested objects inside Resource Bookings records in workPeriods field (similar to how we keep payments inside Work Periods)
• please, note, that we already store WorkPeriodPayments nested inside WorkPeriods, so this would result in double nesting: Resource Bookings -> Work Periods -> Work Periods Payments, so you might need to adjust the logic for endpoints of Work Periods Payments too
• update ES Processor logic to handle such a double nesting for Work Periods and Work Period Payments

⚠️ Retrieving data quickly with filter and sorting ability from ES and DB for this page is essential requirement, so please make sure that this approach would work to achieve all sorting and filtering requirements or suggest alternative, if we could not achieve it, the challenge could be treated as failed.

Unit tests

taas-apis

We don't require special unit tests coverage, but we would like to have good unit tests which would check all the requirements in this challenge.

• "fields" query param: defined, not defined, defined with nested fields
• sorting and filtering by root level fields and nested fields
• we should make all these tests when data come from ES or from DB, for example like this.

taas-es-processor

• Update existent unit tests and e2e tests for Resource Bookings if needed.
• Create unit tests and e2e tests to check logic for Work Periods and Work Periods Payments.

General requirements

As we are updating endpoints and data structure in ES, we have to make sure that we keep all the documentation and utility scripts up to date:

• Swagger - update.
• Postman - add cases demonstrating "fields" query param, sorting and filtering by nested query params.
• ES Mapping - Update mapping definitions for ElasticSearch indexes inside both repositories taas-apis and taas-es-processor.
• Reindex
  • Command index:resourceBookings should reindex data properly taking into account nested Work Periods and subnested Work Period Payments
  • Remove command index:workPeriods as we would not have a separate index anymore.
• Import/Export
  • NPM commands data:import and data:export should support importing/exporting data from/to all the models.
  • Update demo data which is populated by npm run data:import
• Create/Delete Index - NPM commands create-index and delete-index should support creating/deleting all the indexes.
• Split code into reusable methods where applicable.
• Unit tests should pass.
• Lint should pass.

Scorecard Aid

• The major requirement in this challenge that we have to support data retrieving, filtering and sorting of all the necessary data for this admin page.
• Code quality is important to us too and it has to be evaluated in the corresponding sections of the scorecard.
• If there is some test which claims to test something but in reality it doesn't test it - this is very bad as it misleads us, and such cases should be additionally penalized as minor issues.

What To Submit

• Submit a zip file which would include:
  • a Git patch with changes you’ve made to the code in our repository
  • don't need to mention commit hash
• Additionally, the winner would be required to raise a pull request to the repository after the challenge is completed.

[maxceem]

Permissions

• Endpoints "GET /resourceBookings" and "GET /resourceBookings/:id" should only return "memberRate", "workPeriods.memberRate" and "workPeriods.payments" fields to the user who is "Booking Manager", "administrator" or M2M token with correct scope (see for example how it's done for "interviews" inside Job Candidates).
• Fix endpoints "GET /work-periods" and "GET /work-periods/:id" to only return "memberRate" and "payments" fields to the user who is "Booking Manager", "administrator" or M2M token with correct scope (at the moment they don't check scope).

Units Tests for Permissions

• exclude/include "memberRate", "workPeriods.memberRate" and "workPeriods.payments" fields depend on user roles and M2M scopes

[sandhiyakavi]

Specifications mentioned in #212 (comment) are not yet fully implemented. cc: @maxceem

Other than that, tested the below scenarios. Working as expected

1. Verify "GET /resourceBookings" enpoints to return ResourceBookings together with WorkPeriods by specifying fields query param.
2. Verify filtering by "projectId", "workPeriods.endDate", "workPeriods.startDate", "workPeriods.paymentSatus" is possible both by single filter and multiple filter combinations.
3. Verify sorting by "startDate", "endDate", "workPeriods.memberHandle", "workPeriods.customerRate", "workPeriods.paymentStatus", "workPeriods.daysWorked" is possible.
4. Verify whether filtering and sorting by other fields in RB and WP is also possible (id, rateType, startDate, endDate, customerRate, memberRate, status, workPeriods.userHandle, workPeriods.daysWorked, workPeriods.customerRate, workPeriods.memberRate, workPeriods.paymentStatus)
5. Verify whether filtering by invalid attributes returns error.
6. Verify error returned if the field param has sorting by workperiods.* and no WP start /end date

[sandhiyakavi]

Verified on Prod Env with @nkumar-topcoder .