Skip to content

Conversation

winklerj
Copy link
Contributor

What's Changed

  • We've refactored the code to properly use axios and the new message structure
  • The unit tests were broken and should now pass
  • Added 2 types of pagination (see below)
  • Added the ability to rebound based on a JSONata expression applied to the response. For example if you are checking a status in the response or whether a resource exists (which isn't reflected in the status code with a 404)
  • Added backwards compatibility between Ferryman 1.7.x and 2.0.0 based on presence of environment variable used for rebound
  • REST component now handles all the authentication types available in the secret service
  • Ability to override the http response codes to rebound on
  • Ability to apply a JSONata transformation to the response body and store in the snapshot. This can be useful for paging scenarios
  • Ability to emit the incoming message along with the response
  • Ability to retrieve assets from the attachment storage and put downloaded assets into the attachment storage service

Pagination
For the pagination we added two types with the thought to eventually support more.

  1. Polling each page using the snapshot as suggested by elastic.io https://docs.elastic.io/developers/how-to-implement-the-pagination.html
  2. Looping through all pages in one execution of a REST component step

The difference is that #1 allows a consistent flow of data and you can adjust the interval and page size. #2 allows a flow creator to better reason about how they are consuming data and allows closer to real-time syncing.
Right now we are supporting pagination that exists in the URL and in the body. As an example we are using pagination as a URL parameter and Shopify uses a cursor in the body of the GraphQL request

Robb Winkle and others added 30 commits June 25, 2021 10:21
…sform and ability to apply transform to response body and save to the snapshot. This can be used for paging. Converted to new OIH message format and to actually work with axios config and response. Fixed all unit tests. Fixed form data submissions. Added url transform back in. Fixed linting errors. Still one failing test for attachment processing which is missing/removed.
…g auth configuration. Commented out attachments unit test temporarily until functionality added back in. Changed test log level to fatal to avoid logging during build tests
…re in progress. I'm going to create a version without the attachment functionality fully done.
Bumps [yarn](https://github.com/yarnpkg/yarn) from 1.7.0 to 1.22.0.
- [Release notes](https://github.com/yarnpkg/yarn/releases)
- [Changelog](https://github.com/yarnpkg/yarn/blob/master/CHANGELOG.md)
- [Commits](yarnpkg/yarn@v1.7.0...v1.22.0)

---
updated-dependencies:
- dependency-name: yarn
  dependency-type: direct:development
...

Signed-off-by: dependabot[bot] <support@github.com>
…st and requeue it to the component after that delay period has passed.
Added jsonataResponseValidator configuration to rebound/delay a reque…
…In our case we want to add the 404 status code to keep checking until a REST resource exists.
Added ability to configure which error status codes cause a rebound. …
Fixed the output for the error message
winklerj and others added 26 commits October 7, 2021 16:22
Update utils.js. Whitespace deployment
Switch docker build from npm to yarn to reflect lock file, fixed issu…
… to pull in the latest version of ferryman
Reverted changes to master.yml to see if that caused the docker image…
Removed build cache to fix wrong ferryman library somehow being included
Switched ferryman version definition from caret to tilde since versio…
Reversed disabling cache since it wasn't the root cause of the ferryman issue
Updating version number for next release
Updated ferryman version to the latest
…any pre-release tags and will also make the package.json version match the git tag version using the npm version command
Created separate GitHub Action workflow for tags. It will now ignore …
Added snapshot to process method
Copy link

@SvenHoeffler SvenHoeffler left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A good number of fixes and enhancements. I currently see no issue adding them.

@heggert heggert self-requested a review December 6, 2021 11:22
@IoannisLafiotis IoannisLafiotis merged commit c48d173 into openintegrationhub:master Dec 7, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

6 participants