All notable changes to this project will be documented in this file.
The format is based on Keep a Changelog, this project adheres to Semantic Versioning.
- Fixed a bug when running jq with iterator that caused the integration to crash
- Reverted image to
python:3.11-slim-busterto fix the issue with the alpine image
- Replace
python:3.11-slim-bookwormwithpython:3.11-alpineto reduce dependencies and fix vulnerabilities
- Fixed smoke tests to run concurrently and clean up after themselves
- Replace pyjq with jq.py to bump jq version from 1.5.2 to 1.7.1
- Fixed updating state of resync when the resync is being cancelled by a new resync event
- Add smoke test with a live integration to validate core changes
- Fixed failing on initialization of the integration when one of the actions exists in port
- Added fix lint command to the makefile as well as the pre-commit hook
- Replaced StopAsyncIteration with a return statement to ignore prevent errors in cases where empty tasks are sent to the stream_async_iterators_tasks function
- Avoid raising exception when receiving ReadTimeout on batch upsert entities
- Increased both internal port client and third party client timeout to handle long requests
- Add search identifier support (Allow to run a search query to find the identifier of the entity as part of the mapping)
- Fixed error log when looking for existence of integration on initialization
- Test support and helpers
- Fixed upsert entity failure when saving modified data for search relations calculations
- Bugfix Semaphores get fail when moving to the next scheduled resync when syncing a large number of entities, using a single event loop for all threads
- Reverted last bugfix
- Fixed unhashable type: 'dict' error when trying to delete entities with search identifier/relations
- Add support for reporting the integration resync state to expose more information about the integration state in the portal
- Fix kafka listener never ending resync loop due to resyncState updates
- Fixed an issue causing the cli to fail in a directory with no pyproject.toml in it
- Changed action CREATE route to use new v2 option
- Fixed resync issue when calculating the diff of entities failed due to search identifier in relation mapping
- Not showing misleading error message if port state is empty
- Fixed & Aligned scaffolding files
- Fixed an issue where passing an object for OCEAN__INTEGRATION__CONFIG that holds an object might not be parsed correctly and cause validation error for invalid type (#1)
- Fixed an issue where a
ValueErrorwas raised inunregister_rawmethod due to incorrect unpacking of results from asyncio.gather. The fix involved using zip to properly handle the output and ensure both entities and errors are processed correctly.
- Fix vulnerabilities and bump versions of dependencies
- Add python-dateutil to the core dependencies
- Fix misspelling in the
bump-all.shscript
- Flush all remaining buffered logs when exiting application
- Initialize missing _port_app_config
- Handle non existing config mapping for cases where the integration was created by SAAS and the config mapping was not set
- Added Ocean integration config to remove all environment variables from jq access
- Added log for when receiving invalid port app config mapping
- Added log of the used integration mapping for each resync event
- Added log when failed on processing jq mapping for raw result
- Fixed an issue where raw results were not being sent if raw data didn't map to any entity
- Safely get changelogDestination key instead of accessing it directly
- Added validation of whether the integration can run in the desired runtime
- Add search relation support (Allow to to run a search query to find the relation to the entity as part of the mapping)
- Fixed values unpack error in register_raw
- Added pydantic's dotenv extra to the core dependencies for reading .env files on the integration startup
- Added .python-version to the repository for easier setup with pyenv install
- Changed initialization to always apply default mapping if no other mapping is configured
- Fixed incorrect pydantic validation over the integration settings
- Fixed an issue causing integrations with no configuration to fail during the initialization process
- Fixed faulty error handling caused by gather_and_split_errors_from_results raising errors that are not directly under BaseException (#1)
- Improved exception propagation for the entity processing (#1)
- QOL utility (
core.utils.gather_and_split_errors_from_results) for when callingasyncio.gatherwith thereturn_exceptionsparameter set toTrueand there is need for separating the errors from the data itself (#2)
- Fixed unhandled exceptions caused by the entity parsing, resulting in the integration freezing (#1)
- Updated the base image used in the Dockerfile that is created during integration scaffolding from
python:3.11-slim-bustertopython:3.11-slim-bookworm
- Fixed an issue in
send_raw_data_exampleswhen there are slashes in integration kind
- Added
send_raw_data_examplesintegration config to allow sending raw data examples from the third party API to port (on resync), for testing and managing the integration mapping
- Made config.yaml file optional in the integration setup process.
- Integration type is now determined by the name specified in the pyproject.toml file.
- Switched to using the FastAPI lifespan feature instead of the deprecated on_shutdown and on_start methods.
- Fixed the FastAPI server staying stale after shutdown by using the FastAPI lifespan feature for handling shutdown signals, preventing override of the shutdown process.
- Fixed issue with integration continuing to run after shutdown by canceling the resync async generator task.
- Added caching to port-app-config.yml retrieval from port api (only for live events)
- Added a util function that allows to run multiple asynchronous tasks in a bounded way to prevent overload and memory issues
- Use that utility when calculating JQ mapping for raw entities
- Fixed an issue in creating a child event context from the parent context by removing an unnecessary line of code
- Allowing override of parent event context in ocean's event context manager
- Fixed error in
register_rawwhen there's no relevant mappings for a specific kind
- Implemented real-time entity deletion exclusively for instances that haven't matched any selectors.
- Change the JQ calculation to process only identifier and blueprint for raw entities not selected during real-time events to only get the required data for the delete.
- Delete entities that doesn't passed the selector on real time events
- Added a util function that allows to iterate over a list of async iterators and stream the results of each iterator as they are available
- Improved the handling of integration entities by adding retries and running it after the upsert to prevent blocking the resync
- Changed entities search timeout to 30 seconds to prevent blocking the resync
- Added a way to enable request retries for any request even if its request method is not part of the retryable methods
- Fixed application settings to be loaded from the environment variables
- Added integration version label to docker
- Fixed a bug where every time after the first token expiration, the framework didn't actually marked that the token got refreshed, causing the token to be refreshed every time when a request is made to Port. (#1)
- Fixed a bug in loguru which fails to deserialize an exceptions (#1)
- Added the ability to map entities from raw array attributes by introducing
itemsToParsekey in the mapping configuration
- Added array to possible integration configuration types (PORT-7262)
- Changed caching to detect changes in params of function (#1)
- Fixed an issue where a failure in the entity processing step might fail the whole resync (#1)
- Cahnged the JQ Entity processor to work with async callss to allow better parallelism and async work (#1)
- Fixed an issue causing the integration to crash when passing a sensitive configuration with invalid regex characters due to a missing escaping (PORT-6836)
- Added handling for kafka consumer empty partition assignment and shutting the application down with an error (PORT-5475)
- Added QOL decorator to help with caching the third party response (PORT-5475_2)
- Changed the Kafka consumer to run in the event loop in async instead of sync in another thread (PORT-5475)
- Fixed an issue causing all the character to be redacted when passing empty string to a sensitive field
- Added a method for ocean integration to redact sensitive information from the logs and automatically apply it to sensitive configurations and known sensitive patterns. (#1)
- Added an HTTP handler for Ocean logs to facilitate sending the logs to the Port. (#2)
- Seperated the
port_ocean.utilsfile into multiple files within theutilsfolder to improve code organization. (#1) - Changed the Ocean context to be a global variable instead of using Localstack, preventing the framework from re-initiating the context for each thread. (#2)
- Fixed an issue where the event listener was causing the application to continue running even after receiving a termination signal. (#1)
- Fixed a bug that caused some termination signal handlers to not work by consolidating the signal listeners in a single class, as signals can only have one listener. (#2)
- Added sonarcloud files for public integration scaffolding (PORT-6181)
- Replaced the
remove-dockeroption from theocean newcli withprivate&publicflags (PORT-6181)
- Increased the default timeout for requests to 3rd party targets to 30 seconds, and made it configurable (PORT-6074)
- Fixed issue causing app config with no team mapping to fail due the core using None when not set (PORT-5938)
- Fixed missing team parameter in the port app config model (PORT-5938)
- Added capability to create pages as part of the integration setup (PORT-5689)
- Added integration and blueprints existence check before creating default resources (#1)
- Added verbosity to diff deletion process after resync (#2)
- Fixed
ocean newscaffolding error'collections.OrderedDict object' has no attribute 'public_integration'(PORT-5728)
- Added handling for aggregation properties when initializing the integration, so it will patch the aggregation properties after creating the relations (PORT-5717)
- Changed entity property in the
portResourceConfigto be required instead of optional, as we don't support creation of blueprints as part of the app config (PORT-4549)
- Wrapped the httpx async client with implementation that overrides the default transport class with custom transport to apply all default httpx features that are ignored when passing a custom transport instance. This allows the missing behevior of the http proxy environment variable (PORT-5676)
- Changed deprecated
poetry lock --checkin the make files topoetry check(PORT-5711)
- Changed the way we upsert and delete bulk of entities from the catalog to be batched rather than spawning all requests at once
- Added a way to create the integration without the Dockerfile and .dockerignore to use the global Docker files when scaffolding a new integration.
- Fixed the incorrect search of entities by datasource, which was causing entities from older versions not to be deleted. (PORT-5583)
- Allowing POST requests for getting port tokens to be retryable (PORT-5442)
- Changed the default limitations and timeouts for requests to Port in order to handle PoolTimeout error caused by a large amout of requests sent in parallel (PORT-5442)
- Fixed a bug that triggered the integration to update during the initialization process when the integration already existed and the organization lacked default blueprints (PORT-5378).
- Fixed an issue where setting integration type or identifier that contains a capital letter will not show the integration in the UI (PORT-5399)
- Added handling for transport errors like connection timeout error for outbound requests from ocean integrations and core (PORT-5369)
- Changed port request option
mergeto be true by default (PORT-5396)
- Changed the port request options defaults to be constructed in the port app config model instead of setting the defaults in many places (PORT-5369)
- Added a httpx client that recreate itself on new threads using localproxy & localstack bundled with the
RetryTransporttransport featured in 0.4.3 (PORT-5333)
- Added
TokenRetryTransportto the port client httpx client to handle connection errors and create new access tokens when the token is expiring while requesting (PORT-5333) - Removed the retry handler decorator from the port client. Now using the
TokenRetryTransport(PORT-5333) - Handled
CycleErrorfor cyclic dependency in entities with better error message and ocean exception class (PORT-5333)
- Added
RetryTransportas a helper for retrying requests that integrations can use (PORT-5161)
- Fixed kafka consumer to poll messages asynchronously, to avoid max poll timeout when running long resyncs (PORT-5160)
- Fixed a bug where the expiration of a Port token is not properly handled (PORT-5161)
- Fixed a bug where the
retry_everydidn't count failed runs as repetitions (PORT-5161)
- Added the current integration version to the port requests for future features and better debugging (PORT-4310)
- Added the
install/prodcommand to the integration scaffold template as was intended (PORT-5107) - Changed the serializing of the port app config so when initializing it there wont be any None or default values displayed in the UI (PORT-5108)
- Removed version field from the spec.yml in the scaffolded integration (Version will be taken from the pyproject.toml) (PORT-5107)
- Changed the integration type in spec.yml to be the integration slug when scaffolding a new integration (PORT-5107)
- Added more logs to the ocean package for better debugging of the integration (PORT-4780)
- Seperated
SyncRawMixinfromSyncRawMixin(movedSyncRawMixintocore/integrations/mixins/sync_raw.py) - Improved code readability for
SyncRawMixin
- Fixed the
initialize-port-resourcesoption inocean sailto not be a flag. - Changed default of
initialize-port-resourcestotrue. - Catch all exceptions in the resync of ONCE event listener,to make sure the application will exit gracefully
- Added support for running ocean integrations once and new ocean sail options to support it. As part of it we added ImmediateEventListener.
- createMissingRelatedEntities + deleteDependentEntities are now defaulted to true
- Fix missing user agent when apply default resources on initialization (PORT-4813)
- Removed the
batch_work_sizeconfiguration. Integrations should use the async generator syntax instead (PORT-4616)
- Added support for a configurable resync interval for integrations (PORT-4616)
- Added a new feature that will abort a running resync if a new resync is attempting to start (PORT-4619)
- Changed the way an empty port app config is handled in the
PortAppConfig Handler(PORT-4483) - Added yaml linter (#1)
- Removed the Ocean version parameter from the integration scaffold template, the version is now queried directly from the Ocean framework library used by the integration (#2)
- Changed the publish integration workflow to get the integration version from the
pyproject.tomlfile of the integration and not from thespec.ymlfile (#3)
- Fixed a bug that rollbacked all blueprints instead of only those created during integration setup, when the setup encountered an issue with blueprint creation
- Fixed a bug that caused values that resulted with a falsy jq evaluation to convert them to null. The values will now be ingested using their proper falsy representation (0 as 0, empty array as empty array, false as false, etc.)
- Fixed the injections of parameters to the
config.yamlfile, the injected values will now be wrapped with""(#1)
- Added the ability to create and clean the defaults of an integration using the following CLI commands:
ocean defaults dockandocean defaults clean(dock-clean-defaults)
- Optimized dockerfile to produce smaller images (PORT-4485)
- Changed default log level to INFO in the cli
- Fixed an issue with loading the configuration from the environment variables if the config is a dictionary
- Move Resource Config Selector class to public
- Handled delete events from change log where there is no after
- Fixed an issue causing the config yaml providers to not be parsed
- Fixed an issue causing ocean to convert the integration config objects to camelized objects
- Updated the
on_resyncgenerator to use a list of items instead of a single item
- Changed default log level to
INFO - Changed the extra object messages log level from
INFOtoDEBUG - Removed a wrongful error log at the integration installation that says the integration does not exists
- Added support for many relations for the same entity (PORT-4379)
- Added the resource config to the event context (PORT-4398)
- Fixed lack of support for multiple relations (PORT-4411)
- Added traceback output to the integration resync method exception log (PORT-4422)
- Fixed an issue that caused the jq
Nonevalues for relations to become a string with the value"None"instead of being interpreted asnullin JSON
- Fixed an issue preventing the setup of an integration with config values passed exclusively as environment variables. This fix also enables the option to deploy an integration to AWS ECS using Terraform (PORT-4379)
- All integration configuration variables are now passed to the integration code in snake_case format
- Renamed
port_ocean.config.integration->port_ocean.config.settings
- All the settings can now be set using environment variables with prefix of
OCEAN__{The name of the field}and__between nested fields - The broker field in the kafka settings now has the Port production brokers as the default value
- Using pyhumps to automatically camelize the aliases of the settings
- Fixed a crash when there are no resources in the port-app-config
- Changed SAMPLE event listener to POLLING. (Make sure to update your
eventListener.typefield in yourconfig.yamlfor the integration) (PORT-4346)
- Seperated the cli commands to multiple files under the
port_ocean/cli/commandsfolder (PORT-4303) - Improved error messages from the PortClient (PORT-4337)
-
Fixed Webhook event listener not triggering
-
Fixed PortClient using httpx async client from another event loop
(PORT-4306)
-
Fixed
ocean newjinja crash for the config.yaml in the scaffold (PORT-4328) -
Fixed issue where the integration did not create the integration config on creation (PORT-4341)
-
Fixed an issue with initializePortResources that caused failure for unknown file names on init (PORT-4343)
-
-
Handlers
- Added entities state applier first port HTTP implementation.
- Added entity processor first jq implementation.
- Added port app config first port HTTP implementation.
-
Event Listeners
- Added KAFKA event listener.
- Added SAMPLE event listener.
- Added WEBHOOK event listener.
-
Core
- Added Ocean contexts & contexts global variables.
- Added validation to the integration config according to its
.port/spec.yaml. - Added a way to specify default resources to be created on installation.
- Added a new way to return data from the resync using generators.
- Added provider-based injection for the config yaml.
-
CLI
- Added
ocean listto list all public integrations in the port-ocean repo. - Added
ocean newto scaffold an Ocean project. - Added
ocean pullto pull one of the public integrations from the port-ocean repo. - Added
ocean sailto run the integration. - Added
ocean versionto get the framework version. - Added
make newto scaffold in the Ocean repository.
- Added
(PORT-4307)
-