A clojure application designed to receive and store EDI messages from the TBT Cloud into a customer’s local database. The solution is intended to be quite universal so is actively used to integrate other platforms (like, for example, DKDHUB) and applications into the robust and flexible digital environment.
Latest release assets available here.
- NOTE
-
Default configuration implies PG and rely on upstream versioning
There are a lot of possible customisations available as well as adding a new schemas or changing the schema sources. On order to get more information, please use the contact form here
In order to build and run with Makefile you should first install docker, have Leiningen installed and resolve build dependencies.
- NOTE
-
Default JDK/JRE may vary between distros, but by default Java 11+ is intended
sudo apt-get install make default-jdk-headless
Standalone run means you have external PostgreSQL server and successfully prepared your configuration
## 1st time to compile make compile ## 1st time to synchronize schemas and initialize tables make run-sync ## normally just make run
Install docker first, then run
sudo docker build -t edi-receiver -f docker/Dockerfile.pg .
sudo docker build -t edi-receiver -f docker/Dockerfile.mysql .
sudo docker run -it edi-receiver
sudo apt-get update sudo apt-get install wget default-jdk sudo wget https://raw.githubusercontent.com/technomancy/leiningen/stable/bin/lein -P /bin sudo chmod a+x /bin/lein lein upgrade
sudo apt-get update sudo apt-get install -y wget gnupg wget --quiet -O - https://www.postgresql.org/media/keys/ACCC4CF8.asc | sudo apt-key add - sudo sh -c 'echo "deb http://apt.postgresql.org/pub/repos/apt/ `lsb_release -cs`-pgdg main" >> /etc/apt/sources.list.d/pgdg.list' sudo apt-get update sudo apt-get install postgresql
su - postgres -c "psql --command \"ALTER USER postgres WITH PASSWORD 'postgres';\"" su - postgres -c "psql --command \"CREATE DATABASE edi;\""
Config is java .properties file:
| parameter | type | description |
|---|---|---|
upstream.topics |
string |
comma separated list of topics to receive |
upstream.cache |
string |
path where upstream schemas cached |
upstream.sync |
boolean |
set true to validate upstream cache on startup |
api.host |
string |
http server ip address |
api.port |
integer |
http server port, default is 8000 |
api.auth.username |
string |
basic auth username, if auth required |
api.auth.password |
string |
basic auth password, if auth required |
db |
string |
which storage to use, "pg" or "mysql". Options from |
pg.host |
string |
postgresql host, default is "localhost" (from jdbc.host) |
pg.port |
integer |
postgresql port, default is 5432 |
pg.database |
string |
postgresql database, default is "edi" (from jdbc.database) |
pg.user |
string |
postgresql user |
pg.password |
string |
postgresql password |
mysql.host |
string |
mysql host, default is "localhost" (from jdbc.host) |
mysql.port |
integer |
mysql port, default is 3306 |
mysql.database |
string |
mysql database, default is "edi" (from jdbc.database) |
mysql.user |
string |
mysql user |
mysql.password |
string |
mysql password |
See edi-receiver.properties for defaults and more options.
cp resources/edi-receiver.properties local.properties
Then edit local.properties file and run
java -jar target/edi-receiver-standalone.jar -c local.properties
Default properties will be updated with local.properties
Some config options can be customized from command line, see help:
java -jar target/edi-receiver-standalone.jar --help
Sample configuration for proxying
backend.0.name="dear-kafka"
backend.0.type="kafka"
backend.0.bootstrap-servers="127.0.0.1:9092"
backend.0.enabled=true
backend.1.name="weak-connection-api"
backend.1.type="http"
backend.1.method="post"
backend.1.uri="http://my.host/api/topic/{topic}" ; {topic} will be substituted with proxy.N.target
backend.1.auth.enabled=true
backend.1.auth.type="basic"
backend.1.auth.username="test"
backend.1.auth.password="pass"
backend.1.enabled=true
proxy.0.backend="dear-kafka"
proxy.0.source="document"
proxy.0.target="proxy_document" ; kafka topic name
proxy.0.enabled=true
proxy.0.reliable=true ; means not to accept message if kafka producer fails
; optional buffer configuration, used only with reliable=true
proxy.0.buffer.enabled=true ; when true, if producer fails, keep message in buffer
proxy.0.buffer.max-size=1000 ; maximum buffer size, will return 422 in case of buffer owerflow
proxy.0.buffer.max-tries=3 ; retries number to send message, then message will be dropped
proxy.0.buffer.expire-time-s=180 ; message expiration time, seconds, then message will be dropped, can be nil (= forever)
proxy.0.buffer.tries-interval-ms=10000 ; minimal retry interval, default is 10000
; optional logging configuration. Used to store backend responses in "log" table
proxy.0.logging.enabled=true
proxy.0.logging.reference-fields="doctype,id" ; selected field names, stored as reference
proxy.1.backend="weak-connection-api"
proxy.1.source="document"
proxy.1.target="proxy_document"
proxy.1.condition="(= sender \\"tbt\\")"
proxy.1.transform="[[body (str body) (some? body)]]"
proxy.1.enabled=true
proxy.1.reliable=false ; means to ignore api fails
Backend responses can be stored into log table.
| Column | Postgresql type | Mysql type | Fescription |
|---|---|---|---|
|
|
|
time record created at |
|
|
|
Proxy configuration, e.g.: |
|
|
|
Message reference. Selected message keys, as defined in setting |
|
|
|
Backend response |
|
|
|
Backend response raw content, HTTP backend only |
То enable logging, add following lines to proxy configuration:
proxy.0.logging.enabled=true proxy.0.logging.reference-fields="field1,field2"
Setting reference-fields is mandatory. log.reference will be stored like:
{"field1": "value1", "field2": "value2"}
Transformation rules:
[:restrictive ; optional modifier, makes rules restrictive [target.path <expression> <condition>] [another.path <expression2>] ; condition can be optional ...]
Restrictive means to start result from empty map. For example, following rules
[:restrictive [a (str (+ aa 10)) (odd aa)] ; aa is odd, so evaluates [a (str aa) (even aa)] ; no hit as aa is not even [b bb] ; simple assignment [c cc (some? cc)] ; assign c from cc if cc is not nil [new 777]] ; constant new value
applied to
{:aa 1
:bb 2
:cc nil}
will produce
{:a "11"
:b 2
:new 777}
Non-restrictive rules can be used to update or drop some values, for example, rules
[[aa (str (+ aa 10)) (odd aa)] ; aa is odd so update it [aa (str aa) (even aa)] ; no hit as aa is not even [cc :dissoc] ; remove cc anyway [new 777]] ; constant new value
applied to
{:aa 1
:bb 2
:cc nil}
will produce
{:aa "11"
:bb 2 ; there was no rule on bb
:new 777}
Warning: benchmarking will affect your database and backends!
To send test requests to receiver, run
java -jar target/edi-control-standalone.jar --fire fire-config.json
where fire-config.json is file with content like:
{
"threads": 10,
"requests-per-thread": 1000,
"payloads": [
{
"uri": "http://localhost:8000/api/topic/document",
"method": "post",
"auth": {
"enabled": true,
"type": "basic",
"username": "test",
"password": "pass"
},
"headers": {
"Content-Type": "application/json"
},
"body": "{\"sender\":\"tbt\",\"timestamp\":\"2004-10-19 10:23:54+02\",\"doctype\":\"DocReturnOrders\",\"id\":\"3d9cddc1-f71d-0d6a-6343-010ccfc800a6\",\"body\":{\"a\":\"b\"},\"checksum\":\"fd6b34c54a331ccbbfacade3ea4bd225\"}"
}
]
}
edi-control will run 10 threads with 1000 requests per thread and calculate average request time and request count per second. If multiple payloads supplied, payload for each request will be selected randomly.
Execute (go) in repl to start development profile with autoreload and local.properties config.
Copyright © 2020 Kasta Group LLC
Copyright © 2020-2022 TBT LLC
This program and the accompanying materials are made available under the terms of the Eclipse Public License 2.0 which is available at http://www.eclipse.org/legal/epl-2.0.
This Source Code may also be made available under the following Secondary Licenses when the conditions for such availability set forth in the Eclipse Public License, v. 2.0 are satisfied: GNU General Public License as published by the Free Software Foundation, either version 2 of the License, or (at your option) any later version, with the GNU Classpath Exception which is available at https://www.gnu.org/software/classpath/license.html.