Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature/#1263 transfer metrics #659

Merged
merged 93 commits into from
Apr 6, 2020
Merged
Show file tree
Hide file tree
Changes from 88 commits
Commits
Show all changes
93 commits
Select commit Hold shift + click to select a range
b543168
added kafka partitioning algorithm config so that it is aligned with …
mdebarros Jan 8, 2020
1e08b49
Updated dependencies, mainly for event-sdk; and also reverted version…
mdebarros Jan 8, 2020
ef3ea36
Updated circleci config to support incremental snapshot.# releases
mdebarros Jan 8, 2020
d79dfac
Fix for previous update to circleci config to support incremental sna…
mdebarros Jan 8, 2020
c9ba618
updated event-sdk dependencies
mdebarros Jan 9, 2020
ad04487
updated event-sdk dependencies
mdebarros Jan 9, 2020
94f4bcb
updated event-sdk and central-services-shared
Jan 10, 2020
16516e1
#1163 - Allow simple health check (#543)
oderayi Jan 13, 2020
0bd358c
Restore central-services-shared dependency version to 8.8.0-snapshot.…
oderayi Jan 13, 2020
94d18cd
added increased threadpool for central-ledger for perf testing (#553)
rmothilal Jan 14, 2020
1d009e2
Feature/#1140 host logging enhancement optimizations (#570)
rmothilal Jan 24, 2020
71cb0c5
bump central-services-stream to 9.1.0-snapshot
yosheeck Feb 12, 2020
6107e6f
bump central-services-stream to 8.7.2-snapshot
yosheeck Feb 12, 2020
a6f55e7
postponed failing audit checks
yosheeck Feb 12, 2020
7965116
Added labels for action on prepare handler metrics
mdebarros Feb 13, 2020
b040c5c
Added labels for action on prepare handler metrics
mdebarros Feb 13, 2020
27401bf
initial perf-test commit
yosheeck Feb 25, 2020
86fdac6
added perconaDb deploy
yosheeck Feb 25, 2020
a53f69d
helm cleanups and config changes
yosheeck Feb 25, 2020
d1b54c4
helm charts for central-services
yosheeck Feb 25, 2020
ad3f399
added perf prepare service
Feb 25, 2020
f5f4e0e
Merge remote-tracking branch 'upstream/release/9.2.1-perf' into relea…
yosheeck Feb 25, 2020
6c264fb
Merge remote-tracking branch 'origin/release/9.2.1-perf' into release…
mdebarros Feb 25, 2020
0358a43
Merge remote-tracking branch 'origin/release/v9.2.1-perf' into releas…
mdebarros Feb 25, 2020
41734f0
updates for perf testing
mdebarros Feb 25, 2020
4952c30
fixes for perf test
mdebarros Feb 25, 2020
9228f69
list of dfsps and run duration added
Feb 26, 2020
1fe5283
fixes for perf test scripts, also disable autocommits for the transfe…
mdebarros Feb 26, 2020
473d971
Merge remote-tracking branch 'origin/release/v9.2.1-perf' into releas…
mdebarros Feb 26, 2020
115ad7f
updates for perf testing
mdebarros Feb 26, 2020
8749e36
fixed for random fsp generation
mdebarros Feb 26, 2020
37a8948
Merge branch 'master' into release/v9.2.1-perf2
yosheeck Feb 26, 2020
c45b2b3
updated prepare handler values
mdebarros Feb 26, 2020
18fca3d
Merge branch 'release/v9.2.1-perf' into release/v9.2.1-perf2
mdebarros Feb 26, 2020
bd5824e
wip instrumentation positions
Feb 26, 2020
69879f6
fixed merge problem which introduced caching bug
yosheeck Feb 26, 2020
e962569
Merge branch 'release/v9.2.1-perf2' of github.com:mojaloop/central-le…
mdebarros Feb 26, 2020
b8149cd
fixies for perf test scripts
mdebarros Feb 26, 2020
cc1b7d8
Fixes for facade_getByNameAndCurrency metrics, and upgraded perf dock…
mdebarros Feb 27, 2020
fbd9d77
implemented cached version of participant-facade::getByNameAndCurrency()
yosheeck Feb 27, 2020
9e2cb1c
Modified Prepare Handler to use a basic insert algorithm for duplciat…
mdebarros Feb 27, 2020
b1a8a65
wip
Feb 27, 2020
1d140bd
Merge branch 'release/v9.2.1-perf2' of github.com:mojaloop/central-le…
Feb 27, 2020
ca66501
Merge branch 'release/v9.2.1-perf2' of github.com:mojaloop/central-le…
mdebarros Feb 27, 2020
f3fe0da
fixed bad merge
yosheeck Feb 27, 2020
575de2a
updated perf helm values config
mdebarros Feb 27, 2020
621da2b
updated perf helm values config
mdebarros Feb 27, 2020
4d19fb4
fixed missing logs from docker file for perf tests, and also update d…
mdebarros Feb 27, 2020
72b262d
merged position runner into the perf code from 9.2.1-perf2.1
Feb 27, 2020
2cf3b7c
updated perf db deploy script to add metrics annotations
mdebarros Feb 27, 2020
b83147e
merged with latest perf branch
Feb 28, 2020
2e4e855
func call of positions:
Mar 2, 2020
4dc75de
merged histGetParticipantLimitEnd histogram from releases/v9.2.1-perf3
yosheeck Mar 2, 2020
eb81d49
Updated perf script to PREPARE_ENABLED_DUPLCIATE_INSERT_ONLY to set I…
mdebarros Mar 2, 2020
13f381a
optimised the dockerfile for the central-ledger
mdebarros Mar 2, 2020
3cb0652
merge with perf2.1
Mar 3, 2020
0a0cb8d
merge with perf2.1
Mar 3, 2020
3d02d25
added PRODUCE_TO_POSITION_ENABLED env flag to produce to topic or to …
Mar 3, 2020
4d40a8d
fixes for perf scripts for position handler tests
mdebarros Mar 3, 2020
926ba74
Merge branch 'release/v9.2.1-perf2.1' into release/v9.2.1-perf3
Mar 3, 2020
e3a2acb
fulfil to position
Mar 3, 2020
b3ef7ae
wip
Mar 3, 2020
575a757
finished fulfil
Mar 4, 2020
2032ee3
mergin with perf3
Mar 4, 2020
82209f8
work in progress on combined prepare-position handler
Mar 5, 2020
7789cfe
Merge branch 'release/v9.2.0-perf' into release/v9.2.1-perf4
Mar 5, 2020
149e123
Added INSERT_ONLY logic to the fulfil handler
mdebarros Mar 5, 2020
13b445f
added stream-lib with optimised logging
mdebarros Mar 5, 2020
3af15ff
work in progress on POC combined prepare-position handler
Mar 6, 2020
1d79841
DFSP seeding script for quick creation of 10-s of participants
yosheeck Mar 6, 2020
142f0c0
Added tracestate tag support
mdebarros Mar 10, 2020
7221cfd
Added tracestate tag support
mdebarros Mar 10, 2020
7a5e0fe
Merge branch release/v9.2.1-perf4-jeb into release/v9.2.1-perf4.1
mdebarros Mar 10, 2020
edb41ee
added ability to disable handler modules on full registration
mdebarros Mar 10, 2020
ce0450a
fixed state of combined prepare-position to set state correctly after…
mdebarros Mar 12, 2020
c8819f8
Merge branch 'master' into release/v9.2.1-perf4.2
yosheeck Mar 16, 2020
bcac10c
wip
Mar 18, 2020
4afa3c9
combined fulfil and position handlers and transaction
Mar 23, 2020
b71616e
finished combined with the knexraw approach
Mar 24, 2020
df310c7
added env RAW_FULFILPOSITION to select calculation db or be
Mar 24, 2020
d29e9de
Merge branch 'feature/#1268-fulfilPosition' into release/v9.4.0-perf1.0
yosheeck Mar 24, 2020
aac3f3b
Fix for migration issue due to migration script being renamed.
mdebarros Mar 24, 2020
f7f08ba
Fixed typo for fulfilPosition CLI
mdebarros Mar 24, 2020
c1bf9ca
Removed double UV_THREADPOOL_SIZE setting
yosheeck Mar 25, 2020
4c23b33
Added TODO comments on combined prepare-position handler logic
mdebarros Apr 3, 2020
a0a1833
WIP merge metrics
Apr 6, 2020
35099f4
small fixes
Apr 6, 2020
2bde9e0
Merge branch 'master' into feature/#1263-transfer-metrics
vgenev Apr 6, 2020
ba3a78b
removed unused vars from .env
Apr 6, 2020
7caae50
Merge branch 'master' of github.com:mojaloop/central-ledger into feat…
Apr 6, 2020
91d665d
removed unused vars from .env
Apr 6, 2020
b2affda
Merge branch 'feature/#1263-transfer-metrics' of github.com:vgenev/ce…
Apr 6, 2020
9f58d98
switched image for simulator to simulator:latest version used in dock…
Apr 6, 2020
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 5 additions & 2 deletions .env
Original file line number Diff line number Diff line change
Expand Up @@ -8,10 +8,13 @@ LOG_LEVEL="info"
LOG_FILTER="error, warning, info"
EVENT_SDK_LOG_FILTER="audit:*, log:info, log:warning, log:error"
EVENT_SDK_LOG_METADATA_ONLY="true"
EVENT_SDK_SIDECAR_DISABLED="false"
EVENT_SDK_SIDECAR_DISABLED="true"
EVENT_SDK_SERVER_HOST="localhost"
EVENT_SDK_SERVER_PORT="50051"
EVENT_SDK_VENDOR_PREFIX="mojaloop"
EVENT_SDK_TRACESTATE_HEADER_ENABLED="true"
EVENT_SDK_TRACESTATE_HEADER_ENABLED="false"
EVENT_SDK_ASYNC_OVERRIDE_EVENTS="log, trace"
EVENT_SDK_TRACEID_PER_VENDOR="false"
PREPARE_PRODUCE_TO_POSITION_ENABLED="false"
FULFIL_PRODUCE_TO_POSITION_ENABLED="false"
RAW_FULFILPOSITION="false"
12 changes: 7 additions & 5 deletions Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -7,12 +7,8 @@ RUN apk add --no-cache -t build-dependencies git make gcc g++ python libtool aut
&& npm install -g node-gyp

COPY package.json package-lock.json* /opt/central-ledger/
RUN npm install

COPY src /opt/central-ledger/src
COPY config /opt/central-ledger/config
COPY migrations /opt/central-ledger/migrations
COPY seeds /opt/central-ledger/seeds
RUN npm install

FROM node:12.16.0-alpine
WORKDIR /opt/central-ledger
Expand All @@ -21,6 +17,12 @@ COPY --from=builder /opt/central-ledger .

RUN npm prune --production

COPY src /opt/central-ledger/src
COPY config /opt/central-ledger/config
COPY migrations /opt/central-ledger/migrations
COPY seeds /opt/central-ledger/seeds
COPY test /opt/central-ledger/test

# Create empty log file & link stdout to the application log file
RUN mkdir ./logs && touch ./logs/combined.log
RUN ln -sf /dev/stdout ./logs/combined.log
Expand Down
20 changes: 13 additions & 7 deletions config/default.json
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@
"RECONNECT_INTERVAL": 5000
},
"MONGODB": {
"DISABLED": false,
"DISABLED": true,
"URI": "mongodb://localhost:27017/mlos"
},
"ERROR_HANDLING": {
Expand Down Expand Up @@ -307,7 +307,8 @@
"queue.buffering.max.messages": 10000000
},
"topicConf": {
"request.required.acks": "all"
"request.required.acks": "all",
"partitioner": "murmur2_random"
}
}
}
Expand All @@ -327,7 +328,8 @@
"queue.buffering.max.messages": 10000000
},
"topicConf": {
"request.required.acks": "all"
"request.required.acks": "all",
"partitioner": "murmur2_random"
}
}
},
Expand All @@ -345,7 +347,8 @@
"queue.buffering.max.messages": 10000000
},
"topicConf": {
"request.required.acks": "all"
"request.required.acks": "all",
"partitioner": "murmur2_random"
}
}
},
Expand All @@ -363,7 +366,8 @@
"queue.buffering.max.messages": 10000000
},
"topicConf": {
"request.required.acks": "all"
"request.required.acks": "all",
"partitioner": "murmur2_random"
}
}
}
Expand All @@ -383,7 +387,8 @@
"queue.buffering.max.messages": 10000000
},
"topicConf": {
"request.required.acks": "all"
"request.required.acks": "all",
"partitioner": "murmur2_random"
}
}
}
Expand All @@ -403,7 +408,8 @@
"queue.buffering.max.messages": 10000000
},
"topicConf": {
"request.required.acks": "all"
"request.required.acks": "all",
"partitioner": "murmur2_random"
}
}
}
Expand Down
6 changes: 4 additions & 2 deletions docker-compose.yml
Original file line number Diff line number Diff line change
Expand Up @@ -150,16 +150,18 @@ services:
start_period: 40s

simulator:
image: mojaloop/simulator:latest
image: mojaloop/simulator-perf:m9.2.1-perf4.1.2
container_name: cl_simulator
ports:
- "8444:8444"
environment:
- LOG_LEVEL=info
- TRANSFERS_ENDPOINT=http://ml-api-adapter:3000
- TRANSFERS_ENDPOINT=http://host.docker.internal:3000
- QUOTES_ENDPOINT=http://host.docker.internal:3002
- PARTIES_ENDPOINT=http://host.docker.internal:4002
- TRANSFERS_FULFIL_RESPONSE_DISABLED=false
- TRANSFERS_CONDITION=HOr22-H3AfTDHrSkPjJtVPRdKouuMkDXTR4ejlQa8Ks
- TRANSFERS_FULFILMENT=XoSz1cL0tljJSCp_VtIYmPNw-zFUgGfbUqf69AagUzY
networks:
- cl-mojaloop-net
healthcheck:
Expand Down
38 changes: 38 additions & 0 deletions migrations/310203_transferParticipant-indexes.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
/*****
License
--------------
Copyright © 2017 Bill & Melinda Gates Foundation
The Mojaloop files are made available by the Bill & Melinda Gates Foundation under the Apache License, Version 2.0 (the "License") and you may not use these files except in compliance with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, the Mojaloop files are distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.
Contributors
--------------
This is the official list of the Mojaloop project contributors for this file.
Names of the original copyright holders (individuals or organizations)
should be listed with a '*' in the first column. People who have
contributed from an organization can be listed under the organization
that actually holds the copyright for their contributions (see the
Gates Foundation organization for an example). Those individuals should have
their names indented and be marked with a '-'. Email address can be added
optionally within square brackets <email>.
* Gates Foundation
- Name Surname <name.surname@gatesfoundation.com>

* ModusBox
- Georgi Georgiev <georgi.georgiev@modusbox.com>
--------------
******/

'use strict'

exports.up = function (knex) {
return knex.schema.table('transferParticipant', (t) => {
t.index(['transferId', 'transferParticipantRoleTypeId', 'ledgerEntryTypeId'], 'getTransferInfoToChangePosition')
})
}

exports.down = function (knex) {
return knex.schema.table('transferParticipant', (t) => {
t.dropIndex(['transferId', 'transferParticipantRoleTypeId', 'ledgerEntryTypeId'], 'getTransferInfoToChangePosition')
})
}
Loading