Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Mon 11220 unified sql #112

Merged
merged 123 commits into from
Dec 8, 2021
Merged

Mon 11220 unified sql #112

merged 123 commits into from
Dec 8, 2021

Conversation

bouda1
Copy link
Collaborator

@bouda1 bouda1 commented Nov 30, 2021

New module unified_sql to replace sql and storage modules.

REFS: MON-11220

bouda1 added 28 commits December 3, 2021 19:21
The first application of this new feature is in progress.
storage creates the event, sends it to rrd.
rrd deserializes it and begins to read, but cannot for now
rebuild a graph.
Conflicts:
	grpc/centreon-rpc-client.py
Also, events are no more stored by category, just in one tree.
First step for service protobuf message started
Still an issue with ninja but it works with make
We have conflicts with storage and sql modules to work on now.
@sonarqube-decoration
Copy link

@bouda1 bouda1 merged commit 5b17fe7 into develop Dec 8, 2021
@bouda1 bouda1 deleted the MON-11220-unified_sql branch December 8, 2021 08:42
bouda1 added a commit that referenced this pull request Dec 10, 2021
* New output unified_sql
* packaging is now made globally, collect is compiled and then each part is packaged.
* centreon-tests has been completed with tests specific to unified_sql.

REFS: MON-11220
kduret pushed a commit that referenced this pull request Feb 10, 2023
* (stream/lib) add output limit

* fixing my dumb mistakes

* fix bad variable scope
kduret pushed a commit that referenced this pull request Feb 14, 2023
* (stream/lib) add output limit

* fixing my dumb mistakes

* fix bad variable scope
paloth added a commit that referenced this pull request Apr 26, 2023
* MON-15661 issue in anomaly detection ut tests (#460)

REFS:MON-15661

* MON-15830 add invalid_on_negative attribute develop (#448)

* MON-15830 add invalid_on_negative attribute develop

REFS:MON-15830

* enh(packaging): add rpm sign to workflow (#462)

* enh(ci): rebuild yum metadata only one time (#463)

* enh(ci): remove buster from build and delivery (#471)

* enh(ci): remove buster from build and delivery

* remove buster dockerfile

* MON-16170 all event string is sent to log if we can't parse output (#461)

REFS:MON-16170

* enh(ci): add repository_name and delivery (#476)

* enh(ci): add branch stability and delivery

* remove duplicate code

* enable delivery tests

* add deb delivery debug

* add deb delivery debug

* fix credentials

* disable deb delivery tests & debug

* Mon 15428 convert kpi event to pb (#441)

REFS:MON-15428
* MON-15428 use of KpiEvent in place of kpi_event

* MON-15428 fmt 9.1 and PR comments

* * fix(broker/lua): stream connector protobuf parser did not recognize repeated fields

* fix(broker/lua): repeated fields should be ok with streamconnector
* cleanup(broker/lua): deadcode removed
* cleanup(broker/lua): refacto of the code

REFS: MON-15903

* feat(chore): add Deepsource configuration file (#480)

* use RELEASE instead of DISTRIB in package naming (#481)

* MON-16390 no empty output and always fill host_id and service_id (#492)

REFS:MON-16390

* enh(ci): Add rpm delivery cleanup and structure (#499)

* enh(ci): Add rpm delivery cleanup and structure

* Add version to delivery inputs

* Add condition to use sshkey for rpm delivery only

* Update delivery action

* Update action delivery

* Update action delivery

* Update action delivery

* Update action delivery

* Update action delivery

* Enable cleanup

* Update action.yml

* Update action.yml

* Update action.yml

* fix(ci): fix run dir naming on rpm delivery (#503)

* fix(engine/ack): sometimes acknowledgement_type and problem_has_been_… (#495)

REFS: MON-16401

* Mon 15433 convert dimension bv event to protobuf (#442)

* MON-15433 add pb_dimension_bv_event

* MON-15432 ad pb_ba_bv_relation_event

* MON-15432 add test pb_ba_bv_relation_event

* MON-15436 add pb_dimension_timeperiod event

* MON-15430 add pb_dimension_ba_event

* MON-1531 add dimension_kpi_event

* MON-15431 add test and protobuf copy constructor

* MON-15425 add pb_kpi_status

* MON-15425 add test

* MON-15429 add ba_duration_event

* MON-15437 add pb_dimension_ba_timeperiod_relation

* MON-15439 MON-15438 remove dimension_timeperiod_exception and dimension_timeperiod_exclusion

* MON-15434 add pb_dimension_truncate_table

REFS:MON-15433

* fix(broker): ack and stop events well registered with bbdo 3.0.1

* MON-16583 add pb_dimension_truncate_signal case (#514)

REFS:MON-16583

* enh(ci): remove paths to workflow files in workflows for components (#513)

* Mon 16454 add alma 8 to centreon collect nightly tests (#506)

* enh(ci): Add alma8 to centreon-collect nightly

* Add job to robot-nightly workflow

* enable ci test branch

* Update robot nightly workflow

* add debug to get-packages

* Add release to github_env

* add delivery test

* Get pkg dir version name

* Fix url

* fix url

* make curl silent

* fix curl

* add mariadb client

* update get-packages

* disable nightly on alma 8

* update alma8 image

* Add MariaDB packages to alma8 dockerfile

* remove extra sudo

* update collect-test-robot.sh

* Remove duplicate Robot test job

* fix(ci/tests): attempt to upgrade grpc python library

* Install proper grpc version based on OS version id

* help cmake to find protoc

* disable delivery tests

* remove unused lines from actions

* Remove more unused lines

Co-authored-by: David Boucher <dboucher@centreon.com>

* enh(broker/mysql): bulk statements managed by broker

* enh(broker/mysql): first step on columns

* fix(broker/mysql): columns use less emory and are faster

* enh(broker/mysql): columns use a vector to store data

* fix(broker/stmt): better initialization

* fix(broker/mysql): new possibility to prepare statement in bulk

* fix(broker/mysql): work on mysql_bind only is now possible

* fix(broker/unified_sql): application of changes needed by changes in the mysql api

* enh(broker/mysql): improvement of mysql_connection

* fix(broker/stmt): better construction

* enh(broker/unified_sql): bulk statements in place for hosts/services and resources.

* fix(broker/unified_sql): error in a query

* fix(perfdata): error changed into warning for bad metrics

* fix(broker/unified_sql): regressions fixed on rebase

* fix(broker/bam): errors due to rebase fixed

* fix(broker/unified_sql): a log level changed

* fix(tests): they are back

* cleanup(broker/unified_sql): coding style

* fix(broker/conf): init file headers cleaned

* fix(broker/database): new files added

* fix(broker/database): sub-part of the mysql changes

* fix(broker/mysql): main part of the mysql part

* fix(broker/storage): little changes to adapt to the new api

* fix(broker/unified_sql): bulk_bind updated

* fix(deps): golang has a blocking change. So code is updated

* fix(broker/unified_sql): stream updated

* fix(ci): packages artifacts added

* fix(ci): artifacts updated

* fix(ci): using artifact for centreon-tests

* fix(ci): attempt to use the cache for centreon-tests

* fix(ci): wip

* fix(ci): packages artifacts added

* fix(broker/unified_sql): a case of too long string fixed

* fix(broker/mysql): changes after review

* cleanup(broker/mysql_result): comments removed

* fix(broker/mysql_column): _vector_buffer removed because too dangerous for developers

REFS: MON-16231

* fix(broker): floating point numbers and mysql work and an issue with link

REFS: MON-16231

* chore(license): update maintainer and email addresses (#529)

* chore(packaging): package centreon-collect to almalinux 9 (#533)

Refs: MON-16456

* fix(delivery): deliver el9 rpm (#534)

* fix(ci): handle delivery for every enterprise linux (#539)

Refs: MON-16456

* fix(broker): bulk issue (#538)

REFS: MON-16212

* fix(broker/unified_sql): attribute _next_time not defined

REFS: MON-16231

* chore(ci): Update delivery credentials (#543)

* Update delivery credentials

* Update delivery credentials and inputs

* Update deb delivery credentials

* MON-16653 Add extra delivery to artifactory (#528)

* MON-16653 Add extra delivery to artifactory

* add rpm delivery to artifactory

* Update action.yml

* fix(broker): a test too strict (#548)

REFS: MON-16231

* chore(ci): Update registry in workflows (#547)

Refs: MON-16808

* chore(ci): deliver to unstable after nightly robot (#557)

Co-authored-by: David Boucher <dboucher@centreon.com>

* chore(package): remove el7 leftovers (#556)

Refs: MON-16456

* chore(ci): Add extra delivery to artifactory (#554)

* MON-16987 add root bdd access (#565)

REFS:MON-16987

* MON-16995 split external-commands tests (#566)

* MON-16995 split external-commands tests

* MON-16995 fix BEATOI11 (broker core)
REFS:MON-16995

* feat(influxdb-metrics): First release of a Lua connector

* feat(influxdb-neb): New script to send data to influxdb

This script uses neb events and not metrics.

* doc(influxdb): README.md completed to explain influxdn streamconnectors

* Update README.md

fix(doc): precisions on script names.

* doc(elastic): New doc about an elasticsearch stream connector

This version works with metrics events.

* feat(elasticsearch): A new stream connector working with elasticsearch

* doc(elastic): Documentation completed

* Update README.md

* Update README.md

* feat(warp10): New stream connector to export data to warp10

This stream connector will only work with centreon-broker-18.10.1
that contains the new parse_perfdata function.

* doc(README.md): Documentation updated.

* update the doc link

change to lastest instead of 2.8.X

* feat(splunk): New scripts to export to splunk

* fix(splunk): Images names fixed.

* enh(doc): improve splunk part (#2)

* enh(doc): improve splunk part

* fix(influx-neb.lua): parse_perfdata function removed

Thanks to David Guenault, this bug is fixed.
There was previously a bugged parse_perfdata function.
Now we use then centreon-broker one which is much better.

* Add Service Now Stream Connector (#4)

* Add Service Now stream connector

* Update README.md

* feat(README.md) A TOC is added

* fix(README.md): TOC updated

* fix(README.md): Typo

* fix(warp10): Remove space in class name and better class name and labels

* add NDO output (#7)

* add NDO output

* add OMI stream connector

* rename file

* add url configuration and improve readme

* Various improvements to InfluxDB script (#9)

* Support instances in InfluxDB (#10)

Thank you again @cpbn !!

* Properly verify perfdata is present (#11)

* Better handle perfdata err (#12)

* Improve InfluxDB script (#13)

* Properly handle write return value

* Workaround duplicate events

* Add an option to skip anon events

* Improve documentation

* enh(influxdb-neb): add log_path and influxdb_retention_policy parameter, change authen method (#14)

* enh(influxdb-neb): add log_path and influxdb_retention_policy parameter, change authen method

Add a log_path parameter to provide path to log file
Add a influxdb_retention_policy parameter to provide RP when writing to DB
Change the way credentials are provided in request

* enh(doc): add new parameters for influxdb-neb script

* feat(elastic-neb): add elastic-neb.lua script (#15)

* feat(elastic-neb): add elastic-neb.lua script

Handles both metrics and statuses

* feat(doc): add elastic-neb section

* enh(elastic-neb): add min/max/uom to metrics index

* feat(doc): update elastic-neb section

* Update omi_connector.lua (#16)

fix url value in post request

* Escape / replace spec chars (#17)

* Escape / replace spec chars

* Typo

* feat(warp10): add host and srvc groups (#18)

add host and service groups as labels of measures

* InfluxDB enhancements (#21)

* enh(influxdb): Add replacement_character option

* enh(influxdb): Only consider new perfdata

* Ome fix omi doc and minor improvement (#19)

* doc: wrong path for OMI in README.md

* enh: lighter init log (one-liner)

* enh: flush now called when data sizeor age is reached, not exceeded

* feat(pagerduty): adding a connector to PagerDuty based on lua-curl (#22)

* feat(pagerduty): adding a connector to PagerDuty based on lua-curl

* fix(pagerduty): making ifnil* functions local instead of global

* enh(influx) Support perfdate perfdata (#23)

* enh(influx) Support perfdate perfdata

* Typo

* enh(pagerduty): improving various defaut param values + handling metrics (#24)

* enh(pagerduty): Update pager duty (#26)

* enh(pagerduty): improving various defaut param values + handling metrics

* enh(pagerduty): handle the case when the event queue is already full
before appending a new event

* Ome add bsm connector wip (#20)

* doc: wrong path for OMI in README.md

* enh: lighter init log (one-liner)

* enh: flush now called when data sizeor age is reached, not exceeded

* (feat+doc): Add HP BSM connector, forked from OMI

* enh/refacto(bsm): a lot of changes

* use of luacurl
* improving reliability

* fix(merge): restoring OMI doc that was deleted during bad merge in github WUI

* fix(bsm): some issues (typo + special chars handling) (#28)

* doc: wrong path for OMI in README.md

* enh: lighter init log (one-liner)

* enh: flush now called when data sizeor age is reached, not exceeded

* (feat+doc): Add HP BSM connector, forked from OMI

* enh/refacto(bsm): a lot of changes

* use of luacurl
* improving reliability

* fix(merge): restoring OMI doc that was deleted during bad merge in github WUI

* fix(bsm): commented concatenation made the script fail

* fix(bsm): messages were refused when containing non ASCII chars

* fix(README): BSM connector now needs luaxml (#29)

* fix(bsm): typo in content-type header definition (#30)

* fix(influx) Ignore time metric (#34)

* Canopsis (#35)

* add bbdo2canopsis.lua connector

* canopsis fix README

* fix README canopsis install part

* remove no usefull warning README canopsis

Co-authored-by: Paul PRÉMONT <ppremont@¢apensis.fr>

* (enh): Refacto of splunk connector (#27)

* (enh): change host/hostname field

* (fix): change log level to avoid to much log

* (enh): using neb instead of storage + rename the connector for more consistency

* (fix): remove useless lib

* (enh): Change the documentation for Splunk

* (enh): remove mandatory in the commentary

* Update README.md

* Update README.md

* enh(pagerduty): various improvements to PagerDuty connector (#36)

* Allow source field configuration => if user set a value then use this value, else use hostname like it is now
     if pdy_source is configured in the output, then the string is used as the source field. If not, the host name is used.
* Add hostname in the title
     “Hostname/Servicedescription:” added at the beginning of the title (“Summary”)
* Remove metrics (it will avoid the parsing bug also)
     Done.
* Add all possible dimensions in customer details (*Groups / *Categories)
     Categories are not available if they are not severities. Severity numbers have been added.

* Fix hostgroups for services + state change condition (#37)

* add hostgroups for service events and fix state change condition

* fix service extra hostgroups information

* fix(doc) - add prerequisites for OMI

* avoid "attempt to compare number with nil"

* add prometheus gateway stream connector (#41)

* create prom gateway stream connector

* fix hostgroup filter

* enh(influxdb) Add an option to store states (#43)

* Snow rework (#39)

* start snow connector v2

* add useful functions

* starting refacto

* refacto mapping

* refacto event functions

* refacto methods comment

* rework snow provider

* bring back old expiration time

* better default parameters

* ev manage json v2 compatibility + bulk insert

* hide passwords in log file

* add missing token method

* fix variable name

* remove old file

* fix typo

* fix global var

* fix oauth

* add hostgroup filter

* improve log readability

* add missing parameter

* fix hostgroups and wrong var name

* remove debug comment

* simplify introduction

* add opsgenie stream connector (#42)

* create opsgenie stream connector

* add severity to priority conversion

* fix priority + add alerts dedup + fix endpoint

* add bam incidents

* nil managment

* reset log level

* simplify documentation

* Stream connectors v2 (#45)

* Modules Docs + Modules bug fixes + Modules new features  (#46)

* add doc index and logger module documentation

* add sc_common module documentation

* fix wrong link

* add table of content for sc_common doc

* refacto cache structure for the event

* method naming

* fix get_servicegroups method

* fix get_instance method

* fix get_bv_infos method

* add sc_broker module documentation

* fix example in doc

* better sc_logger initialization

* adds sc_param module documentation

* better skip_nil_id handling

* put broker cache info in cache table of the event

* add scope details for params

* add sc_event module documentation

* fix links in documentation

* fix links again

* fix links for is_service_valid

* fix downtime link

* better method naming

* fix wrong method name in doc

* adds the possibility to filter using poller name

* adds severity filter feature

* fix rockspec 0.1.0-1

* add rockspec file for production ready release

* typo

Co-authored-by: pkriko <32265250+pkriko@users.noreply.github.com>

* typo

Co-authored-by: pkriko <32265250+pkriko@users.noreply.github.com>

* typo

Co-authored-by: pkriko <32265250+pkriko@users.noreply.github.com>

* typo

Co-authored-by: pkriko <32265250+pkriko@users.noreply.github.com>

* typo

Co-authored-by: pkriko <32265250+pkriko@users.noreply.github.com>

Co-authored-by: pkriko <32265250+pkriko@users.noreply.github.com>

* fix memory leak + sc naming (#47)

* Add downtime, ack, dedup features (#48)

* add ack event status param

* add new filters for downtime and ack

* start dedup

* add dedup and finish downtime and ack

* add downtime event to the list of filtered event

* fix wrong filter (cat instead of elem)

* fix compil error (missing method in module)

* change return of split method

* add status mapping for downtime event

* add a downtime event test stream connector

* improve downtime event handling +doc

* start a broker data structure doc

* ack broker structure doc +ack event small refacto

* add kafka test script

* kafka param methods + doc

* integrate lua librdkafka into centreon sc modules

* fix method name and regex in sc_params

* rockespecing modules for 1.1.0

* fix missing standard param (#49)

* fix rockspec v1.1.0-2 (#50)

* fix requires in test_kafka script (#51)

* fix rockspec (#52)

* fix requires in test_kafka script

* fix rockspec again

* prepare  1.2 release add new stream connectors

* fix requires in test_kafka script

* fix rockspec again

* put mandatory param in std param list

* organize spec files

* new method for url param

* fix spec file 1.1.0-2

* conflicting

* solving conflict

* add a new loglevel (info)

* smarter url encoding

* fix mandatory params

* start bq_table module

* start bigquery module

* create google oauth module

* start bigquery stream connector

* create a macro module

* add time related params

* change severity in cache to match macro system

* prepare spec file for 1.2 release

* add time params in the doc

* force broker api v2

* document google oauth module

* improve bq_tables

* minor refacto on new modules

* improve google modules

* better cache and skip_anon + small fixes

* add bigquery module documentation

* add logging in sc_macros module

* remove unnecessary require in sc_macros module

* enable event dedup by default

* add sc_macros documentation

* add new broker event data structures

* better method naming in sc_macros module

* index new documentations in readme

* git conflicting

* add service now api v2

* bigquery stream connector

* remove sleep in stream connectors

* Update centreon-stream-connectors-lib-1.2.0-1.rockspec

* patch for el8 compatibility (#55)

* fix requires in test_kafka script

* fix rockspec again

* prepare el8 compatibility

* el8 compat patch

* fix(influxdb) Properly replace backslashes (#54)

* fix(influxdb) events may not have last_check value (#56)

* Splunk rework + prepare new release (#57)

* add splunk event apiv2 SC

* better cat and el handling

* add new flush module

* add template handling

* add load json file method

* add template file param + fix new elems

* fix wrong log message

* fix wrong send_data method naming

* add new sc_metrics module

* add default params value

* use is_valid_perfdata method where needed

* add new splunk metrics stream connector

* load needed modules

* improve logging and comments in metrics module

* add logging and comments for flush module

* add a test option

* comment code in sc_common module

* document new common method

* refacto flush module and cata nd el mapping

* refacto cat + el mapping

* add new methods in sc_param documentation

* fix cat and el in modules

* add documentation of the sc_flush module

* copy pasta errors in sc_event doc

* add documentation for sc_metrics module

* update readme index

* add a documentation about mappings

* add templating documentation

* fix param doc link

* better stream connector naming

* fix thousand bugs

* fix macro bugs

* add rockspec files for 1.3.0 release (#58)

* fix wrong logging default configuration (#59)

* typo

* improve kafka and doc readme (#60)

* fix requires in test_kafka script

* fix rockspec again

* update kafka and fix readme

* enh(influxdb) get rid of deduplicate (#61)

* Fix splunk (#62)

* add rockspec files for 1.3.0 release

* fix options in stream connector

* add number to boolean method

* handle basic connection parameters

* add rockspec file for 1.4.0 release

* Forgot doc (#63)

* doc for allow_insecure_connection param

* Fix splunk metrics (#64)

* fix metric handling for splunk

* Streamline kafka (#65)

* add a more generic kafka event format

* add default values for el and cat params snow stream connectors

* fix require for kafka (#66)

* add rockspec file for 1.4.1-1 (#67)

* add rockspec file for release

* Fix bigquery (#68)

* fix sc_macro object creation + remove hardcoded id

* fix log message by removing var name conflict

* add rockspec for 1.4.2-1 (#69)

* add rockspec file for 1.4.2-1

* fix Cache and flag and json (#70)

* fix transformation flags not working

* fix cache when setting skip_annon_events to 0

* better naming in sc_common

* Fix kafka format event (#71)

* avoid function naming conflict

* Refacto pagerduty (#73)

* fix proxy param (#74)

* fix proxy param

* add rockspec 1.5.1

* Buffer issue (#75)

* fix bulk events

* add rockspec for 1.5.2

* bad log messages and typos (#76)

* bad log messages and typos

* fix wrong severity mapping pagerduty

* remove event bulking pagerduty

* update snow sc with latest features

* prevent bad macro replacement when % in string

* update snow

* handle state conversion flag for ack events

* fix bad log message for log params

* add log_level corresponding table in doc

* add rockspec for version 1.5.3

* fix splunk json (#79)

avoid having special characters in field names and fix json structure

* fix Splunk metric event

We have to remove the line 'event = "metric_event"' in order to write an index of type "Mesure"

* fix wrong variable name

* enh(doc) update links (#81)

update Documentation URL's

* enh(elastic) add events apiv2  (#72)

* (stream) signl4-sc initial release (#84)

* (stream)first-commit-signl4-connector

* (stream)fix-example-signl4-connector

* + review fix and enhancements

* enh: replace broker_log by wrapper

* Training course (#88)

* Training course (#89)

* (stream)signl4 bugfix and json simplification (#87)

* (stream)signl4 bugfix and json simplification

* - info instead of debug

* (lib)sc_macro fix buggy transform_date function (#86)

* Traning course plus (#90)

* Update centreon_classroom.lua

* Update exercises.md

* exercise 3 answer

* exercises 4 and 5 answers

* exercise 4 and 5 answers

* Update exercices_answers.md

* add summary

* Update exercices_answers.md

* + add Signl4 doc link

* Refacto streamconnector BSM v2

* Review code after firts recommandations

* Change name file

* Review function write

* Refacto event treatment for when send_data_test=1

* Review code with of formated data

* delete comment block

* refacto to v3

* Create function xml_escape for bsm-event SC

* Delete file because already in others PR

* Apply suggestions from code review

Co-authored-by: tcharles <tcharles@centreon.com>

* Create sc_common.lua

* Last review

* Add spec 1 5 4 (#92)

* refacto hp omi to apiv2

* Delete bsm-events-apiv2.lua

* Update omi_events-apiv2.lua

* Delete max_buffer_size params because not correctly event format

* Enhancement event format to multi event

* (stream/lib): sc_flush - refacto queue system and compatibility patch for apiv2 stream connectors (#93)

* fix requires in test_kafka script

* fix rockspec again

* update kafka and fix readme

* typos

* (stream/lib) sc_flush: new queuing system

* (stream/connectors) compat patch for queue system

* (stream/documentation) add doc for new params

* (stream/lib): sc_flush - fix bad param name

* shaping flush methods

* naming + missing parameter

* better useless event management with flush

* avoid calling same method twice

* update all apiv2 sc

* add xml_escape method

* add rockspec file

* add doc for xml escape method

* prettier markdown

* fix host_status event dedup

* enhencament format event for event in payload

* (stream/lib) sc_event: add new event outputs options (#97)

* add new output management options

* better pagerduty event format

* (streams/lib): add rockspec 2.1.0-1 (#102)

* add rockspec 2.1.0

* forgot require sc_flush lib

* (stream/events) missing sc_flush new instance

* (stream/event): fix add queue method (#106)

this was still the old add() method so it wasn't adding the event to the right queue and nothing was flushed at all

* (stream/events): add Datadog sc (#101)

* fix requires in test_kafka script

* fix rockspec again

* update kafka and fix readme

* typos

* (stream/lib) sc_flush: new queuing system

* (stream/connectors) compat patch for queue system

* (stream/documentation) add doc for new params

* (stream/lib): sc_flush - fix bad param name

* shaping flush methods

* naming + missing parameter

* better useless event management with flush

* avoid calling same method twice

* update all apiv2 sc

* add xml_escape method

* add rockspec file

* add doc for xml escape method

* prettier markdown

* fix host_status event dedup

* add new output management options

* better pagerduty event format

* (stream/events): add datadog sc + spec release 2.1

* delete rock spec

* + replace success by info

Co-authored-by: Simon Bomm <sbomm@centreon.com>

* (stream/lib) Metric v2  (#103)

* fix requires in test_kafka script

* fix rockspec again

* update kafka and fix readme

* typos

* (stream/lib) sc_flush: new queuing system

* (stream/connectors) compat patch for queue system

* (stream/documentation) add doc for new params

* (stream/lib): sc_flush - fix bad param name

* shaping flush methods

* naming + missing parameter

* better useless event management with flush

* avoid calling same method twice

* update all apiv2 sc

* add xml_escape method

* add rockspec file

* add doc for xml escape method

* prettier markdown

* fix host_status event dedup

* add new output management options

* better pagerduty event format

* add a dumper method

* add some metric stuff

* metrics first tests

* fixes datadog metrics bugs

* fix log message sc_event

* add new params doc + fix md

* improve doc + more doc

* simplify sc_metrics

* add new standard parameters

* better dumper display

* new snow stream connector

* improve snow sc

* metric v2 splunk metrics

* debug + params for datadog metrics

* finishing metrics sc

* forgot replacement character

* "better" splunk metrics event format

* revert change

* add default buffer size

* fix datadog metric name regex

* add specfile (#109)

* (stream/lib) sc_param bbdo3 and bddo2 compat (#108)

* manage bbdo3 and bddo2 compat

* add comment

* streamline compat idea

* fix missing state in new bbdo elements

* force var type for bbdo_version

* fix downtime state bbdo3

* compat patch for ack

* fix bad cache handling

* fix bad boolean for ack

* add spec file (#110)

* add spec file

* fix bad spec file name

* (stream/events) add logstash-events stream connector (#111)

* add logstash-events stream connector

* fix communication protocol

* fix wrong event format method (#114)

* (stream/lib) add output limit (#112)

* (stream/lib) add output limit

* fixing my dumb mistakes

* fix bad variable scope

* handle last_check & last_update for better compat (#116)

* add 3.2.0-1 spec file (#117)

* Update datadog-events-apiv2.lua

* Enhance macro system (new group macro + new transformation flags) (#118)

* add new escape method + refacto other ones

* always put cache data in event + fix poller cache

* add group macro handling (hg, sg, bv)

* add new transformation flags

* add doc for lua_regex_escape method

* add new macros documentation

* put new methods into readme

* fix wrong link

* add trim function

* fix many bugs with macro conversion

* add trim documentation

* add doc for new macros method

* add new methods in the readme

* remove testing debug

* add custom code feature for stream connectors (#119)

* add new custom_code_file param handling

* add custom code execution

* load custom code in each api v2 SC

* put custom_code in the params table

* fix bad  logger instance name

* Add custom code documentation

* avoid breaking change

* add new param in the doc

* imrpove doc

* wording

* better code block

* Apply suggestions from code review

* typos

Co-authored-by: Simon Bomm <sbomm@centreon.com>

* add specfile for 3.3.0 release (#120)

* add canopsis stream certified stream connector + queue system enhancement (#122)

* prepare canopsis stream connector

* add queue metadatas

* start writing http query

* fix log message in datadog sc

* finish canopsis stream connector

* update flush class with metadata

* add specfile 3.4.0-1 (#123)

* fix ack remove and dt cache and more sc options (#124)

* fix ack remove dt remove and dt send (#126)

* fix ack remove dt remove and dt send

* force buffer size to 1

* Flapping and canopsis4 (#127)

* handle flapping

* update event doc with flapping

* add flapping parameter

* add documentation for flapping param

* add new methods in the readme

* rename canopsis and handle duplicate dt edge case

* add canopsis api v4 sc

* improve queue metadata system

* fix queue metadata system

* use authkey instead of basic auth

* update the flush class documentation

* add queue metadata system to all apiv2 sc

* improve logs

* fix markdown

* add add_queue_metadata method documentation

* put all sc_flush method in the readme index

* fix code comment

* add spec file for 3.5.0

* fix broken long output with line break (#128)

* fix wrong mandatory param and wrong dt index (#129)

* add a new method to create shell curl commands (#130)

* add get_curl_command method

* add proxy_protocol param

* move get_curl_command from sc_common to sc_logger

* add log_curl_commands param

* document new parameters

* add log_curl_command documentation

* encapsulate strings in log_curl_command

* add new log_curl_command method in the sc

* fix bugs with log curl command

* log curl commands for all possible sc

* fix log message

* fix bad proxy verif

* add 3.6.0 specifle

* fix wrong method name (#131)

* add instance and service as grouping key (#133)

* add instance and service as grouping key

* not working without the host label

---------

Co-authored-by: Thomas Vasseur <t.vasseur@axione.fr>

* enh(ci): avoid to always trigger collect workflow

workflow

workflow

workflow

workflow

workflow

workflow

workflow

workflow

workflow

workflow

workflow

workflow

workflow

workflow

test

test

test

test

test

test

test

test

test

test

test

test

test

test

test

test

test

test

test

* MON-16987 add root bdd access (#565)

REFS:MON-16987

* MON-16995 split external-commands tests (#566)

* MON-16995 split external-commands tests

* MON-16995 fix BEATOI11 (broker core)
REFS:MON-16995

* fix stream connectors packaging

test

test

test

* fix(ci): avoid deliver on dev branches

* fix(ci): fix get version of canary packages

* fix(ci): update deb delivery inputs for nexus

Co-authored-by: tuntoja <58987095+tuntoja@users.noreply.github.com>

* fic(ci): remove anchors in centreon-collect workflow (#569)

* MON-17012 metrics are copied in _metrics, tests/init_sql.sh can create another database than centreon_storage (#568)

REFS:MON17012

* chore(ci): preserve artifacts 1 day (#577)

Refs: MON-17125

* MON-17037 fix ad and bbdo-protobuf (#575)

REFS:MON-17037

* chore(ci): move to harbor docker registry (#580)

* chore(ci): package stream connectors (#570)

Refs: MON-16420

* fix(stream-connectors): bulk event and improve canopsis4 (#584)

Co-authored-by: tcharles <tcharles@centreon.com>

* chore(ci): remove stream-collectors from collect repository (#593)

Refs: MON-16173

* MON-16987 fix timestamp floor issue (#594)

REFS:MON-16987

* Mon 17037 ad test fix (#595)

* MON-17037 more downtime timeout

* MON-17037 remove centengine.cmd before centengine start
REFS:MON-17037

* chore(license): update package licenses (#558)

* enh(ci): handle debian delivery failure (#598)

* Mon 17346 trace coredump (#596)

* MON-17037 more downtime timeout

* MON-17037 remove centengine.cmd before centengine start

* MON-17346 unified-ssql trace fix

* fix(engine/config): tag and severity had several issues

* enh(conan): grpc upgraded
* fix(cmake): we force conan version to avoid issues with version 2

REFS: MON-17262

* chore(ci): publish to new repositories (#607)

Refs: MON-16173

* Mon 17390 scheduler test (#609)

* fix(engine/config): tag and severity had several issues

* enh(conan): grpc upgraded
* fix(cmake): we force conan version to avoid issues with version 2

* MON-17390 fix scheduler and connector_ssh tests

* MON-17390 fix perl-connectors test

* MON-17390 fix engine/stat-stop

* MON-17390 engine dont core if cbmod fail start, add sql traces

* MON-17390 add periodical commit

* MON-17390 review remarks

* MON-17390 less lock

REFS:MON-17390

Co-authored-by: David Boucher <dboucher@centreon.com>

* ci(sync-branches): use technique-ci PAT (#613)

* fix(collect): develop backport of 22.04 patches

* Mon 16333 add periodic log flush  develop (#610)
* Mon 17390 scheduler test (#609)
* fix(engine/config): tag and severity had several issues
* enh(conan): grpc upgraded
* fix(cmake): we force conan version to avoid issues with version 2
* MON-17390 fix scheduler and connector_ssh tests
* MON-17390 fix perl-connectors test
* MON-17390 fix engine/stat-stop
* MON-17390 engine dont core if cbmod fail start, add sql traces
* MON-17390 add periodical commit
* MON-17390 review remarks
* MON-17390 less lock
* MON-16333 develop backport
* MON-16333 backport periodic flush
* MON-16333 an asio timer flushes logs
* MON-16333 tcp_async is now a sahred_from_this
* MON-16333 add somes modifications, a logger non configured can be activated with ccc
* MON-16333 develop add SetLogLevel
* MON-16333 issues after rebase
* MON-16333-add-SetLogFlushPeriod
* MON-16333-force push
* MON-16333 review remarks
* fix(broker/mysql): new tests added and a little fix in mysql on stats (#616)
* fix(broker/unified_sql): A bad query is fixed here (#617)
* fix(broker): compilation warnings and errors fixed for some compilers
* MON-17457-improve-test-coverage-develop (#619)
* MON-17458-develop-improve-monitoring-stream-ack (#620)
* enh(collect): log_v2 is removed from main static libraries
* fix(collect/log_v2): uniformization of log_v2 between engine and broker
* fix(broker): unit tests on rrd cached + deadlock in tcp_async
* MON-17461 develop update doc (#622)
* fix(broker/mysql): Bind for results was missing
* fix(broker/sql): issue in bulkd_queries fixed and new test
* cleanup(broker/mysql): unused functions removed
* fix(broker): little fixes due cherry-picks
* MON-17465 fix uncached exceptions (#624)
* fix(broker/splitter): splitter better protected during read/write
* MON-16986 splitter open files in binary mode and write bug fix
* wip(broker/splitter)
* fix(broker/splitter): splitter improved with new tests
* cleanup(broker/splitter): comments added.
* cleanup(broker/compression)
* fix(broker/tests): bad comment fixed
* enh(broker/compression): optimization
* fix(broker/mysql): An assert was there by mistake
* fix(broker/splitter): fwrite error management better
* cleanup(broker/splitter): log updated
* fix(broker/splitter): log fixed
* fix(broker/splitter): no exception when limit size reached
* fix(broker/unified_sql): bulk_bind row filling must be protected (#615)
* wip(broker/splitter)
* fix(broker/splitter): splitter improved with new tests
* cleanup(broker/config): logs uniformized
* fix(broker/muxer): reuse muxer in case of reload
* cleanup(broker/muxer): comments fixed
* cleanup(broker/muxer): comments updated
* enh(broker/engine): little optimization

REFS: MON-17453 MON-17454 MON-17455 MON-17456 MON-17457 MON-17458 MON-17459 MON-17460 MON-17461 MON-17462 MON-17463 MON-17464 MON-17465 MON-17466 MON-17467 MON-17468 MON-17897

Co-authored-by: Jean Christophe Roques <jcroques@centreon.com>
Co-authored-by: David Boucher <dboucher@centreon.com>

* MON-17452 multiplexing sources are compiled in multiplexing static library (#632)

REFS:MON-17452

* Mon 17390 connector ssh test fail (#639)

* MON-17390 add sshd traces

* MON-17390 add openssh-server in docker files
REFS:MON-17390

* fix(delivery): restore release with timestamp/hash on unstable (#646)

* fix(tests): ccc tests work (#650)

REFS: MON-17452

* fix(tests): BESSBQ1 failed because of a change in broker logs (#653)

REFS: MON-17452

* MON-17452-develop-no-fatal-request (#652)

REFS:MON-17452

* fix(tests/mysql): issues with mysql connection disconnect/reconnect fixed

* fix(broker/sql): issue with misconfigurations

* fix(broker/mysql_connection): commit could make an infinite loop

* cleanup(broker/mysql): comments added

* fix(broker/mysql): sync queries updates the same way as commit

REFS: MON-17452

* MON-17452 fix mysql_ping (#661)

REFS:MON-17452

* fix(broker/grpc): issue in test because of change in the api

* fix(broker): issues with log grpc api

* cleanup(tests): comment removed

REFS: MON-17452

* MON-17452 fix filters.robot bam*.robot grpc-stream.robot (#666)

REFS:MON-17452

* enh(engine/tests): LogLevel message has a logger instead of a name

* fix(broker): issues with log grpc api

* cleanup(tests): comment removed

REFS: MON-17452

* chore(debian): manage version threshold (#669)

* fix(broker): protobuf events not sent when empty

REFS: MON-17452

* MON-17452 wait for mariadb to terminate (#670)

* MON-17452 wait for mariadb to terminate

* MON-17452 fix output-table, log-v2_engine

* MON-17452 fix bbdo-server-client and bbdo-server-client-reversed

* MON-17452 fix severities

* fix(tests): rrd cbd is started but not configured (#673)

* fix(tests): rrd cbd is started but not configured
* fix(tests/sql): issue with dates
* fix(tests/rrd): attempt to set floor(date) for start time
* fix(tests): LOGV2BEU2 removed because new logs have been added only in log_v2 and this breaks that test

REFS: MON-17452

* MON-16990 add 2 tests (#675)

REFS:MON-16990

* MON-17452 tags fix (#677)

REFS:MON-17452

* fix(tests/rrd): order of ids may be different in tests and logs (#678)

* fix(tests/rrd): order of ids may be different in tests and logs

* cleanup(tests/rrd): comment removed

* fix(tests/rrd): bad index fixed

* fix(tests/rrd): index missing

* enh(tests): new tests on bulk get from 22.04

* fix(tests): to update db_variable automatically for my needs

* fix(tests): wip

* fix(robot): db_variables

* fix(tests/retention): lookup up in the wrong file

* fix(tests): bulk tests timeout increased

* fix(tests): start date floor round

REFS: MON-17452

* fix(tests): dates round to the floor in server-client-reversed (#680)

REFS: MON-17452

* fix(tests): bam_pb did not wait sufficiently to send external commands (#681)

* enh(packaging): add commit hash (#682)

* Add commit_sha to nightly robot workflow

* fix(ci): Add fail to create jira version curl 2304

* Mon 18167 add pb status pb metric to graphite release (#685)

* MON-18167 add pb_metric and pb_status to graphite

* MON-18167 add pb_metric and pb_status to influxdb

* MON-16167 only one influxdb version
REFS:MON-18167

* fix(docker): fix docker builder context (#692)

* chore(ci):update tag workflow

* chore(ci): publish releases on download.centreon.com (#699)

Refs: MON-15793

* enh(ci): update debian delivery (#705)

* enh(ci): update debian delivery

* Update action.yml

add missing trailing slash

* fixdelivery path for debian

* Update .github/workflows/get-version.yml

* MON-17253 value of custom variable are filled (#716)

REFS:MON-17253

---------

Co-authored-by: jean-christophe81 <98889244+jean-christophe81@users.noreply.github.com>
Co-authored-by: tuntoja <58987095+tuntoja@users.noreply.github.com>
Co-authored-by: Kevin Duret <kduret@centreon.com>
Co-authored-by: David Boucher <dboucher@centreon.com>
Co-authored-by: Stéphane Chapron <34628915+sc979@users.noreply.github.com>
Co-authored-by: quanghungb <quanghungb@users.noreply.github.com>
Co-authored-by: Etienne G <Moujimouja@users.noreply.github.com>
Co-authored-by: Maximilien Bersoult <mbersoult@centreon.com>
Co-authored-by: benrobert <brobert@centreon.com>
Co-authored-by: CPbN <40244829+CPbN@users.noreply.github.com>
Co-authored-by: Colin Gagnaire <cgagnaire@centreon.com>
Co-authored-by: SAD <zoulou31@free.fr>
Co-authored-by: Thomas Arnaud <38663853+Nohzoh@users.noreply.github.com>
Co-authored-by: UrBnW <40244829+UrBnW@users.noreply.github.com>
Co-authored-by: omercier <32134301+omercier@users.noreply.github.com>
Co-authored-by: Polo <paul@premont.fr>
Co-authored-by: Paul PRÉMONT <ppremont@¢apensis.fr>
Co-authored-by: pkriko <32265250+pkriko@users.noreply.github.com>
Co-authored-by: ppremont-capensis <72454406+ppremont-capensis@users.noreply.github.com>
Co-authored-by: Stéphane Duret <sduret@centreon.com>
Co-authored-by: tcharles <tcharles@centreon.com>
Co-authored-by: ponchoh <hmorales@centreon.com>
Co-authored-by: Simon Bomm <sbomm@centreon.com>
Co-authored-by: psame <44295022+psamecentreon@users.noreply.github.com>
Co-authored-by: psamecentreon <psame@centreon.com>
Co-authored-by: tcharles <tanguyvda@hotmail.fr>
Co-authored-by: matoy <matoy@users.noreply.github.com>
Co-authored-by: xenofree <35098906+xenofree@users.noreply.github.com>
Co-authored-by: Thomas Vasseur <t.vasseur@axione.fr>
Co-authored-by: Jean Christophe Roques <jcroques@centreon.com>
Co-authored-by: tuntoja <tuntoja@centreon.com>
Co-authored-by: chgautier <cgautier@centreon.com>
kduret pushed a commit that referenced this pull request Jun 24, 2024
* wip

* enh(sshclient): processcopy action working
kduret pushed a commit that referenced this pull request Jul 10, 2024
* wip

* enh(sshclient): processcopy action working
kduret pushed a commit that referenced this pull request Jul 10, 2024
* wip

* enh(sshclient): processcopy action working
kduret pushed a commit that referenced this pull request Jul 11, 2024
* wip

* enh(sshclient): processcopy action working
Evan-Adam pushed a commit that referenced this pull request Jul 16, 2024
* wip

* enh(sshclient): processcopy action working
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants