ydb-configure missing features, part 3 #15370

Jorres · 2025-03-05T15:15:28Z

Changelog entry

supported 7 config.yaml style options in ydb_configure (e.g. audit_config, see PR for details)

Changelog category

Bugfix

Description for reviewers

storage_config_generation option is now transferred from template to resulting config
audit_config is supported now, we can write config.yaml style audit_config directly in template:

audit_config:
  stderr_backend:
    format: JSON_LOG_COMPATIBLE
    log_json_envelope: '...'

blob_storage_config's disks now contains pdisk_config section if it is specified in host_configs

blob_storage_config:
  service_set:
    groups:
      - erasure_species: 9
        ...
        rings:
          - fail_domains:
              - vdisk_locations:
                  - node_id: 1
                    ...
                    pdisk_config:          <<<<<<<<<<<<<
                      expected_slot_count: 20         <<<<<<<

actor_system_config field is supported instead of sys:

sys: # the old way
  cpu_count: 16
  node_type: STORAGE
  use_auto_config: true

actor_system_config: # config.yaml way
  cpu_count: 40
  node_type: STORAGE
  use_auto_config: true

entry in log_config is now properly base64-encoded and decoded (it's underlying proto type is bytes):

log_config:
  format: json
  default_level: 5
  entry:
    - component: BS_LOAD_TEST <<<<<<<
      level: 3
    - component: BS_HULLRECS <<<<<<<
      level: 2

Multiple nodes with the same FQDN but different ports are now supported in hosts:

hosts:
  - host: node-3235.common-prod.ik8s.man.nbhost.net
    host_config_id: 1
    location:
      body: 1230
      data_center: man
      rack: Data-EoR-01
    node_id: 1
    port: 19001
    ...
  - host: node-3235.common-prod.ik8s.man.nbhost.net
    host_config_id: 2
    location:
      body: 1230
      data_center: man
      rack: Data-EoR-01
    node_id: 17
    port: 19002

github-actions · 2025-03-05T15:16:02Z

⚪ 2025-03-05 15:16:02 UTC Pre-commit check linux-x86_64-relwithdebinfo for 5d47950 has started.
⚪ 2025-03-05 15:16:25 UTC Artifacts will be uploaded here
⚪ 2025-03-05 15:19:14 UTC ya make is running...
⚫ 2025-03-05 15:41:24 UTC Check cancelled

github-actions · 2025-03-05T15:16:40Z

🔴 2025-03-05 15:40:50 UTC The validation of the Pull Request description has failed. Please update the description.

Bugfix requires a linked issue in the changelog entry

github-actions · 2025-03-05T15:17:14Z

⚪ 2025-03-05 15:17:13 UTC Pre-commit check linux-x86_64-release-asan for 5d47950 has started.
⚪ 2025-03-05 15:17:27 UTC Artifacts will be uploaded here
⚪ 2025-03-05 15:19:49 UTC ya make is running...
⚫ 2025-03-05 15:41:24 UTC Check cancelled

github-actions · 2025-03-05T15:42:02Z

⚪ 2025-03-05 15:42:01 UTC Pre-commit check linux-x86_64-relwithdebinfo for ee8d864 has started.
⚪ 2025-03-05 15:42:16 UTC Artifacts will be uploaded here
⚪ 2025-03-05 15:45:09 UTC ya make is running...
⚫ 2025-03-05 15:52:36 UTC Check cancelled

github-actions · 2025-03-05T15:42:56Z

⚪ 2025-03-05 15:42:55 UTC Pre-commit check linux-x86_64-release-asan for ee8d864 has started.
⚪ 2025-03-05 15:43:10 UTC Artifacts will be uploaded here
⚪ 2025-03-05 15:45:33 UTC ya make is running...
⚫ 2025-03-05 15:52:36 UTC Check cancelled

github-actions · 2025-03-05T15:58:51Z

⚪ 2025-03-05 15:58:50 UTC Pre-commit check linux-x86_64-release-asan for 603978f has started.
⚪ 2025-03-05 15:59:06 UTC Artifacts will be uploaded here
⚪ 2025-03-05 16:01:36 UTC ya make is running...
⚫ 2025-03-05 16:02:35 UTC Check cancelled

github-actions · 2025-03-05T15:59:37Z

⚪ 2025-03-05 15:59:36 UTC Pre-commit check linux-x86_64-relwithdebinfo for 603978f has started.
⚪ 2025-03-05 15:59:51 UTC Artifacts will be uploaded here
⚫ 2025-03-05 16:02:23 UTC Check cancelled

github-actions · 2025-03-05T16:04:07Z

⚪ 2025-03-05 16:04:06 UTC Pre-commit check linux-x86_64-relwithdebinfo for 7fe1ee5 has started.
⚪ 2025-03-05 16:04:22 UTC Artifacts will be uploaded here
⚪ 2025-03-05 16:06:49 UTC ya make is running...
🟡 2025-03-05 16:33:18 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
4810	4683	0	2	40	85

⚪ 2025-03-05 16:33:57 UTC ya make is running... (failed tests rerun, try 2)
🟡 2025-03-05 16:38:08 UTC ydbd size 2.1 GiB changed* by +331.8 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: `d128660`	merge: `7fe1ee5`	diff	diff %
ydbd size	2 293 723 288 Bytes	2 294 063 048 Bytes	+331.8 KiB	+0.015%
ydbd stripped size	480 346 144 Bytes	480 394 752 Bytes	+47.5 KiB	+0.010%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}
⚫ 2025-03-05 16:38:10 UTC Check cancelled

github-actions · 2025-03-05T16:05:23Z

⚪ 2025-03-05 16:05:23 UTC Pre-commit check linux-x86_64-release-asan for 7fe1ee5 has started.
⚪ 2025-03-05 16:05:39 UTC Artifacts will be uploaded here
⚪ 2025-03-05 16:07:59 UTC ya make is running...
🟡 2025-03-05 16:33:49 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
1735	1599	0	80	46	10

⚪ 2025-03-05 16:34:06 UTC ya make is running... (failed tests rerun, try 2)
🟡 2025-03-05 16:38:15 UTC ydbd size 3.7 GiB changed* by +637.3 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: `d128660`	merge: `7fe1ee5`	diff	diff %
ydbd size	3 996 519 504 Bytes	3 997 172 056 Bytes	+637.3 KiB	+0.016%
ydbd stripped size	1 390 121 896 Bytes	1 390 280 936 Bytes	+155.3 KiB	+0.011%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}
⚫ 2025-03-05 16:38:17 UTC Check cancelled

github-actions · 2025-03-05T16:40:29Z

⚪ 2025-03-05 16:40:28 UTC Pre-commit check linux-x86_64-relwithdebinfo for 6a8efc8 has started.
⚪ 2025-03-05 16:40:43 UTC Artifacts will be uploaded here
⚪ 2025-03-05 16:43:04 UTC ya make is running...
🟡 2025-03-05 17:10:18 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
4810	4676	0	1	48	85

⚪ 2025-03-05 17:10:54 UTC ya make is running... (failed tests rerun, try 2)
🟢 2025-03-05 17:15:37 UTC Tests successful.

Test history | Ya make output | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
112 (only retried tests)	38	0	0	0	74

🟢 2025-03-05 17:15:44 UTC Build successful.
🟢 2025-03-05 17:15:57 UTC ydbd size 2.1 GiB changed* by +8.2 KiB, which is < 100.0 KiB vs main: OK

ydbd size dash	main: `afb7274`	merge: `6a8efc8`	diff	diff %
ydbd size	2 294 063 048 Bytes	2 294 071 472 Bytes	+8.2 KiB	+0.000%
ydbd stripped size	480 394 752 Bytes	480 396 288 Bytes	+1.5 KiB	+0.000%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2025-03-05T16:40:48Z

⚪ 2025-03-05 16:40:47 UTC Pre-commit check linux-x86_64-release-asan for 6a8efc8 has started.
⚪ 2025-03-05 16:41:02 UTC Artifacts will be uploaded here
⚪ 2025-03-05 16:43:28 UTC ya make is running...
🟡 2025-03-05 17:08:35 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet Going to retry failed tests...

Test history | Ya make output | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
1735	1599	0	80	45	11

⚪ 2025-03-05 17:09:01 UTC ya make is running... (failed tests rerun, try 2)
🟡 2025-03-05 17:20:35 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet Going to retry failed tests...

Test history | Ya make output | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
148 (only retried tests)	31	0	53	55	9

⚪ 2025-03-05 17:20:44 UTC ya make is running... (failed tests rerun, try 3)
🟡 2025-03-05 17:31:56 UTC Some tests failed, follow the links below. This fail is not in blocking policy yet

Test history | Ya make output | Test bloat | Test bloat | Test bloat

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
124 (only retried tests)	42	0	62	12	8

🟢 2025-03-05 17:32:07 UTC Build successful.
🟡 2025-03-05 17:32:27 UTC ydbd size 3.7 GiB changed* by +392.4 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: `7a630cf`	merge: `6a8efc8`	diff	diff %
ydbd size	3 996 782 216 Bytes	3 997 184 040 Bytes	+392.4 KiB	+0.010%
ydbd stripped size	1 390 173 096 Bytes	1 390 283 944 Bytes	+108.2 KiB	+0.008%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

bma13 · 2025-03-05T17:04:43Z

looks good. I have successfully tested

Jorres added 6 commits March 5, 2025 11:48

wip

5d7515a

fix: node port validation and log_config.entry

6e072da

add audit config

931a972

fix: support multiple nodes with same fqdn

afb9818

fix: support actor_system_config

005cc82

fix: support pdisk_config in blob_storage_config

e9f009d

github-actions bot added the new-feature label Mar 5, 2025

github-actions bot added bugfix and removed new-feature bugfix labels Mar 5, 2025

fix: linter

409643c

Jorres requested review from bma13 and sourcecd March 5, 2025 15:42

feat: features -> feature_flags

3d62698

fix: duplicated nameservice_config.accept_uuids

4958481

fix: delete port:2135 properly

40a6d74

sourcecd approved these changes Mar 5, 2025

View reviewed changes

Jorres merged commit 2ef1fc8 into ydb-platform:main Mar 6, 2025
12 checks passed

GrigoriyPA mentioned this pull request Mar 6, 2025

YQ added YDB_AUTH_TICKET_HEADER for fq proxy ydb requests #15416

Merged

This was referenced Mar 10, 2025

Fix segfault in Group Layout Sanitizer, add stress tests with INACTIVE/FAULTY disks #15516

Merged

Add compaction selected strategy counters #15689

Merged

alexnick88 mentioned this pull request Mar 19, 2025

provide database in schemecache request #15938

Closed

blinkov pushed a commit that referenced this pull request Mar 21, 2025

ydb-configure, missing features, part 3 (#15370)

6455838

liruoko added the changelog/f25-3 PR участвует в списке изменений label Nov 19, 2025

ydb-configure missing features, part 3 #15370

ydb-configure missing features, part 3 #15370

Uh oh!

Conversation

Jorres commented Mar 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changelog entry

Changelog category

Description for reviewers

Uh oh!

github-actions bot commented Mar 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Mar 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Mar 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Mar 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Mar 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Mar 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Mar 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Mar 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Mar 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Mar 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Mar 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bma13 commented Mar 5, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Jorres commented Mar 5, 2025 •

edited

Loading

github-actions bot commented Mar 5, 2025 •

edited

Loading

github-actions bot commented Mar 5, 2025 •

edited

Loading

github-actions bot commented Mar 5, 2025 •

edited

Loading

github-actions bot commented Mar 5, 2025 •

edited

Loading

github-actions bot commented Mar 5, 2025 •

edited

Loading

github-actions bot commented Mar 5, 2025 •

edited

Loading

github-actions bot commented Mar 5, 2025 •

edited

Loading

github-actions bot commented Mar 5, 2025 •

edited

Loading

github-actions bot commented Mar 5, 2025 •

edited

Loading

github-actions bot commented Mar 5, 2025 •

edited

Loading

github-actions bot commented Mar 5, 2025 •

edited

Loading