auditlog: add exports/imports #8550

ijon · 2024-08-30T15:39:12Z

Add audit logging for (database/tables) export and import operations.

Cancel/Forget requests for export/imports are still not audit logged.

KIKIMR-21797

Changelog entry

Add audit logging for (database/tables) export and import operations.

Changelog category

Improvement

Add audit logging for (database/tables) export and import operations. Export/import cancel/forget requests are still not logged. KIKIMR-21797

ijon · 2024-09-03T08:12:08Z

ydb/core/tx/schemeshard/schemeshard_build_index__create.cpp

-    static TString GetUid(const google::protobuf::Map<TString, TString>& labels) {
-        auto it = labels.find("uid");
-        if (it == labels.end()) {
-            return TString();
        }
-
-        return it->second;
+        return true;
    }


GetUid has moved to schemeshard_xxport__helpers.*

ijon · 2024-09-03T08:32:34Z

ydb/core/tx/schemeshard/ut_login/ut_login.cpp

-    NAudit::TAuditLogBackends CreateTestAuditLogBackends(std::vector<std::string>& buffer) {
-        NAudit::TAuditLogBackends logBackends;
-        logBackends[NKikimrConfig::TAuditConfig::TXT].emplace_back(new TMemoryLogBackend(buffer));
-        return logBackends;
-    }
-


CreateTestAuditLogBackends() moved to ut_helpers/auditlog_helpers.*

ydb/core/tx/schemeshard/schemeshard_audit_log.cpp

ydb/core/tx/schemeshard/ut_restore/ut_restore.cpp

- rename `tx_id` to `id` - rename `export_uid` to `uid` - add testing for `uid` - add additional parts for export to yt

github-actions · 2024-09-03T11:09:49Z

⚪ 2024-09-03 11:09:49 UTC Pre-commit check linux-x86_64-release-clang14 for 6e85fee has started.
⚪ 2024-09-03 11:12:55 UTC ya make is running...
🟢 2024-09-03 11:19:35 UTC Build successful.

github-actions · 2024-09-03T11:09:52Z

⚪ 2024-09-03 11:09:52 UTC Pre-commit check linux-x86_64-relwithdebinfo for 6e85fee has started.
⚪ 2024-09-03 11:12:58 UTC ya make is running...
🟡 2024-09-03 12:23:14 UTC Some tests failed, follow the links below. Going to retry failed tests...

Test history | Ya make output

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
40730	35294	0	6	5406	24

⚪ 2024-09-03 12:26:51 UTC ya make is running... (failed tests rerun, try 2)
🟢 2024-09-03 12:36:43 UTC Tests successful.

Test history | Ya make output

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
42 (only retried tests)	22	0	0	0	20

🟢 2024-09-03 12:36:50 UTC Build successful.
🟡 2024-09-03 12:37:24 UTC ydbd size 8.3 GiB changed* by +466.5 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: `199ba9d`	merge: `6e85fee`	diff	diff %
ydbd size	8 917 968 560 Bytes	8 918 446 240 Bytes	+466.5 KiB	+0.005%
ydbd stripped size	481 779 912 Bytes	481 810 472 Bytes	+29.8 KiB	+0.006%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

github-actions · 2024-09-03T11:10:05Z

⚪ 2024-09-03 11:10:05 UTC Pre-commit check linux-x86_64-release-asan for 6e85fee has started.
⚪ 2024-09-03 11:13:42 UTC ya make is running...
🔴 2024-09-03 13:02:07 UTC Some tests failed, follow the links below.

Test history | Ya make output

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
11493	11380	0	38	38	37

🟢 2024-09-03 13:03:09 UTC Build successful.
🟡 2024-09-03 13:03:50 UTC ydbd size 5.5 GiB changed* by +575.0 KiB, which is >= 100.0 KiB vs main: Warning

ydbd size dash	main: `199ba9d`	merge: `6e85fee`	diff	diff %
ydbd size	5 954 133 800 Bytes	5 954 722 584 Bytes	+575.0 KiB	+0.010%
ydbd stripped size	1 489 375 920 Bytes	1 489 479 248 Bytes	+100.9 KiB	+0.007%

^{*please be aware that the difference is based on comparing your commit and the last completed build from the post-commit, check comparation}

Add audit logging for (database/tables) export and import operations. Export/import cancel/forget requests are still not logged. KIKIMR-21797

Add audit logging for (database/tables) export and import operations. Cancel/Forget requests for export/imports are still not audit logged. KIKIMR-21797 merge #8550 (a418278) from `main`.

Add audit logging for (database/tables) export and import operations. Cancel/Forget requests for export/imports are still not audit logged. KIKIMR-21797 merge ydb-platform#8550 (a418278) from `main`.

github-actions bot added improvement and removed improvement labels Aug 30, 2024

This comment was marked as outdated.

Sign in to view

ijon force-pushed the auditlog-add-database-export-import branch from cedfab5 to ce4635e Compare September 2, 2024 15:46

This comment was marked as outdated.

Sign in to view

auditlog: add exports/imports

bb6235c

Add audit logging for (database/tables) export and import operations. Export/import cancel/forget requests are still not logged. KIKIMR-21797

ijon force-pushed the auditlog-add-database-export-import branch from ce4635e to bb6235c Compare September 2, 2024 16:00

This comment was marked as outdated.

Sign in to view

ijon commented Sep 3, 2024

View reviewed changes

github-actions bot added improvement and removed improvement labels Sep 3, 2024

ijon requested a review from molotkov-and September 3, 2024 08:35

molotkov-and reviewed Sep 3, 2024

View reviewed changes

ydb/core/tx/schemeshard/schemeshard_audit_log.cpp Outdated Show resolved Hide resolved

molotkov-and reviewed Sep 3, 2024

View reviewed changes

ydb/core/tx/schemeshard/schemeshard_audit_log.cpp Show resolved Hide resolved

molotkov-and reviewed Sep 3, 2024

View reviewed changes

ydb/core/tx/schemeshard/ut_restore/ut_restore.cpp Outdated Show resolved Hide resolved

fixup

7d179bf

- rename `tx_id` to `id` - rename `export_uid` to `uid` - add testing for `uid` - add additional parts for export to yt

This comment was marked as outdated.

Sign in to view

fixup: move FindAuditLine() to auditlog_helpers

98e3379

ijon requested a review from molotkov-and September 3, 2024 11:47

molotkov-and approved these changes Sep 3, 2024

View reviewed changes

ijon merged commit a418278 into ydb-platform:main Sep 3, 2024
10 of 12 checks passed

ijon deleted the auditlog-add-database-export-import branch September 3, 2024 13:47

GrigoriyPA mentioned this pull request Sep 4, 2024

YQ-3151 added issues for timeout and cancelled #8718

Merged

ijon added a commit to ijon/ydb that referenced this pull request Sep 4, 2024

auditlog: add exports/imports (ydb-platform#8550)

41e034e

Add audit logging for (database/tables) export and import operations. Export/import cancel/forget requests are still not logged. KIKIMR-21797

ijon mentioned this pull request Sep 4, 2024

24-2: auditlog: add exports/imports #8750

Merged

ijon added a commit to ijon/ydb that referenced this pull request Sep 4, 2024

auditlog: add exports/imports (ydb-platform#8550)

6485cb3

Add audit logging for (database/tables) export and import operations. Export/import cancel/forget requests are still not logged. KIKIMR-21797

ijon mentioned this pull request Sep 4, 2024

24-3: auditlog: add exports/imports #8751

Merged

ijon added a commit to ijon/ydb that referenced this pull request Sep 4, 2024

auditlog: add exports/imports (ydb-platform#8550)

5b173cc

Add audit logging for (database/tables) export and import operations. Export/import cancel/forget requests are still not logged. KIKIMR-21797

ijon added a commit to ijon/ydb that referenced this pull request Sep 4, 2024

auditlog: add exports/imports (ydb-platform#8550)

4033dd9

Add audit logging for (database/tables) export and import operations. Export/import cancel/forget requests are still not logged. KIKIMR-21797

This was referenced Sep 5, 2024

YQ kqprun added cancel after for script requests #8768

Merged

Fixed unit test ManyColumnShards #8777

Merged

niksaveliev mentioned this pull request Sep 9, 2024

Fix CLI max partitions default #8944

Merged

niksaveliev mentioned this pull request Sep 12, 2024

Fix reading without consumer from federation #9136

Merged

StekPerepolnen mentioned this pull request Sep 17, 2024

doc hc api images source #9376

Merged

ijon mentioned this pull request Jan 15, 2025

audiglog: fix item_count field name for the imports #13390

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

auditlog: add exports/imports #8550

auditlog: add exports/imports #8550

ijon commented Aug 30, 2024 •

edited

Loading

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

ijon Sep 3, 2024 •

edited

Loading

ijon Sep 3, 2024 •

edited

Loading

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

github-actions bot commented Sep 3, 2024 •

edited

Loading

github-actions bot commented Sep 3, 2024 •

edited

Loading

github-actions bot commented Sep 3, 2024 •

edited

Loading

auditlog: add exports/imports #8550

auditlog: add exports/imports #8550

Conversation

ijon commented Aug 30, 2024 • edited Loading

Changelog entry

Changelog category

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

ijon Sep 3, 2024 • edited Loading

Choose a reason for hiding this comment

ijon Sep 3, 2024 • edited Loading

Choose a reason for hiding this comment

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

github-actions bot commented Sep 3, 2024 • edited Loading

github-actions bot commented Sep 3, 2024 • edited Loading

github-actions bot commented Sep 3, 2024 • edited Loading

ijon commented Aug 30, 2024 •

edited

Loading

ijon Sep 3, 2024 •

edited

Loading

ijon Sep 3, 2024 •

edited

Loading

github-actions bot commented Sep 3, 2024 •

edited

Loading

github-actions bot commented Sep 3, 2024 •

edited

Loading

github-actions bot commented Sep 3, 2024 •

edited

Loading