Write metadata cache data to mappings _meta with refresh time update #805
+1,052
−71
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…rch-project#744) * write mock metadata cache data to mappings _meta Signed-off-by: Sean Kao <seankao@amazon.com> * Enable write to cache by default Signed-off-by: Sean Kao <seankao@amazon.com> * bugfix: _meta.latestId missing when create index Signed-off-by: Sean Kao <seankao@amazon.com> * set and unset config in test suite Signed-off-by: Sean Kao <seankao@amazon.com> * fix: use member flintSparkConf Signed-off-by: Sean Kao <seankao@amazon.com> --------- Signed-off-by: Sean Kao <seankao@amazon.com>
Signed-off-by: Sean Kao <seankao@amazon.com>
Signed-off-by: Sean Kao <seankao@amazon.com>
Signed-off-by: Sean Kao <seankao@amazon.com>
Signed-off-by: Sean Kao <seankao@amazon.com>
Signed-off-by: Sean Kao <seankao@amazon.com>
Signed-off-by: Sean Kao <seankao@amazon.com>
Signed-off-by: Sean Kao <seankao@amazon.com>
Signed-off-by: Sean Kao <seankao@amazon.com>
Signed-off-by: Sean Kao <seankao@amazon.com>
Signed-off-by: Sean Kao <seankao@amazon.com>
Signed-off-by: Sean Kao <seankao@amazon.com>
Signed-off-by: Sean Kao <seankao@amazon.com>
Signed-off-by: Sean Kao <seankao@amazon.com>
Signed-off-by: Sean Kao <seankao@amazon.com>
Signed-off-by: Sean Kao <seankao@amazon.com>
Signed-off-by: Sean Kao <seankao@amazon.com>
seankao-az
requested review from
dai-chen,
rupal-bq,
vamsi-amazon,
penghuo,
anirudha,
kaituo,
YANG-DB and
LantaoJin
as code owners
Signed-off-by: Sean Kao <seankao@amazon.com>
|
add label to backport to the nexus branch. |
Signed-off-by: Sean Kao <seankao@amazon.com>
seankao-az
requested review from
mengweieric,
noCharger and
ykmr1224
as code owners
Signed-off-by: Sean Kao <seankao@amazon.com>
Signed-off-by: Sean Kao <seankao@amazon.com>
seankao-az
force-pushed
the
write-metadata-cache
branch
from
5f3af3b
to
7a8e1f3
Compare
ykmr1224
reviewed
Oct 25, 2024
| * Handles refresh for refresh mode AUTO, which is used exclusively by auto refresh index with | ||
| * internal scheduler. | ||
| */ | ||
| private def refreshIndexAuto( |
There was a problem hiding this comment.
Why don't we update for auto refresh?
| * @throws IllegalArgumentException if the schedule string is invalid | ||
| */ | ||
| public static IntervalSchedule parse(String scheduleStr) { | ||
| public static Long parseMillis(String scheduleStr) { |
There was a problem hiding this comment.
ParseMillis sounds like parsing millisec string. Should we call it parseAndConvertToMillis?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Metadata Cache Writer
For the most part, same as
In addition to the regular metadata storage using
FlintIndexMetadataService, we're dual-writing additional fields, defined byFlintMetadataCache, to the index mappings_metafield. It's intended for frontend users to access some crucial metadata for an index quickly without invoking another backend API call.This PR adds such fields for all indexes, if the spark config
spark.flint.metadataCacheWrite.enabledis set to true._meta.properties.metadataCacheVersion: "1.0"_meta.properties.refreshInterval: Integer. Refresh interval of an index measured in seconds. This field is added only if index refresh type is auto refresh and refresh_interval is set_meta.properties.sourceTables: Array of Strings. For now, it's mocked data. Update coming in later PR._meta.properties.lastRefreshTime: Long. Timestamp in milliseconds when last refresh happened. This field is added only if index already gets refreshed at least onceLast Refresh Time
Added two new fields in
FlintMetadataLogEntryand bumped version of its json doc from 1.0 to 1.1 (because adding new field but not changing existing fields)These are accurate only for manual refresh (full, incremental) and external scheduler for auto refresh.
For internal scheduler, the
jobStartTime(orcreateTimeinFlintMetadataLogEntry) is used to track streaming job start time.I'm not reusing
createTimebecause they should be updated at different times.For createTime (for internal scheduler) it's during
refreshIndex,recoverIndex,updateIndexManualToAutoBut for lastRefreshStartTime and lastRefreshCompleteTime (for manual refresh and external scheduler) it's only updated in
refreshIndexRelated Issues
_metaas read cache for frontend user to access #746By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.