Skip to content

coord: Prioritize read latency over write latency #13967

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jkosh44 merged 5 commits into from
Aug 11, 2022
Merged

coord: Prioritize read latency over write latency #13967

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
5024870
coord: Prioritize read latency over write latency
jkosh44 Jul 29, 2022
e6d5473
Respond to Frank's comments
jkosh44 Aug 8, 2022
4188858
Deflake test
jkosh44 Aug 8, 2022
535bda1
Merge branch 'main' of github.com:MaterializeInc/materialize into adv…
jkosh44 Aug 8, 2022
a87ab70
Further deflake test
jkosh44 Aug 8, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 21 additions & 14 deletions src/adapter/src/coord.rs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -82,7 +82,7 @@ use rand::seq::SliceRandom;
use timely::progress::{Antichain, Timestamp as TimelyTimestamp};
use tokio::runtime::Handle as TokioHandle;
use tokio::select;
use tokio::sync::{mpsc, oneshot, watch};
use tokio::sync::{mpsc, oneshot, watch, OwnedMutexGuard};
use tracing::{span, Level};
use uuid::Uuid;

Expand Down Expand Up @@ -114,7 +114,7 @@ use crate::catalog::{
};
use crate::client::{Client, ConnectionId, Handle};
use crate::command::{Canceled, Command, ExecuteResponse};
use crate::coord::appends::{AdvanceLocalInput, Deferred, PendingWriteTxn};
use crate::coord::appends::{BuiltinTableUpdateSource, Deferred, PendingWriteTxn};
use crate::coord::id_bundle::CollectionIdBundle;
use crate::coord::peek::PendingPeek;
use crate::coord::read_policy::{ReadCapability, ReadHolds};
Expand All @@ -123,7 +123,7 @@ use crate::error::AdapterError;
use crate::session::{EndTransactionAction, Session};
use crate::sink_connection;
use crate::tail::PendingTail;
use crate::util::ClientTransmitter;
use crate::util::{ClientTransmitter, CompletedClientTransmitter};

pub(crate) mod id_bundle;
pub(crate) mod peek;
Expand Down Expand Up @@ -155,11 +155,21 @@ pub enum Message<T = mz_repr::Timestamp> {
SinkConnectionReady(SinkConnectionReady),
SendDiffs(SendDiffs),
WriteLockGrant(tokio::sync::OwnedMutexGuard<()>),
AdvanceTimelines,
AdvanceLocalInput(AdvanceLocalInput<T>),
GroupCommit,
/// Initiates a group commit.
GroupCommitInitiate,
/// Makes a group commit visible to all clients.
GroupCommitApply(
/// Timestamp of the writes in the group commit.
T,
/// Clients waiting on responses from the group commit.
Vec<CompletedClientTransmitter<ExecuteResponse>>,
/// Optional lock if the group commit contained writes to user tables.
Option<OwnedMutexGuard<()>>,
),
ComputeInstanceStatus(ComputeInstanceEvent),
RemovePendingPeeks { conn_id: ConnectionId },
RemovePendingPeeks {
conn_id: ConnectionId,
},
LinearizeReads(Vec<PendingTxn>),
StorageUsage,
}
Expand Down Expand Up @@ -658,7 +668,7 @@ impl<S: Append + 'static> Coordinator<S> {
let WriteTimestamp {
timestamp: _,
advance_to,
} = self.get_and_step_local_write_ts().await;
} = self.get_local_write_ts().await;
let appends = entries
.iter()
.filter(|entry| entry.is_table())
Expand Down Expand Up @@ -694,7 +704,8 @@ impl<S: Append + 'static> Coordinator<S> {
builtin_table_updates.extend(retractions);
}

self.send_builtin_table_updates(builtin_table_updates).await;
self.send_builtin_table_updates(builtin_table_updates, BuiltinTableUpdateSource::DDL)
.await;

Ok(())
}
Expand Down Expand Up @@ -764,7 +775,7 @@ impl<S: Append + 'static> Coordinator<S> {
}
// `tick()` on `Interval` is cancel-safe:
// https://docs.rs/tokio/1.19.2/tokio/time/struct.Interval.html#cancel-safety
_ = advance_timelines_interval.tick() => Message::AdvanceTimelines,
_ = advance_timelines_interval.tick() => Message::GroupCommitInitiate,
Copy link
Copy Markdown
Contributor

@frankmcsherry frankmcsherry Aug 6, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we wanted to keep things moving, we could tick this each time the group commit completes.

Copy link
Copy Markdown
Contributor Author

@jkosh44 jkosh44 Aug 8, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wouldn't be against that. My one fear is that we'd end up clogging the Coordinator up somehow, but since append is now done asynchronously it should be fine?

_ = storage_usage_update_interval.tick() => Message::StorageUsage,
};

Expand All @@ -774,10 +785,6 @@ impl<S: Append + 'static> Coordinator<S> {
let _enter = span.enter();

self.handle_message(msg).await;

if let Some(timestamp) = self.get_local_timestamp_oracle_mut().should_advance_to() {
self.queue_local_input_advances(timestamp).await;
}
Comment on lines -778 to -780
Copy link
Copy Markdown
Contributor

@frankmcsherry frankmcsherry Aug 6, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

By removing this, do we risk making the table advancement a low priority event? Seems removing this takes it from "once per message received" to "lowest priority event".

Copy link
Copy Markdown
Contributor Author

@jkosh44 jkosh44 Aug 8, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Queuing a table advancement was happening on a "once per message received" rate. However, actually executing the table advancements was already the lowest priority message that the Coordinator handled.

In practice the only things that triggered this were group commit, DDL, and the advance_timelines_interval, all of which now explicitly advance tables.

}
}

Expand Down
Loading