Skip to content

feat(replays): add replays to self hosted #1990

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Feb 28, 2023
Merged

feat(replays): add replays to self hosted #1990

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 22 additions & 0 deletions _integration-test/run.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -171,6 +171,28 @@ $dcr --no-deps web python3 /etc/sentry/test-custom-ca-roots.py
source _integration-test/custom-ca-roots/teardown.sh
echo "${_endgroup}"

echo "${_group}Test that replays work ..."
echo "Creating test replay..."
TEST_REPLAY_ID=$(
export LC_ALL=C
head /dev/urandom | tr -dc "a-f0-9" | head -c 32
)
TIME_IN_SECONDS=$(date +%s)
curl -sf --data '{"event_id":"'"$TEST_REPLAY_ID"'","sdk":{"name":"sentry.javascript.browser","version":"7.38.0"}}
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

for some reason the bash syntax of $' for C escaped strings for newlines wasn't working so just put in the real values

{"type":"replay_event"}
{"type":"replay_event","replay_start_timestamp":$TIME_IN_SECONDS,"timestamp":$TIME_IN_SECONDS,"error_ids":[],"trace_ids":[],"urls":["example.com"],"replay_id":"'"$TEST_REPLAY_ID"'","segment_id":0,"replay_type":"session","event_id":"'"$TEST_REPLAY_ID"'","environment":"production","sdk":{"name":"sentry.javascript.browser","version":"7.38.0"},"request":{"url":"example.com","headers":{"platform":"javascript","contexts":{"replay":{"session_sample_rate":1,"error_sample_rate":1}}}
{"type":"replay_recording","length":19}
{"segment_id":0}
[]' -H 'Content-Type: application/json' -H "X-Sentry-Auth: Sentry sentry_version=7, sentry_key=$SENTRY_KEY, sentry_client=test-bash/0.1" "$SENTRY_TEST_HOST/api/$PROJECT_ID/envelope/" -o /dev/null

printf "Getting the test replay back"
REPLAY_SEGMENT_PATH="api/0/projects/sentry/internal/replays/$TEST_EVENT_ID/recording-segments/?download"
REPLAY_EVENT_PATH="api/0/projects/sentry/internal/replays/$TEST_EVENT_ID/"
timeout 60 bash -c 'until $(sentry_api_request "$REPLAY_EVENT_PATH" -Isf -X GET -o /dev/null); do printf '.'; sleep 0.5; done'
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So this test itself will take more than 2 minutes? Is there any way to cut down on that time? I know dev-infra isn't super happy about self-hosted e2e tests taking over 15 minutes

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hmm it shouldn't, i just copied the defaults from the other tests.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

are we noticing its increasing the time of the test by two minutes?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i can lower the timeout here if needed i don't think the tests take more than a few seconds

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ahh I read that wrong, guess I'm still waking up 😅

timeout 60 bash -c 'until $(sentry_api_request "$REPLAY_SEGMENT_PATH" -Isf -X GET -o /dev/null); do printf '.'; sleep 0.5; done'
echo " got it!"
echo "${_endgroup}"

# Table formatting based on https://stackoverflow.com/a/39144364
COMPOSE_PS_OUTPUT=$(docker compose ps --format json | jq -r \
'.[] |
Expand Down
6 changes: 6 additions & 0 deletions docker-compose.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -258,6 +258,9 @@ services:
snuba-transactions-consumer:
<<: *snuba_defaults
command: consumer --storage transactions --consumer-group transactions_group --auto-offset-reset=latest --max-batch-time-ms 750
snuba-replays-consumer:
<<: *snuba_defaults
command: consumer --storage replays --auto-offset-reset=latest --max-batch-time-ms 750
snuba-replacer:
<<: *snuba_defaults
command: replacer --storage errors --auto-offset-reset=latest
Expand Down Expand Up @@ -322,6 +325,9 @@ services:
ingest-consumer:
<<: *sentry_defaults
command: run ingest-consumer --all-consumer-types
ingest-replay-recordings:
<<: *sentry_defaults
command: run ingest-replay-recordings
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Dang 33 services 😭

clickhouse
cron
geoipupdate
ingest-consumer
ingest-replay-recordings
kafka
memcached
nginx
post-process-forwarder-errors
post-process-forwarder-transactions
postgres
redis
relay
sentry-cleanup
smtp
snuba-api
snuba-cleanup
snuba-consumer
snuba-outcomes-consumer
snuba-replacer
snuba-replays-consumer
snuba-sessions-consumer
snuba-subscription-consumer-events
snuba-subscription-consumer-transactions
snuba-transactions-cleanup
snuba-transactions-consumer
subscription-consumer-events
subscription-consumer-transactions
symbolicator
symbolicator-cleanup
web
worker
zookeeper

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

11 snubas 😖

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

getsentry/snuba#1670

That said this would only help with a 33% reduction.

Looks like we need a similar issue for Sentry to consolidate all those query-subscription-consumer and post-process-forwarder services.

post-process-forwarder-errors:
<<: *sentry_defaults
# Increase `--commit-batch-size 1` below to deal with high-load environments.
Expand Down
2 changes: 1 addition & 1 deletion install/create-kafka-topics.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ echo "${_group}Creating additional Kafka topics ..."
# NOTE: This step relies on `kafka` being available from the previous `snuba-api bootstrap` step
# XXX(BYK): We cannot use auto.create.topics as Confluence and Apache hates it now (and makes it very hard to enable)
EXISTING_KAFKA_TOPICS=$($dcr -T kafka kafka-topics --list --bootstrap-server kafka:9092 2>/dev/null)
NEEDED_KAFKA_TOPICS="ingest-attachments ingest-transactions ingest-events"
NEEDED_KAFKA_TOPICS="ingest-attachments ingest-transactions ingest-events ingest-replay-recordings"
for topic in $NEEDED_KAFKA_TOPICS; do
if ! echo "$EXISTING_KAFKA_TOPICS" | grep -wq $topic; then
$dcr kafka kafka-topics --create --topic $topic --bootstrap-server kafka:9092
Expand Down
2 changes: 2 additions & 0 deletions sentry/sentry.conf.example.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -258,6 +258,8 @@ def get_internal_network():
"organizations:sso-saml2",
"organizations:performance-view",
"organizations:advanced-search",
"organizations:session-replay",
"organizations:session-replay-ui",
"projects:custom-inbound-filters",
"projects:data-forwarding",
"projects:discard-groups",
Expand Down