Skip to content

test(replay): Fix flakey extendNetworkBreadcrumbs/fetch tests #11142

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 13 commits into from
May 13, 2024
Merged

test(replay): Fix flakey extendNetworkBreadcrumbs/fetch tests #11142

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
7d59b6f
test(replay): Fix flakey `extendNetworkBreadcrumbs/fetch` tests
billyvg Mar 15, 2024
e01233b
formatting
billyvg Mar 15, 2024
ac017ba
collectReplayRequests
billyvg Mar 15, 2024
b4bc852
more flakes
billyvg Mar 15, 2024
9e8e3fd
move around replay snapshots
billyvg Apr 15, 2024
d050be3
upload artifacts on failure
billyvg Apr 15, 2024
000e1b7
oops dont need continue on error
billyvg Apr 15, 2024
b50c563
format
billyvg Apr 15, 2024
4534e7e
js habits
billyvg Apr 15, 2024
1939833
Merge branch 'develop' into test-replay-flakes-integration-network-cr…
billyvg Apr 16, 2024
a7c9865
Merge branch 'develop' into test-replay-flakes-integration-network-cr…
billyvg Apr 17, 2024
c88e1a1
Merge branch 'develop' into test-replay-flakes-integration-network-cr…
billyvg May 13, 2024
cfdcb3a
Update .github/workflows/flaky-test-detector.yml
billyvg May 13, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions .github/workflows/flaky-test-detector.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -79,8 +79,17 @@ jobs:
browser_integration: dev-packages/browser-integration-tests/suites/**/test.ts

- name: Detect flaky tests
id: test
run: yarn test:detect-flaky
working-directory: dev-packages/browser-integration-tests
env:
CHANGED_TEST_PATHS: ${{ steps.changed.outputs.browser_integration_files }}
TEST_RUN_COUNT: 'AUTO'

- name: Artifacts upload
uses: actions/upload-artifact@v4
if: failure() && steps.test.outcome == 'failure'
with:
name: playwright-test-results
path: test-results
retention-days: 5
188 changes: 98 additions & 90 deletions ...egration-tests/suites/replay/extendNetworkBreadcrumbs/fetch/captureRequestHeaders/test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,9 +8,7 @@ import {
shouldSkipReplayTest,
} from '../../../../../utils/replayHelpers';

// Skipping because this test is flaky
// https://github.com/getsentry/sentry-javascript/issues/11062
sentryTest.skip('handles empty/missing request headers', async ({ getLocalTestPath, page, browserName }) => {
sentryTest('handles empty/missing request headers', async ({ getLocalTestPath, page, browserName }) => {
if (shouldSkipReplayTest()) {
sentryTest.skip();
}
Expand All @@ -37,18 +35,21 @@ sentryTest.skip('handles empty/missing request headers', async ({ getLocalTestPa
const url = await getLocalTestPath({ testDir: __dirname });
await page.goto(url);

await page.evaluate(() => {
/* eslint-disable */
fetch('http://localhost:7654/foo', {
method: 'POST',
}).then(() => {
// @ts-expect-error Sentry is a global
Sentry.captureException('test error');
});
/* eslint-enable */
});
const [, request, { replayRecordingSnapshots }] = await Promise.all([
page.evaluate(() => {
/* eslint-disable */
fetch('http://localhost:7654/foo', {
method: 'POST',
}).then(() => {
// @ts-expect-error Sentry is a global
Sentry.captureException('test error');
});
/* eslint-enable */
}),
requestPromise,
replayRequestPromise,
]);

const request = await requestPromise;
const eventData = envelopeRequestParser(request);

expect(eventData.exception?.values).toHaveLength(1);
Expand All @@ -65,7 +66,6 @@ sentryTest.skip('handles empty/missing request headers', async ({ getLocalTestPa
},
});

const { replayRecordingSnapshots } = await replayRequestPromise;
expect(getReplayPerformanceSpans(replayRecordingSnapshots).filter(span => span.op === 'resource.fetch')).toEqual([
{
data: {
Expand Down Expand Up @@ -110,25 +110,28 @@ sentryTest('captures request headers as POJO', async ({ getLocalTestPath, page,
const url = await getLocalTestPath({ testDir: __dirname });
await page.goto(url);

await page.evaluate(() => {
/* eslint-disable */
fetch('http://localhost:7654/foo', {
method: 'POST',
headers: {
Accept: 'application/json',
'Content-Type': 'application/json',
Cache: 'no-cache',
'X-Custom-Header': 'foo',
'X-Test-Header': 'test-value',
},
}).then(() => {
// @ts-expect-error Sentry is a global
Sentry.captureException('test error');
});
/* eslint-enable */
});
const [, request, { replayRecordingSnapshots }] = await Promise.all([
page.evaluate(() => {
/* eslint-disable */
fetch('http://localhost:7654/foo', {
method: 'POST',
headers: {
Accept: 'application/json',
'Content-Type': 'application/json',
Cache: 'no-cache',
'X-Custom-Header': 'foo',
'X-Test-Header': 'test-value',
},
}).then(() => {
// @ts-expect-error Sentry is a global
Sentry.captureException('test error');
});
/* eslint-enable */
}),
requestPromise,
replayRequestPromise,
]);

const request = await requestPromise;
const eventData = envelopeRequestParser(request);

expect(eventData.exception?.values).toHaveLength(1);
Expand All @@ -145,7 +148,6 @@ sentryTest('captures request headers as POJO', async ({ getLocalTestPath, page,
},
});

const { replayRecordingSnapshots } = await replayRequestPromise;
expect(getReplayPerformanceSpans(replayRecordingSnapshots).filter(span => span.op === 'resource.fetch')).toEqual([
{
data: {
Expand Down Expand Up @@ -195,25 +197,28 @@ sentryTest('captures request headers on Request', async ({ getLocalTestPath, pag
const url = await getLocalTestPath({ testDir: __dirname });
await page.goto(url);

await page.evaluate(() => {
const request = new Request('http://localhost:7654/foo', {
method: 'POST',
headers: {
Accept: 'application/json',
'Content-Type': 'application/json',
Cache: 'no-cache',
'X-Custom-Header': 'foo',
},
});
/* eslint-disable */
fetch(request).then(() => {
// @ts-expect-error Sentry is a global
Sentry.captureException('test error');
});
/* eslint-enable */
});
const [, request, { replayRecordingSnapshots }] = await Promise.all([
page.evaluate(() => {
const request = new Request('http://localhost:7654/foo', {
method: 'POST',
headers: {
Accept: 'application/json',
'Content-Type': 'application/json',
Cache: 'no-cache',
'X-Custom-Header': 'foo',
},
});
/* eslint-disable */
fetch(request).then(() => {
// @ts-expect-error Sentry is a global
Sentry.captureException('test error');
});
/* eslint-enable */
}),
requestPromise,
replayRequestPromise,
]);

const request = await requestPromise;
const eventData = envelopeRequestParser(request);

expect(eventData.exception?.values).toHaveLength(1);
Expand All @@ -230,7 +235,6 @@ sentryTest('captures request headers on Request', async ({ getLocalTestPath, pag
},
});

const { replayRecordingSnapshots } = await replayRequestPromise;
expect(getReplayPerformanceSpans(replayRecordingSnapshots).filter(span => span.op === 'resource.fetch')).toEqual([
{
data: {
Expand Down Expand Up @@ -280,25 +284,28 @@ sentryTest('captures request headers as Headers instance', async ({ getLocalTest

await page.goto(url);

await page.evaluate(() => {
const headers = new Headers();
headers.append('Accept', 'application/json');
headers.append('Content-Type', 'application/json');
headers.append('Cache', 'no-cache');
headers.append('X-Custom-Header', 'foo');
const [, request, { replayRecordingSnapshots }] = await Promise.all([
page.evaluate(() => {
const headers = new Headers();
headers.append('Accept', 'application/json');
headers.append('Content-Type', 'application/json');
headers.append('Cache', 'no-cache');
headers.append('X-Custom-Header', 'foo');

/* eslint-disable */
fetch('http://localhost:7654/foo', {
method: 'POST',
headers,
}).then(() => {
// @ts-expect-error Sentry is a global
Sentry.captureException('test error');
});
/* eslint-enable */
});
/* eslint-disable */
fetch('http://localhost:7654/foo', {
method: 'POST',
headers,
}).then(() => {
// @ts-expect-error Sentry is a global
Sentry.captureException('test error');
});
/* eslint-enable */
}),
requestPromise,
replayRequestPromise,
]);

const request = await requestPromise;
const eventData = envelopeRequestParser(request);

expect(eventData.exception?.values).toHaveLength(1);
Expand All @@ -315,7 +322,6 @@ sentryTest('captures request headers as Headers instance', async ({ getLocalTest
},
});

const { replayRecordingSnapshots } = await replayRequestPromise;
expect(getReplayPerformanceSpans(replayRecordingSnapshots).filter(span => span.op === 'resource.fetch')).toEqual([
{
data: {
Expand Down Expand Up @@ -364,25 +370,28 @@ sentryTest('does not captures request headers if URL does not match', async ({ g
const url = await getLocalTestPath({ testDir: __dirname });
await page.goto(url);

await page.evaluate(() => {
/* eslint-disable */
fetch('http://localhost:7654/bar', {
method: 'POST',
headers: {
Accept: 'application/json',
'Content-Type': 'application/json',
Cache: 'no-cache',
'X-Custom-Header': 'foo',
'X-Test-Header': 'test-value',
},
}).then(() => {
// @ts-expect-error Sentry is a global
Sentry.captureException('test error');
});
/* eslint-enable */
});
const [, request, { replayRecordingSnapshots }] = await Promise.all([
page.evaluate(() => {
/* eslint-disable */
fetch('http://localhost:7654/bar', {
method: 'POST',
headers: {
Accept: 'application/json',
'Content-Type': 'application/json',
Cache: 'no-cache',
'X-Custom-Header': 'foo',
'X-Test-Header': 'test-value',
},
}).then(() => {
// @ts-expect-error Sentry is a global
Sentry.captureException('test error');
});
/* eslint-enable */
}),
requestPromise,
replayRequestPromise,
]);

const request = await requestPromise;
const eventData = envelopeRequestParser(request);

expect(eventData.exception?.values).toHaveLength(1);
Expand All @@ -399,7 +408,6 @@ sentryTest('does not captures request headers if URL does not match', async ({ g
},
});

const { replayRecordingSnapshots } = await replayRequestPromise;
expect(getReplayPerformanceSpans(replayRecordingSnapshots).filter(span => span.op === 'resource.fetch')).toEqual([
{
data: {
Expand Down
62 changes: 32 additions & 30 deletions ...integration-tests/suites/replay/extendNetworkBreadcrumbs/fetch/captureRequestSize/test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,9 +8,7 @@ import {
shouldSkipReplayTest,
} from '../../../../../utils/replayHelpers';

// Skipping because this test is flaky
// https://github.com/getsentry/sentry-javascript/issues/10395
sentryTest.skip('captures request body size when body is sent', async ({ getLocalTestPath, page }) => {
sentryTest('captures request body size when body is sent', async ({ getLocalTestPath, page }) => {
if (shouldSkipReplayTest()) {
sentryTest.skip();
}
Expand All @@ -37,19 +35,22 @@ sentryTest.skip('captures request body size when body is sent', async ({ getLoca
const url = await getLocalTestPath({ testDir: __dirname });
await page.goto(url);

await page.evaluate(() => {
/* eslint-disable */
fetch('http://localhost:7654/foo', {
method: 'POST',
body: '{"foo":"bar"}',
}).then(() => {
// @ts-expect-error Sentry is a global
Sentry.captureException('test error');
});
/* eslint-enable */
});
const [, request, { replayRecordingSnapshots }] = await Promise.all([
page.evaluate(() => {
/* eslint-disable */
fetch('http://localhost:7654/foo', {
method: 'POST',
body: '{"foo":"bar"}',
}).then(() => {
// @ts-expect-error Sentry is a global
Sentry.captureException('test error');
});
/* eslint-enable */
}),
requestPromise,
replayRequestPromise,
]);

const request = await requestPromise;
const eventData = envelopeRequestParser(request);

expect(eventData.exception?.values).toHaveLength(1);
Expand All @@ -67,7 +68,6 @@ sentryTest.skip('captures request body size when body is sent', async ({ getLoca
},
});

const { replayRecordingSnapshots } = await replayRequestPromise;
expect(getReplayPerformanceSpans(replayRecordingSnapshots).filter(span => span.op === 'resource.fetch')).toEqual([
{
data: {
Expand Down Expand Up @@ -122,21 +122,24 @@ sentryTest('captures request size from non-text request body', async ({ getLocal
const url = await getLocalTestPath({ testDir: __dirname });
await page.goto(url);

await page.evaluate(() => {
/* eslint-disable */
const blob = new Blob(['<html>Hello world!!</html>'], { type: 'text/html' });
const [, request, { replayRecordingSnapshots }] = await Promise.all([
page.evaluate(() => {
/* eslint-disable */
const blob = new Blob(['<html>Hello world!!</html>'], { type: 'text/html' });

fetch('http://localhost:7654/foo', {
method: 'POST',
body: blob,
}).then(() => {
// @ts-expect-error Sentry is a global
Sentry.captureException('test error');
});
/* eslint-enable */
});
fetch('http://localhost:7654/foo', {
method: 'POST',
body: blob,
}).then(() => {
// @ts-expect-error Sentry is a global
Sentry.captureException('test error');
});
/* eslint-enable */
}),
requestPromise,
replayRequestPromise,
]);

const request = await requestPromise;
const eventData = envelopeRequestParser(request);

expect(eventData.exception?.values).toHaveLength(1);
Expand All @@ -154,7 +157,6 @@ sentryTest('captures request size from non-text request body', async ({ getLocal
},
});

const { replayRecordingSnapshots } = await replayRequestPromise;
expect(getReplayPerformanceSpans(replayRecordingSnapshots).filter(span => span.op === 'resource.fetch')).toEqual([
{
data: {
Expand Down
Loading
Loading