Skip to content

test(e2e): Add app start crash test for iOS #4593

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 33 commits into from
Feb 25, 2025
Merged

test(e2e): Add app start crash test for iOS #4593

merged 33 commits into from
Feb 25, 2025

Conversation

krystofwoldrich
Copy link
Member

📢 Type of change

  • Enhancement
  • Refactoring

📜 Description

This PR adds tests of iOS capturing a crash before the JS starts.

💚 How did you test it?

sample app, e2e test

#skip-changelog

@krystofwoldrich
chore(samples): Add package scripts for native builds, dsn and testing
cc9543f
@krystofwoldrich
Merge branch 'capture-app-start-errors' into kw-simplify-local-sample…
c20ac08 
…-testing
@krystofwoldrich
fix ci
f2a14c9
@krystofwoldrich
fix android archive path
d9fec19
@krystofwoldrich
fix ios build release
6b8f314
@krystofwoldrich
fix(sample-e2e): Fix type errors missing sentry/core and afterAll
25ab23b
@krystofwoldrich
Merge branch 'kw-fix-bad-merge' into kw-simplify-local-sample-testing
20a132e
@krystofwoldrich
fix IOS_DEVICE empty check
8b2227a
@krystofwoldrich
Merge branch 'capture-app-start-errors' into kw-simplify-local-sample…
d5e8e92 
…-testing
@krystofwoldrich
fix ios archive
20e7f55
@krystofwoldrich
test(e2e): Verify captured Errors Screen transaction
56fe4b8
@krystofwoldrich
add more tests
4cfc937
@krystofwoldrich
Merge branch 'capture-app-start-errors' into kw-test-transactions
b484b07
@krystofwoldrich
revert dsn change
86a675e
@krystofwoldrich
fix lint
71fada9
@krystofwoldrich
remove jest extended from sample e2e tests
0d6f543
@krystofwoldrich
Relax test to expect any type of app start, remove native frames as m…
c5c93d2 
…issing is also valid
@krystofwoldrich
add app, device and react native context smoke tests
def6e30
@krystofwoldrich
include transaction name in the tests
3aae5bf
@krystofwoldrich
test(e2e): Add auto init from JS tests
46782a8
@krystofwoldrich
test(e2e): Add app start crash test for iOS
6b663e0
@krystofwoldrich
split ios test config to manual and auto
bf4ee0d
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Feb 25, 2025 •
edited
Loading

Android (legacy) Performance metrics 🚀

  Plain With Sentry Diff
Startup time 432.45 ms 504.11 ms 71.67 ms
Size 17.75 MiB 20.12 MiB 2.37 MiB

Baseline results on branch: capture-app-start-errors

Startup times

Revision Plain With Sentry Diff
454f971 436.27 ms 478.76 ms 42.48 ms
208f4af 439.23 ms 427.31 ms -11.92 ms
1a89392 412.59 ms 424.69 ms 12.10 ms
555070f 438.67 ms 428.30 ms -10.37 ms
c8578fb 397.17 ms 415.72 ms 18.55 ms
b6d933c 442.52 ms 461.82 ms 19.30 ms
8d0daf6 422.36 ms 449.40 ms 27.04 ms
b75148e 440.04 ms 421.36 ms -18.68 ms
18a5066 427.89 ms 436.47 ms 8.57 ms
5625ce7 476.74 ms 488.24 ms 11.50 ms

App size

Revision Plain With Sentry Diff
454f971 17.75 MiB 20.11 MiB 2.37 MiB
208f4af 17.75 MiB 20.11 MiB 2.37 MiB
1a89392 17.75 MiB 20.11 MiB 2.37 MiB
555070f 17.75 MiB 20.11 MiB 2.37 MiB
c8578fb 17.75 MiB 20.11 MiB 2.36 MiB
b6d933c 17.75 MiB 20.11 MiB 2.37 MiB
8d0daf6 17.75 MiB 20.11 MiB 2.36 MiB
b75148e 17.75 MiB 20.11 MiB 2.37 MiB
18a5066 17.75 MiB 20.11 MiB 2.37 MiB
5625ce7 17.75 MiB 20.11 MiB 2.37 MiB

Previous results on branch: kw-add-app-start-crash-ios

Startup times

Revision Plain With Sentry Diff
b75af25 451.91 ms 467.54 ms 15.63 ms

App size

Revision Plain With Sentry Diff
b75af25 17.75 MiB 20.12 MiB 2.37 MiB

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Feb 25, 2025 •
edited
Loading

Android (new) Performance metrics 🚀

  Plain With Sentry Diff
Startup time 432.85 ms 436.78 ms 3.93 ms
Size 7.15 MiB 8.39 MiB 1.24 MiB

Baseline results on branch: capture-app-start-errors

Startup times

Revision Plain With Sentry Diff
b6d933c+dirty 398.43 ms 456.62 ms 58.19 ms
555070f+dirty 388.25 ms 424.44 ms 36.19 ms
454f971+dirty 439.04 ms 423.88 ms -15.16 ms
b75148e+dirty 428.91 ms 461.26 ms 32.35 ms
5625ce7+dirty 358.15 ms 416.65 ms 58.50 ms
18a5066+dirty 370.06 ms 414.10 ms 44.04 ms
208f4af+dirty 346.93 ms 402.77 ms 55.84 ms
8d0daf6+dirty 393.58 ms 394.84 ms 1.26 ms
1a89392+dirty 425.56 ms 530.65 ms 105.09 ms
c8578fb+dirty 374.17 ms 362.06 ms -12.11 ms

App size

Revision Plain With Sentry Diff
b6d933c+dirty 7.15 MiB 8.38 MiB 1.23 MiB
555070f+dirty 7.15 MiB 8.38 MiB 1.23 MiB
454f971+dirty 7.15 MiB 8.38 MiB 1.23 MiB
b75148e+dirty 7.15 MiB 8.38 MiB 1.23 MiB
5625ce7+dirty 7.15 MiB 8.38 MiB 1.23 MiB
18a5066+dirty 7.15 MiB 8.38 MiB 1.23 MiB
208f4af+dirty 7.15 MiB 8.38 MiB 1.23 MiB
8d0daf6+dirty 7.15 MiB 8.38 MiB 1.23 MiB
1a89392+dirty 7.15 MiB 8.38 MiB 1.23 MiB
c8578fb+dirty 7.15 MiB 8.38 MiB 1.23 MiB

Previous results on branch: kw-add-app-start-crash-ios

Startup times

Revision Plain With Sentry Diff
b75af25+dirty 370.17 ms 357.88 ms -12.30 ms

App size

Revision Plain With Sentry Diff
b75af25+dirty 7.15 MiB 8.39 MiB 1.24 MiB

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Feb 25, 2025 •
edited
Loading

iOS (legacy) Performance metrics 🚀

  Plain With Sentry Diff
Startup time 1220.81 ms 1224.41 ms 3.60 ms
Size 2.63 MiB 3.75 MiB 1.12 MiB

Baseline results on branch: capture-app-start-errors

Startup times

Revision Plain With Sentry Diff
18a5066+dirty 1218.00 ms 1212.27 ms -5.73 ms
b6d933c+dirty 1217.71 ms 1220.58 ms 2.87 ms
1a89392+dirty 1220.12 ms 1216.69 ms -3.43 ms
c8578fb+dirty 1218.55 ms 1217.23 ms -1.32 ms
208f4af+dirty 1209.44 ms 1217.13 ms 7.70 ms
454f971+dirty 1220.55 ms 1225.82 ms 5.27 ms
8d0daf6+dirty 1230.90 ms 1233.16 ms 2.27 ms
b75148e+dirty 1221.53 ms 1220.85 ms -0.68 ms
5625ce7+dirty 1226.98 ms 1217.77 ms -9.21 ms
555070f+dirty 1213.59 ms 1217.79 ms 4.20 ms

App size

Revision Plain With Sentry Diff
18a5066+dirty 2.63 MiB 3.70 MiB 1.06 MiB
b6d933c+dirty 2.63 MiB 3.70 MiB 1.06 MiB
1a89392+dirty 2.63 MiB 3.70 MiB 1.06 MiB
c8578fb+dirty 2.63 MiB 3.74 MiB 1.11 MiB
208f4af+dirty 2.63 MiB 3.69 MiB 1.05 MiB
454f971+dirty 2.63 MiB 3.74 MiB 1.11 MiB
8d0daf6+dirty 2.63 MiB 3.74 MiB 1.11 MiB
b75148e+dirty 2.63 MiB 3.69 MiB 1.06 MiB
5625ce7+dirty 2.63 MiB 3.70 MiB 1.06 MiB
555070f+dirty 2.63 MiB 3.69 MiB 1.05 MiB

Previous results on branch: kw-add-app-start-crash-ios

Startup times

Revision Plain With Sentry Diff
b75af25+dirty 1219.00 ms 1218.45 ms -0.55 ms

App size

Revision Plain With Sentry Diff
b75af25+dirty 2.63 MiB 3.75 MiB 1.12 MiB

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Feb 25, 2025 •
edited
Loading

iOS (new) Performance metrics 🚀

  Plain With Sentry Diff
Startup time 1222.10 ms 1221.63 ms -0.47 ms
Size 3.19 MiB 4.32 MiB 1.13 MiB

Baseline results on branch: capture-app-start-errors

Startup times

Revision Plain With Sentry Diff
18a5066+dirty 1244.20 ms 1251.54 ms 7.34 ms
b6d933c+dirty 1228.02 ms 1235.32 ms 7.30 ms
1a89392+dirty 1229.00 ms 1234.78 ms 5.78 ms
c8578fb+dirty 1225.49 ms 1224.29 ms -1.20 ms
208f4af+dirty 1213.08 ms 1223.82 ms 10.73 ms
454f971+dirty 1233.51 ms 1239.18 ms 5.67 ms
8d0daf6+dirty 1227.80 ms 1236.83 ms 9.03 ms
b75148e+dirty 1202.72 ms 1212.04 ms 9.32 ms
5625ce7+dirty 1219.73 ms 1223.80 ms 4.07 ms
555070f+dirty 1223.61 ms 1227.57 ms 3.96 ms

App size

Revision Plain With Sentry Diff
18a5066+dirty 3.19 MiB 4.26 MiB 1.08 MiB
b6d933c+dirty 3.19 MiB 4.26 MiB 1.08 MiB
1a89392+dirty 3.19 MiB 4.26 MiB 1.08 MiB
c8578fb+dirty 3.19 MiB 4.30 MiB 1.12 MiB
208f4af+dirty 3.19 MiB 4.25 MiB 1.06 MiB
454f971+dirty 3.19 MiB 4.31 MiB 1.12 MiB
8d0daf6+dirty 3.19 MiB 4.30 MiB 1.12 MiB
b75148e+dirty 3.19 MiB 4.25 MiB 1.07 MiB
5625ce7+dirty 3.19 MiB 4.26 MiB 1.08 MiB
555070f+dirty 3.19 MiB 4.25 MiB 1.06 MiB

Previous results on branch: kw-add-app-start-crash-ios

Startup times

Revision Plain With Sentry Diff
b75af25+dirty 1223.55 ms 1211.18 ms -12.37 ms

App size

Revision Plain With Sentry Diff
b75af25+dirty 3.19 MiB 4.32 MiB 1.13 MiB

antonis
antonis reviewed Feb 25, 2025
View reviewed changes
Base automatically changed from kw-add-auto-init-tests-ios to capture-app-start-errors February 25, 2025 13:25
antonis
antonis approved these changes Feb 25, 2025
View reviewed changes
Copy link
Collaborator

@antonis antonis left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Thank you for extending the tests Krystof 🙇

@krystofwoldrich krystofwoldrich merged commit cbb85b2 into capture-app-start-errors Feb 25, 2025
69 checks passed
@krystofwoldrich krystofwoldrich deleted the kw-add-app-start-crash-ios branch February 25, 2025 16:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@antonis antonis antonis approved these changes

@lucas-zimerman lucas-zimerman Awaiting requested review from lucas-zimerman lucas-zimerman is a code owner

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@krystofwoldrich @antonis