Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

log panic full stack #7410

Merged
merged 2 commits into from
Oct 13, 2022
Merged

log panic full stack #7410

merged 2 commits into from
Oct 13, 2022

Conversation

liguozhong
Copy link
Contributor

What this PR does / why we need it:
log panic full stack.

This pr can't fix any issue directly, but it can provide a log the next time the panic occurs, the nil pointer bug is difficult to locate without a full stack

level=info ts=2022-10-13T08:10:35.236282467Z caller=index_set.go:422 table-name=loki_index_19278 file-name=loki-write-0-1664440320185296918-1665646200.gz msg="downloaded file" total_time=127.79166ms
level=info ts=2022-10-13T08:10:35.329011798Z caller=index_set.go:422 table-name=loki_index_19278 file-name=loki-write-0-1664440320185296918-1665631800.gz msg="downloaded file" total_time=220.608618ms
level=error ts=2022-10-13T08:10:35.341914548Z caller=index_set.go:265 table-name=loki_index_19278 msg="sync failed, retrying it" err="recovered from panic opening boltdb file: runtime error: invalid memory address or nil pointer dereference"
level=error ts=2022-10-13T08:10:35.341957794Z caller=index_set.go:104 table-name=loki_index_19278 msg="failed to initialize table loki_index_19278, cleaning it up" err="recovered from panic opening boltdb file: runtime error: invalid memory address or nil pointer dereference"
level=error ts=2022-10-13T08:10:35.341970873Z caller=table.go:294 table-name=loki_index_19278 msg="failed to init user index set " err="recovered from panic opening boltdb file: runtime error: invalid memory address or nil pointer dereference"
level=error ts=2022-10-13T08:10:35.341988856Z caller=table.go:312 table-name=loki_index_19278 org_id=fake msg="index set  has some problem, cleaning it up" err="recovered from panic opening boltdb file: runtime error: invalid memory address or nil pointer dereference"
level=error ts=2022-10-13T08:10:35.384714473Z caller=table.go:314 table-name=loki_index_19278 msg="failed to cleanup broken index set " err="unlinkat /var/loki/boltdb-shipper-cache/loki_index_19278/.nfs00000000002a1

Which issue(s) this PR fixes:
Fixes #7406

Special notes for your reviewer:

Checklist

  • Reviewed the CONTRIBUTING.md guide
  • Documentation added
  • Tests updated
  • CHANGELOG.md updated
  • Changes that require user attention or interaction to upgrade are documented in docs/sources/upgrading/_index.md

@liguozhong liguozhong requested a review from a team as a code owner October 13, 2022 11:17
@grafanabot
Copy link
Collaborator

./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0%
+        distributor	0%
+            querier	0%
+ querier/queryrange	0%
+               iter	0%
+            storage	0%
+           chunkenc	0%
+              logql	0%
+               loki	0%

@grafanabot
Copy link
Collaborator

./tools/diff_coverage.sh ../loki-main/test_results.txt test_results.txt ingester,distributor,querier,querier/queryrange,iter,storage,chunkenc,logql,loki

Change in test coverage per package. Green indicates 0 or positive change, red indicates that test coverage for a package fell.

+           ingester	0%
+        distributor	0%
+            querier	0%
+ querier/queryrange	0%
+               iter	0%
+            storage	0%
+           chunkenc	0%
+              logql	0%
+               loki	0%

Copy link
Contributor

@dannykopping dannykopping left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@liguozhong you're on fire!

@dannykopping dannykopping merged commit 435c055 into grafana:main Oct 13, 2022
lxwzy pushed a commit to lxwzy/loki that referenced this pull request Nov 7, 2022
changhyuni pushed a commit to changhyuni/loki that referenced this pull request Nov 8, 2022
Abuelodelanada pushed a commit to canonical/loki that referenced this pull request Dec 1, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Cannot get logs from loki API
3 participants