Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Simplify the logs for query #770

Closed
zouxiang1993 opened this issue Mar 27, 2023 · 1 comment · Fixed by #776
Closed

Simplify the logs for query #770

zouxiang1993 opened this issue Mar 27, 2023 · 1 comment · Fixed by #776
Labels
feature New feature or request

Comments

@zouxiang1993
Copy link
Contributor

Describe This Problem

Now there are too many logs in the query path:

2023-03-26 17:54:30.510 INFO [analytic_engine/src/sst/parquet/async_reader.rs:482] RecordBatchProjector dropped, path:0/14/761.sst rows:2103, cost:199ms.
2023-03-26 17:54:30.539 INFO [components/object_store/src/disk_cache.rs:150] Remove disk cache, filename:/data/ceresdb/sst_cache/0-14-686.sst-0-2484243.
2023-03-26 17:54:30.637 INFO [components/object_store/src/disk_cache.rs:150] Remove disk cache, filename:/data/ceresdb/sst_cache/0-8-969.sst-243269632-247463936.
2023-03-26 17:54:30.652 INFO [components/object_store/src/disk_cache.rs:150] Remove disk cache, filename:/data/ceresdb/sst_cache/0-8-969.sst-247463936-251658240.
2023-03-26 17:54:30.742 INFO [components/object_store/src/disk_cache.rs:150] Remove disk cache, filename:/data/ceresdb/sst_cache/0-6-949.sst-239075328-243269632.
2023-03-26 17:54:30.771 INFO [components/object_store/src/disk_cache.rs:150] Remove disk cache, filename:/data/ceresdb/sst_cache/0-9-959.sst-138412032-142606336.
2023-03-26 17:54:30.825 INFO [components/object_store/src/disk_cache.rs:150] Remove disk cache, filename:/data/ceresdb/sst_cache/0-12-997.sst-12582912-16777216.
2023-03-26 17:54:30.876 INFO [components/object_store/src/disk_cache.rs:150] Remove disk cache, filename:/data/ceresdb/sst_cache/0-12-983.sst-33554432-37748736.
2023-03-26 17:54:31.004 INFO [components/object_store/src/disk_cache.rs:150] Remove disk cache, filename:/data/ceresdb/sst_cache/0-12-514.sst-243269632-247463936.
2023-03-26 17:54:31.037 INFO [components/object_store/src/disk_cache.rs:150] Remove disk cache, filename:/data/ceresdb/sst_cache/0-9-959.sst-142606336-146800640.
2023-03-26 17:54:31.063 INFO [analytic_engine/src/sst/parquet/async_reader.rs:174] Reader fetch record batches, path:0/14/763.sst, row_groups total:4, after prune:4
2023-03-26 17:54:31.063 INFO [analytic_engine/src/sst/parquet/async_reader.rs:194] Reader fetch record batches parallelly, parallelism suggest:8, real:4, chunk_size:2
2023-03-26 17:54:31.069 INFO [components/object_store/src/disk_cache.rs:150] Remove disk cache, filename:/data/ceresdb/sst_cache/0-6-949.sst-243269632-247463936.
2023-03-26 17:54:31.084 INFO [components/object_store/src/disk_cache.rs:150] Remove disk cache, filename:/data/ceresdb/sst_cache/0-12-983.sst-37748736-41943040.
2023-03-26 17:54:31.143 INFO [components/object_store/src/disk_cache.rs:150] Remove disk cache, filename:/data/ceresdb/sst_cache/0-12-514.sst-247463936-251658240.
2023-03-26 17:54:31.248 INFO [components/object_store/src/disk_cache.rs:150] Remove disk cache, filename:/data/ceresdb/sst_cache/0-9-483.sst-549453824-553648128.
2023-03-26 17:54:31.283 INFO [analytic_engine/src/sst/parquet/async_reader.rs:482] RecordBatchProjector dropped, path:0/14/763.sst rows:8192, cost:219ms.
2023-03-26 17:54:31.286 INFO [analytic_engine/src/sst/parquet/async_reader.rs:482] RecordBatchProjector dropped, path:0/14/763.sst rows:8192, cost:222ms.
2023-03-26 17:54:31.288 INFO [analytic_engine/src/sst/parquet/async_reader.rs:482] RecordBatchProjector dropped, path:0/14/763.sst rows:8192, cost:224ms.
2023-03-26 17:54:31.291 INFO [analytic_engine/src/sst/parquet/async_reader.rs:482] RecordBatchProjector dropped, path:0/14/763.sst rows:2311, cost:227ms.
2023-03-26 17:54:31.291 INFO [components/object_store/src/disk_cache.rs:150] Remove disk cache, filename:/data/ceresdb/sst_cache/0-8-969.sst-251658240-255852544.
2023-03-26 17:54:31.355 INFO [components/object_store/src/disk_cache.rs:150] Remove disk cache, filename:/data/ceresdb/sst_cache/0-12-999.sst-0-2395290.
2023-03-26 17:54:31.396 INFO [components/object_store/src/disk_cache.rs:150] Remove disk cache, filename:/data/ceresdb/sst_cache/0-9-959.sst-146800640-150994944.
2023-03-26 17:54:31.464 INFO [components/object_store/src/disk_cache.rs:150] Remove disk cache, filename:/data/ceresdb/sst_cache/0-12-997.sst-16777216-20971520.
2023-03-26 17:54:31.502 INFO [components/object_store/src/disk_cache.rs:150] Remove disk cache, filename:/data/ceresdb/sst_cache/0-12-983.sst-41943040-46137344.
2023-03-26 17:54:31.549 INFO [components/object_store/src/disk_cache.rs:150] Remove disk cache, filename:/data/ceresdb/sst_cache/0-12-514.sst-251658240-255852544.
2023-03-26 17:54:31.564 INFO [components/object_store/src/disk_cache.rs:150] Remove disk cache, filename:/data/ceresdb/sst_cache/0-14-688.sst-0-2488352.
2023-03-26 17:54:31.648 INFO [components/object_store/src/disk_cache.rs:150] Remove disk cache, filename:/data/ceresdb/sst_cache/0-9-483.sst-553648128-557842432.
2023-03-26 17:54:31.708 INFO [components/object_store/src/disk_cache.rs:150] Remove disk cache, filename:/data/ceresdb/sst_cache/0-8-969.sst-255852544-260046848.
2023-03-26 17:54:31.887 INFO [analytic_engine/src/sst/parquet/async_reader.rs:174] Reader fetch record batches, path:0/14/765.sst, row_groups total:4, after prune:4
2023-03-26 17:54:31.887 INFO [analytic_engine/src/sst/parquet/async_reader.rs:194] Reader fetch record batches parallelly, parallelism suggest:8, real:4, chunk_size:2
2023-03-26 17:54:31.905 INFO [components/object_store/src/disk_cache.rs:150] Remove disk cache, filename:/data/ceresdb/sst_cache/0-6-949.sst-247463936-251658240.
2023-03-26 17:54:31.954 INFO [components/object_store/src/disk_cache.rs:150] Remove disk cache, filename:/data/ceresdb/sst_cache/0-12-997.sst-20971520-25165824.
2023-03-26 17:54:31.974 INFO [components/object_store/src/disk_cache.rs:150] Remove disk cache, filename:/data/ceresdb/sst_cache/0-12-983.sst-46137344-50331648.
2023-03-26 17:54:32.069 INFO [analytic_engine/src/sst/parquet/async_reader.rs:482] RecordBatchProjector dropped, path:0/14/765.sst rows:8192, cost:182ms.
2023-03-26 17:54:32.072 INFO [analytic_engine/src/sst/parquet/async_reader.rs:482] RecordBatchProjector dropped, path:0/14/765.sst rows:8192, cost:185ms.
2023-03-26 17:54:32.074 INFO [analytic_engine/src/sst/parquet/async_reader.rs:482] RecordBatchProjector dropped, path:0/14/765.sst rows:8192, cost:186ms.
2023-03-26 17:54:32.075 INFO [analytic_engine/src/sst/parquet/async_reader.rs:482] RecordBatchProjector dropped, path:0/14/765.sst rows:2031, cost:188ms.
2023-03-26 17:54:32.075 INFO [components/object_store/src/disk_cache.rs:150] Remove disk cache, filename:/data/ceresdb/sst_cache/0-9-959.sst-150994944-155189248.
2023-03-26 17:54:32.106 INFO [components/object_store/src/disk_cache.rs:150] Remove disk cache, filename:/data/ceresdb/sst_cache/0-9-483.sst-557842432-562036736.
2023-03-26 17:54:32.149 INFO [components/object_store/src/disk_cache.rs:150] Remove disk cache, filename:/data/ceresdb/sst_cache/0-6-949.sst-251658240-255852544.
2023-03-26 17:54:32.203 INFO [components/object_store/src/disk_cache.rs:150] Remove disk cache, filename:/data/ceresdb/sst_cache/0-12-983.sst-50331648-54525952.
2023-03-26 17:54:32.288 INFO [components/object_store/src/disk_cache.rs:150] Remove disk cache, filename:/data/ceresdb/sst_cache/0-12-514.sst-255852544-260046848.
2023-03-26 17:54:32.330 INFO [components/object_store/src/disk_cache.rs:150] Remove disk cache, filename:/data/ceresdb/sst_cache/0-9-959.sst-155189248-159383552.
2023-03-26 17:54:32.391 INFO [components/object_store/src/disk_cache.rs:150] Remove disk cache, filename:/data/ceresdb/sst_cache/0-8-969.sst-260046848-264241152.
2023-03-26 17:54:32.405 INFO [components/object_store/src/disk_cache.rs:150] Remove disk cache, filename:/data/ceresdb/sst_cache/0-9-483.sst-562036736-566231040.
2023-03-26 17:54:32.475 INFO [components/object_store/src/disk_cache.rs:150] Remove disk cache, filename:/data/ceresdb/sst_cache/0-6-949.sst-255852544-260046848.
2023-03-26 17:54:32.518 INFO [components/object_store/src/disk_cache.rs:150] Remove disk cache, filename:/data/ceresdb/sst_cache/0-12-997.sst-25165824-29360128.
2023-03-26 17:54:32.617 INFO [components/object_store/src/disk_cache.rs:150] Remove disk cache, filename:/data/ceresdb/sst_cache/0-12-983.sst-54525952-58720256.
2023-03-26 17:54:32.646 INFO [analytic_engine/src/sst/parquet/async_reader.rs:174] Reader fetch record batches, path:0/14/767.sst, row_groups total:4, after prune:4
2023-03-26 17:54:32.646 INFO [analytic_engine/src/sst/parquet/async_reader.rs:194] Reader fetch record batches parallelly, parallelism suggest:8, real:4, chunk_size:2
2023-03-26 17:54:32.685 INFO [components/object_store/src/disk_cache.rs:150] Remove disk cache, filename:/data/ceresdb/sst_cache/0-12-514.sst-260046848-264241152.
2023-03-26 17:54:32.752 INFO [components/object_store/src/disk_cache.rs:150] Remove disk cache, filename:/data/ceresdb/sst_cache/0-14-690.sst-0-2429693.
2023-03-26 17:54:32.845 INFO [components/object_store/src/disk_cache.rs:150] Remove disk cache, filename:/data/ceresdb/sst_cache/0-8-969.sst-264241152-268435456.
2023-03-26 17:54:32.861 INFO [analytic_engine/src/sst/parquet/async_reader.rs:482] RecordBatchProjector dropped, path:0/14/767.sst rows:8192, cost:215ms.
2023-03-26 17:54:32.863 INFO [analytic_engine/src/sst/parquet/async_reader.rs:482] RecordBatchProjector dropped, path:0/14/767.sst rows:8192, cost:217ms.
2023-03-26 17:54:32.865 INFO [analytic_engine/src/sst/parquet/async_reader.rs:482] RecordBatchProjector dropped, path:0/14/767.sst rows:8192, cost:219ms.
2023-03-26 17:54:32.868 INFO [analytic_engine/src/sst/parquet/async_reader.rs:482] RecordBatchProjector dropped, path:0/14/767.sst rows:2155, cost:221ms.
2023-03-26 17:54:32.885 INFO [components/object_store/src/disk_cache.rs:150] Remove disk cache, filename:/data/ceresdb/sst_cache/0-9-483.sst-566231040-570425344.
2023-03-26 17:54:32.968 INFO [components/object_store/src/disk_cache.rs:150] Remove disk cache, filename:/data/ceresdb/sst_cache/0-9-959.sst-159383552-163577856.
2023-03-26 17:54:32.982 INFO [components/object_store/src/disk_cache.rs:150] Remove disk cache, filename:/data/ceresdb/sst_cache/0-12-997.sst-29360128-33554432.
2023-03-26 17:54:33.060 INFO [components/object_store/src/disk_cache.rs:150] Remove disk cache, filename:/data/ceresdb/sst_cache/0-6-949.sst-260046848-264241152.
2023-03-26 17:54:33.094 INFO [components/object_store/src/disk_cache.rs:150] Remove disk cache, filename:/data/ceresdb/sst_cache/0-12-514.sst-264241152-268435456.
2023-03-26 17:54:33.140 INFO [components/object_store/src/disk_cache.rs:150] Remove disk cache, filename:/data/ceresdb/sst_cache/0-8-969.sst-268435456-272629760.
2023-03-26 17:54:33.150 INFO [components/object_store/src/disk_cache.rs:150] Remove disk cache, filename:/data/ceresdb/sst_cache/0-9-483.sst-570425344-574619648.
2023-03-26 17:54:33.214 INFO [components/object_store/src/disk_cache.rs:150] Remove disk cache, filename:/data/ceresdb/sst_cache/0-9-959.sst-163577856-167772160.
2023-03-26 17:54:33.293 INFO [components/object_store/src/disk_cache.rs:150] Remove disk cache, filename:/data/ceresdb/sst_cache/0-12-997.sst-33554432-37748736.
2023-03-26 17:54:33.427 INFO [components/object_store/src/disk_cache.rs:150] Remove disk cache, filename:/data/ceresdb/sst_cache/0-6-949.sst-264241152-268435456.
2023-03-26 17:54:33.449 INFO [analytic_engine/src/sst/parquet/async_reader.rs:174] Reader fetch record batches, path:0/14/769.sst, row_groups total:4, after prune:4
2023-03-26 17:54:33.449 INFO [analytic_engine/src/sst/parquet/async_reader.rs:194] Reader fetch record batches parallelly, parallelism suggest:8, real:4, chunk_size:2
2023-03-26 17:54:33.479 INFO [components/object_store/src/disk_cache.rs:150] Remove disk cache, filename:/data/ceresdb/sst_cache/0-12-514.sst-268435456-272629760.
2023-03-26 17:54:33.496 INFO [components/object_store/src/disk_cache.rs:150] Remove disk cache, filename:/data/ceresdb/sst_cache/0-8-969.sst-272629760-276824064.
2023-03-26 17:54:33.571 INFO [components/object_store/src/disk_cache.rs:150] Remove disk cache, filename:/data/ceresdb/sst_cache/0-12-997.sst-37748736-41943040.
2023-03-26 17:54:33.590 INFO [components/object_store/src/disk_cache.rs:150] Remove disk cache, filename:/data/ceresdb/sst_cache/0-12-983.sst-58720256-62914560.
2023-03-26 17:54:33.616 INFO [analytic_engine/src/sst/parquet/async_reader.rs:482] RecordBatchProjector dropped, path:0/14/769.sst rows:8192, cost:167ms.
2023-03-26 17:54:33.618 INFO [analytic_engine/src/sst/parquet/async_reader.rs:482] RecordBatchProjector dropped, path:0/14/769.sst rows:8192, cost:169ms.
2023-03-26 17:54:33.621 INFO [analytic_engine/src/sst/parquet/async_reader.rs:482] RecordBatchProjector dropped, path:0/14/769.sst rows:8192, cost:172ms.
2023-03-26 17:54:33.623 INFO [analytic_engine/src/sst/parquet/async_reader.rs:482] RecordBatchProjector dropped, path:0/14/769.sst rows:2037, cost:174ms.
2023-03-26 17:54:33.685 INFO [components/object_store/src/disk_cache.rs:150] Remove disk cache, filename:/data/ceresdb/sst_cache/0-9-959.sst-167772160-171966464.
2023-03-26 17:54:33.771 INFO [components/object_store/src/disk_cache.rs:150] Remove disk cache, filename:/data/ceresdb/sst_cache/0-14-692.sst-0-2463524.
2023-03-26 17:54:33.787 INFO [components/object_store/src/disk_cache.rs:150] Remove disk cache, filename:/data/ceresdb/sst_cache/0-9-483.sst-574619648-578813952.
2023-03-26 17:54:33.896 INFO [components/object_store/src/disk_cache.rs:150] Remove disk cache, filename:/data/ceresdb/sst_cache/0-6-949.sst-268435456-272629760.
2023-03-26 17:54:33.943 INFO [components/object_store/src/disk_cache.rs:150] Remove disk cache, filename:/data/ceresdb/sst_cache/0-8-969.sst-276824064-281018368.
2023-03-26 17:54:33.975 INFO [components/object_store/src/disk_cache.rs:150] Remove disk cache, filename:/data/ceresdb/sst_cache/0-12-983.sst-62914560-67108864.
2023-03-26 17:54:34.035 INFO [components/object_store/src/disk_cache.rs:150] Remove disk cache, filename:/data/ceresdb/sst_cache/0-12-514.sst-272629760-276824064.
2023-03-26 17:54:34.142 INFO [components/object_store/src/disk_cache.rs:150] Remove disk cache, filename:/data/ceresdb/sst_cache/0-9-959.sst-171966464-176160768.

Proposal

Maybe we can change the log level to DEBUG

Additional Context

None
@zouxiang1993 zouxiang1993 added the feature New feature or request label Mar 27, 2023
@ShiKaiWi
Copy link
Member

ShiKaiWi commented Mar 27, 2023 •
edited
Loading

Oops. The disk cache is disabled in most cases, so we didn't find these logs before. Would you like to submit a PR to fix it by changing its log level to DEBUG?

@zouxiang1993 zouxiang1993 mentioned this issue Mar 27, 2023
Merged
jiacai2050 pushed a commit that referenced this issue Mar 28, 2023
@zouxiang1993
fix: simplify the logs in query path (#770) (#776)
0117c23
chunshao90 pushed a commit to chunshao90/ceresdb that referenced this issue May 15, 2023
@zouxiang1993
fix: simplify the logs in query path (apache#770) (apache#776)
b8b4422
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
feature New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix: simplify the logs in query path (#770)
2 participants
@ShiKaiWi @zouxiang1993