Skip to content

Influx DB: truncate timestamps to whole seconds to save disk space #9962

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Jan 17, 2024

Conversation

Al2Klimov
Copy link
Member

@Al2Klimov Al2Klimov commented Jan 15, 2024

This reverts:

More precise timestamps need more storage, see:

fixes #9948

@cla-bot cla-bot bot added the cla/signed label Jan 15, 2024
@icinga-probot icinga-probot bot added area/influxdb Metrics to InfluxDB ref/IP labels Jan 15, 2024
@Al2Klimov Al2Klimov changed the title InfluxdbCommonWriter#SendMetric(): round timestamp to check frequency Influx DB: truncate timestamps to whole seconds to save disk space Jan 16, 2024
@Al2Klimov Al2Klimov added this to the 2.15.0 milestone Jan 16, 2024
@Al2Klimov Al2Klimov added the consider backporting Should be considered for inclusion in a bugfix release label Jan 16, 2024
@Al2Klimov Al2Klimov requested a review from yhabteab January 16, 2024 11:23
@yhabteab yhabteab added the bug Something isn't working label Jan 16, 2024
@yhabteab
Copy link
Member

yhabteab commented Jan 16, 2024

Used image:

  influxdb:
    image: influxdb:latest
    container_name: influxdb
    ports:
      - "8086:8086"
    volumes:
      - influxdb_data:/var/lib/influxdb2

Create an influxdb user and a bucket via the user interface. Then simply execute the following script:

NOTE: The timestamps are always last metric timestamp + 5m.

Second precision

#!/bin/bash

start=$(date +%s);
for i in {0..29}; do
  for j in {0..99999}; do
    printf 'filldisk,icinga=213 unit="bytes",value=%d%d %d\n' $i $j $((start + $i + $j * 300))
  done > data.txt

  curl -fsSX POST --data-binary @data.txt --header "Authorization: Token P8Rpymn1CSDKVwEBRSZ40ADLyTESPzwNEids60kdXYh_6jGpatuc0Iek_I22DETP9dd_BRPNGcEQWHU6C2s3HQ==" 'http://localhost:8086/write?db=bucketsecondprecision&precision=s'
done

Result:

root@eb1feac020da:/# du -sh /var/lib/influxdb2/engine/data/ef4fda1e37d8d95e/
8.3M    /var/lib/influxdb2/engine/data/ef4fda1e37d8d95e/

Nanosecond precision

#!/bin/bash

start=$(date +%s);
for i in {0..29}; do
  for j in {0..99999}; do
    printf 'filldisk,icinga=213 unit="bytes",value=%d%d %09d\n' $i $j $((start + $i + $j * 300))
  done > data.txt

  curl -fsSX POST --data-binary @data.txt --header "Authorization: Token P8Rpymn1CSDKVwEBRSZ40ADLyTESPzwNEids60kdXYh_6jGpatuc0Iek_I22DETP9dd_BRPNGcEQWHU6C2s3HQ==" 'http://localhost:8086/write?db=bucketnanosecondprecision&precision=ns'
done

Result:

root@eb1feac020da:/# du -sh /var/lib/influxdb2/engine/data/f39177df6a8dce01/
16M     /var/lib/influxdb2/engine/data/f39177df6a8dce01/

Conclusion

Metrics with nanosecond granularity occupy twice as much storage space as metrics with second precision.

@julianbrost julianbrost merged commit b1fe15f into master Jan 17, 2024
@julianbrost julianbrost deleted the influx-disk-9948 branch January 17, 2024 07:50
@yhabteab yhabteab mentioned this pull request Jan 17, 2024
2 tasks
@Al2Klimov Al2Klimov added backported Fix was included in a bugfix release and removed consider backporting Should be considered for inclusion in a bugfix release labels May 14, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/influxdb Metrics to InfluxDB backported Fix was included in a bugfix release bug Something isn't working cla/signed ref/IP
Projects
None yet
Development

Successfully merging this pull request may close these issues.

High InfluxDB disk usage after upgrading to 2.14.0
3 participants