How Prometheus data is being stored and How to change the retention period of this data

How Prometheus data is being stored and How to change the retention period of this data

Prometheus includes a local on-disk time series database. Prometheus's local time series database stores time series data in a custom format on disk.

Ingested samples are grouped into blocks of two hours. Each two-hour block consists of a directory containing one or more chunk files that contain all time series samples for that window of time, as well as a metadata file and index file (which indexes metric names and labels to time series in the chunk files). When series are deleted via the API, deletion records are stored in separate tombstone files (instead of deleting the data immediately from the chunk files). 

The block for currently incoming samples is kept in memory and not fully persisted yet. It is secured against crashes by a write-ahead-log (WAL) that can be replayed when the Prometheus server restarts after a crash. Write-ahead log files are stored in the wal directory in 128MB segments. These files contain raw data that has not been compacted yet, so they are significantly larger than regular block files. Prometheus will keep a minimum of 3 write-ahead log files, however high-traffic servers may see more than three WAL files since it needs to keep at least two hours worth of raw data.

The directory structure of a Prometheus server's data directory will look something like this:

[root@promserver data]# ls
01DN4C1BBMHCDXP9HFS7QW49HA  01DNFYTK2QX7P4JTGNHPSVHGSJ 01DNC36YS7TXS4E9SN3KXVWPAR  01DNQP0546HDVEQEJFEKYPJAXE  01DP38TCFSM0XCY5BQRA0HQRVF  wal
01DNE1102QD63R448BK7D0Z4WB  01DNSKSK8ZZTNAZ8ZQ5Y30NTDH  01DP56K6K40190T0C255CTG74D
[root@promserver data]# ls -latr 01DN4C1BBMHCDXP9HFS7QW49HA
total 73788
drwxr-xr-x  2 nfsnobody nfsnobody       34 Sep 19 04:01 chunks
-rw-r--r--  1 nfsnobody nfsnobody 75542704 Sep 19 04:02 index
-rw-r--r--  1 nfsnobody nfsnobody        9 Sep 19 04:02 tombstones
-rw-r--r--  1 nfsnobody nfsnobody      932 Sep 22 22:03 meta.json
drwxr-xr-x  3 nfsnobody nfsnobody       68 Sep 22 22:03 .
drwxr-xr-x 25 nfsnobody nfsnobody     4096 Oct  3 12:00 ..
[root@promserver data]# file 01DN4C1BBMHCDXP9HFS7QW49HA/index
01DN4C1BBMHCDXP9HFS7QW49HA/index: data
[root@promserver data]# ls -la 01DN4C1BBMHCDXP9HFS7QW49HA/chunks/
total 903560
drwxr-xr-x 2 nfsnobody nfsnobody        34 Sep 19 04:01 .
drwxr-xr-x 3 nfsnobody nfsnobody        68 Sep 22 22:03 ..
-rw-r--r-- 1 nfsnobody nfsnobody 536869976 Sep 19 04:01 000001
-rw-r--r-- 1 nfsnobody nfsnobody 388371978 Sep 19 04:02 000002
[root@promserver data]# file 01DN4C1BBMHCDXP9HFS7QW49HA/tombstones
01DN4C1BBMHCDXP9HFS7QW49HA/tombstones: data
[root@promserver data]# ls -latr wal/
total 3667816
-rw-r--r--  1 nfsnobody nfsnobody 134119424 Oct  3 08:29 00008604
-rw-r--r--  1 nfsnobody nfsnobody 134217728 Oct  3 08:40 00008605
-rw-r--r--  1 nfsnobody nfsnobody 134119424 Oct  3 08:51 00008606
-rw-r--r--  1 nfsnobody nfsnobody 134119424 Oct  3 09:02 00008607
-rw-r--r--  1 nfsnobody nfsnobody 134119424 Oct  3 09:13 00008608
-rw-r--r--  1 nfsnobody nfsnobody 134119424 Oct  3 09:24 00008609
drwxr-xr-x 25 nfsnobody nfsnobody      4096 Oct  3 12:00 ..
drwxr-xr-x  2 nfsnobody nfsnobody        22 Oct  3 12:00 checkpoint.008603
-rw-r--r--  1 nfsnobody nfsnobody  75327131 Oct  3 13:21 00008631
[root@promserver data]#

Data Retention :

Prometheus does the retention in 2 ways. 
1. Time based retention
2. Size based retention


1. Time based retention
You need to start prometheus with following flag with your desired time.
--storage.tsdb.retention.time: This determines when to remove old data. Defaults to 15d. 

2. Size based retention
You need to start prometheus with following flag with your desired size.
--storage.tsdb.retention.size:  This determines the maximum number of bytes that storage blocks can use (note that this does not include the WAL size, which can be substantial). The oldest data will be removed first. Defaults to 0 or disabled. Following Units are supported: KB, MB, GB, PB. for Ex: "512MB"