In the last post, I mentioned that my Gitlab CE space was used very fast. In 4 months, I have to expand the disk space again.
The folder /var/opt/gitlab/prometheus/data/wal is more than 85GB.
There are so many files in the wal folder.
It took me two hours to figure out what happened.
The Prometheus system is the key. GitLab has built-in Prometheus. Official help.
They mentioned:
Prometheus and its exporters are on by default. Prometheus runs as the
gitlab-prometheususer and listen onhttp://localhost:9090. By default, Prometheus is only accessible from the GitLab server itself. Each exporter is automatically set up as a monitoring target for Prometheus, unless individually disabled.
I am not using Prometheus at all. So I like to disable it and remove it.
Edit /etc/gitlab/gitlab.rb
Find the first two items and uncomment them and add the last item, set them to false
prometheus_monitoring['enable'] = false sidekiq['metrics_enabled'] = false # Already set to `false` by default, but you can explicitly disable it to be sure puma['exporter_enabled'] = false
save the file, and run the following command to take effect.
sudo gitlab-ctl reconfigure
After the GitLab runs normally. Remove the whole folder.
sudo rm -rf /var/opt/gitlab/prometheus/data/wal/
Now. I saved 83GB. Wonderful.
