git-cache-proxy: Periodically run `git gc --quiet'

git has a feature which is supposed to automatically run git gc.

But if your repository accumulates enough loose objects, then this
occurs:

 $ git gc --auto
 Auto packing the repository in background for optimum performance.
 See "git help gc" for manual housekeeping.
 error: The last gc run reported the following. Please correct the root cause
 and remove gc.log.
 Automatic cleanup will not be performed until the file is removed.

 warning: There are too many unreachable loose objects; run 'git prune' to remove them.

Removing $GIT_DIR/gc.log generally simply causes git gc --auto to
print, again, the message:

 warning: There are too many unreachable loose objects; run 'git prune' to remove them.

Ie, a repository can get into a state where it needs gc so badly that
git will not gc it automatically.  This is, of course, mad.  It seems
to have been a misguided attempt at a safety catch.  Unfortunately
there does not even appear to be a configuration knob to set this `too
many' limit to infinity.

Work around this problem by running `git gc --quiet' (every 10 days,
by default).  This is not ideal because it runs even if we haven't
updated the tree, but we would rather do it out of housekeeping rather
than in the middle of fetch/clone.

`git gc' without --auto does not refuse to prune the objects that need
pruning, and fixes the repository.

We leave the gc.auto config option enabled since I think it is largely
harmless, and disabling it would be extra work.

Signed-off-by: Ian Jackson <ian.jackson@eu.citrix.com>

diff --git a/scripts/git-cache-proxy b/scripts/git-cache-proxy
index d798f2bab30fbbdfc4a940bc5d7882db8c5963ab..7dd6f3ec531d53553633905a2c491bd8ffb43645 100755 (executable)
--- a/scripts/git-cache-proxy
+++ b/scripts/git-cache-proxy
@@ -64,6 +64,7 @@ our $us = 'git-cache-proxy';
 
 our $debug = 0;
 our $housekeepingeverydays = 1;
+our $gcintervaldays = 10;
 our $treeexpiredays = 21;
 our $fetchtimeout = 3600;
 our $maxfetchtimeout = 7200;
@@ -176,6 +177,7 @@ for (;;) {
                       | serve-timeout
                       | tree-expire-days
                       | housekeeping-interval-days
+                      | gc-interval-days
                       )=(\d+)$//x) {
            my $vn = $1;
            $vn =~ y/-//d;
@@ -298,6 +300,13 @@ sub readcommand () {
     servinfo "locking";
 }
 
+sub update_gcstamp ($) {
+    my ($gitdir) = (@_);
+    my $gcdone = "$gitdir/cache-proxy-gc.stamp";
+    open GCSTAMP, '>', $gcdone or fail "create $gcdone: $!";
+    close GCSTAMP;
+}
+
 sub clonefetch () {
     lockfile \*LOCK, $lock, LOCK_EX;
 
@@ -366,6 +375,7 @@ sub clonefetch () {
         alarm 0;
 
        if (!$exists) {
+           update_gcstamp($tmpd);
            rename $tmpd, $gitd or fail "rename fresh $tmpd to $gitd: $!";
            $exists = 1;
        }
@@ -396,14 +406,42 @@ sub housekeeping () {
     logm 'info', "housekeeping started";
     foreach $lock (<[a-z]*\\.lock>) {
        my $subdir = $lock;  $subdir =~ s/\\.lock$//;
+       my $gcdone = "$subdir\\.git/cache-proxy-gc.stamp";
        if (!lstat $lock) {
            $! == ENOENT or hkfail "$lock: lstat: $!";
            next;
        }
        my ($mode_what,$mode_locknb,$mode_action);
        if (-M _ <= $treeexpiredays) {
-           logm 'debug', "housekeeping: subdirs $subdir: touched recently";
-           next;
+           if (!lstat "$gcdone") {
+               $! == ENOENT or hkfail "$gcdone: lstat: $!";
+               logm 'debug',
+ "housekeeping: subdirs $subdir: touched recently, never gc'd!";
+           } elsif (-M _ <= $gcintervaldays) {
+               logm 'debug',
+ "housekeeping: subdirs $subdir: touched recently, gc'd recently";
+               next;
+           } else {
+               logm 'debug',
+ "housekeeping: subdirs $subdir: touched recently, needs gc";
+           }
+           $mode_what = 'garbage collecting';
+           $mode_locknb = 0;
+           $mode_action = sub {
+               my $gclog = "$subdir/gc.log";
+               unlink $gclog or $!==ENOENT or hkfail "remove $gclog: $!";
+               my $r = system qw(sh -ec),
+                   'cd "$1"; shift; exec "$@"', 'x', "$subdir\\.git",
+                   qw(git gc --quiet);
+               if ($r) {
+                   logm 'err',
+                       "housekeeping: subdirs $subdir: gc failed (status $r)";
+               } else {
+                   update_gcstamp("$subdir\\.git");
+                   logm 'debug',
+                       "housekeeping: subdirs $subdir: gc done";
+               }
+           };
        } else {
            $mode_what = 'cleaning';
            $mode_locknb = LOCK_NB;