Monday, August 5, 2019

On modern software

Today I noticed that gfdb updates were broken again. It was the third time it happened, so I knew this deserved a post.

I caught first update breakage purely by accident. One day I looked at the process list and noticed a chain of git commands sitting there doing nothing. Apparently at some point git decided it was time to do garbage collection, so it ran "git gc --auto", which, for reasons yet unknown to me, could not complete in 50 minutes on a fairly small and completely linear repository. This step should have happened before the push took place, but since it just hung there (with no cpu/disk activity, by the way) it was killed by task scheduler every hour to start new task, which repeated everything. This was going on for 5 days before I noticed. I run "git gc" manually and it finished in about a minute, cleared the clog and all was good (for a while).