git, nagios and hooks, corrupted git repo

Background

We're using nagios to monitor our infrastructure. We don't have the nagios configs under version control at the moment, and there are two of us that manage nagios configuration. As such, I'm working to get our nagios config into a central git repo, using some hooks to do syntax checking and then if the configs look good, make them "active". I'm using this guy's post as a starting point.

The general workflow I'm trying to implement is:

1. Edit local git repo of nagios config. Add edited files, commit locally.


2. git push origin master to the remote repo.


3. Push is intercepted by the pre-receive hook, which takes the files, moves them to a temporary directory on the server, and runs them through the nagios syntax checker.


4. If the syntax checker passes, accept the push, then use the post-commit hook to git pull the new code into the live nagios configuration directory and then restart nagios.


5. If the syntax checker fails, reject the push, showing the nagios syntax error to the user.

I'm running into an odd behavior, though, when I reject a git push due to syntax errors in the nagios config. What I expect to happen is that if I reject the hook, the attempted push should leave the repository just how it was, untouched. That doesn't appear to be the case, though. Below are the details of what I'm seeing:

Problem

I edit the nagios config locally, intentionally including a syntax error, add, then commit locally:

host:nagios erik$ vi nagios.cfg
host:nagios erik$ git add nagios.cfg
host:nagios erik$ git commit -m "syntax error"

[master da71aed] syntax error
 1 files changed, 1 insertions(+), 0 deletions(-)

Now I push those changes to the master repo. This will be rejected due to the syntax error:

host:nagios erik$ git push origin master
Counting objects: 5, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (2/2), done.

Writing objects: 100% (3/3), 12.74 KiB, done.
Total 3 (delta 1), reused 2 (delta 1)
remote: Previous HEAD position was 3ddc880... removed syntax error
remote: HEAD is now at da71aed... syntax error
remote: Nagios Config Check Exit Status: 254
remote: Your configs did not parse correctly, there was an error. Output follows.
remote:
remote: Nagios Core 3.2.3
remote: Copyright (c) 2009-2010 Nagios Core Development Team and Community Contributors
remote: Copyright (c) 1999-2009 Ethan Galstad

remote: Last Modified: 10-03-2010
remote: License: GPL
remote:
remote: Website: http://www.nagios.org
remote: Reading configuration data...
remote: Error in configuration file '/tmp/nagiosworkdir/nagios.cfg' - Line 23 (NULL value)
remote:    Error processing main config file!
remote:
remote:
remote:

remote: ***> One or more problems was encountered while processing the config files...
remote:
remote:      Check your configuration file(s) to ensure that they contain valid
remote:      directives and data defintions.  If you are upgrading from a previous
remote:      version of Nagios, you should be aware that some variables/definitions
remote:      may have been removed or modified in this version.  Make sure to read
remote:      the HTML documentation regarding the config files, as well as the
remote:      'Whats New' section to find out what has changed.
remote:
To git@remote-server.example.com:nagios

 ! [remote rejected] master -> master (pre-receive hook declined)
error: failed to push some refs to 'git@remote-server.example.com:nagios'

This shouldn't have touched the remote repo, but it did. If I change to another local temp directory and try to clone the repo, I get:

host:temp erik$ git clone git@remote-server.example.com:nagios
Cloning into nagios...
remote: Counting objects: 30, done.
remote: Compressing objects: 100% (29/29), done.

remote: Total 30 (delta 12), reused 0 (delta 0)
Receiving objects: 100% (30/30), 29.81 KiB, done.
Resolving deltas: 100% (12/12), done.
error: Trying to write ref HEAD with nonexistant object da71aedfde2e0469288acd9e45bb8b57a6e5a7b3
fatal: Cannot update the ref 'HEAD'.

Now I go back to the original work directory, fix the syntax error, add, commit, and push:

host:nagios erik$ vi nagios.cfg

host:nagios erik$ git add nagios.cfg
host:nagios erik$ git commit -m "removing syntax error, push should succeed this time"
[master f147ded] removing syntax error, push should succeed this time
 1 files changed, 0 insertions(+), 2 deletions(-)
host:nagios erik$ git push origin master
Counting objects: 6, done.
Delta compression using up to 8 threads.
Compressing objects: 100% (4/4), done.
Writing objects: 100% (4/4), 487 bytes, done.
Total 4 (delta 2), reused 0 (delta 0)

remote: Previous HEAD position was 4c80d45... syntax error
remote: HEAD is now at f147ded... removing syntax error, push should succeed this time
remote: Nagios Config Check Exit Status: 0
remote: Your configs look good and parsed correctly.
To git@remote-server.example.com:nagios
   3ddc880..f147ded  master -> master

At this point, the repository is fine, and I'm able to change to a temporary directory and clone the repo again:

host:temp erik$ git clone git@remote-server.example.com:nagios
Cloning into nagios...
remote: Counting objects: 34, done.
remote: Compressing objects: 100% (33/33), done.
remote: Total 34 (delta 14), reused 0 (delta 0)
Receiving objects: 100% (34/34), 30.22 KiB, done.
Resolving deltas: 100% (14/14), done.

Here is the pre-receive hook I'm using.

I'm using git v1.7.5.4 on the client, and v1.7.2.3 on the server.

So, to the question: why is the repository being left in an inconsistent state when I reject the push? Is something awry with my git hook or perhaps my understanding of git is lacking?

Answer

You're doing:

export GIT_WORK_TREE=/tmp/nagiosworkdir
/usr/bin/git checkout -f $NEW_SHA1

in your hook. Although it's not touching your usual working-copy it is updating references in the git-dir (specifically the HEAD reference), as shown in your error:

...
remote: HEAD is now at da71aed... syntax error
...

Your hook is doing exit 1 to reject the update, but it's not (re)resetting the HEAD reference after failure.

I think you need to update the failure branch in your hook like so:

...
if [ "$NAGIOS_CHECK_STATUS" -ne 0 ]
   then
   echo "Your configs did not parse correctly, there was an error. Output follows."
   cat $GIT_WORK_TREE/check.out
   /usr/bin/git reset --hard $OLD_SHA1    # <-- Add This
   exit 1
else

   ...