Using git hooks to trigger Rundeck jobs #

2017-04-27

At work, we keep our hiera yaml files in a git repo (encrypted using the excellent hiera-eyaml backend). I got really tired of doing a git pull on the puppetmaster each time I made a change to a hiera file. So, I wanted to set up a way to pull these changes automatically each time I did a commit. The first thought I had was to just set up some keys so the git server could ssh to the puppet master and run a git pull. Instead, I decided to use Rundeck for this. I knew that, in the future, I was going to want to do much more complex things from git hooks that would necessitate the use of Rundeck jobs. So, I used this task as an experiment to see how this might work.

First of all, I created the Rundeck job I would need. This was quite simple. We have two Puppet masters right now as we’re migrating from v3 to v4. My Rundeck job runs over both of these nodes and runs a git pull in the /var/db/hiera directory where our yaml files are stored. I took note of the ID of the job as I’ll need this later. You can just copy this from the browser address bar when you’re viewing the job page. It will look something like 12345678-abcd-abcd-abcd-012345678901.

Next, we need to generate an API token in Rundeck as we will be making a few calls to the API from our git hook. In a work setting, I find it’s best to use a non-human user to generate the token. This means that things don’t mysteriously break in the future when you leave the company. It’s also good to create one user for each service which will be making requests to the Rundeck API. In this case, I’m accessing the Rundeck API from our Gitlab instance, so I have a user in Rundeck called gitlab. Log into Rundeck as this user. Open up the users' profile by clicking on profile in the drop-down in the top right corner of the menu. Create a token by clicking Generate New Token. Copy the token as you’ll need it later.

Now it’s time to write the git hook. I want it to:

Run the rundeck job I created above
Wait for this job to complete
Print the log output from the job

Step 1 : Running the job #

Looking at the API docs, the API endpoint we want is an HTTP POST to /api/17/job/[ID]/run. We also need to specify the token we generated earlier. There are a couple of ways to do this. The way I chose is to specify it using the X-Rundeck-Auth-Token header. I find json relatively easy to deal with in bash scripts (at least compared to XML) thanks to jq. I also set the Accept header in the request to tell the API I want to use json. The full command to run the job is:

/usr/local/bin/curl -s \
  -H "X-Rundeck-Auth-Token:mytokengoeshereblahblah" \
  -H "Accept:application/json" \
  -XPOST "https://my.rundeck.server/api/17/job/12345678-abcd-abcd-abcd-012345678901/run"

This will return json that looks something like this.

{
  "id": 1234,
  "href": "https://my.rundeck.server/api/17/execution/1234",
  "permalink": "https://my.rundeck.server/project/Ops/execution/show/1234",
  "status": "running",
  "project": "Ops",
  "user": "gitlab",
  "date-started": {
    "unixtime": 1493312378762,
    "date": "2017-04-27T16:59:38Z"
  },
  "job": {
    "id": "12345678-abcd-abcd-abcd-012345678901",
    "averageDuration": 6267,
    "name": "pull-hiera",
    "group": "Puppet/hiera",
    "project": "Ops",
    "description": "Run a git pull on the puppetmasters in the hiera dir.",
    "href": "https://my.rundeck.server/api/17/job/12345678-abcd-abcd-abcd-012345678901",
    "permalink": "https://my.rundeck.server/project/Ops/job/show/12345678-abcd-abcd-abcd-012345678901"
  },
  "description": "#!/usr/bin/env bash\r\n\r\necho \"Updating /var/db/hiera on @node.name@\"\r\ncd /var/db/hiera\r\nsudo git pull",
  "argstring": null
}

The important part in this response is the job id. We need this in the next step. I use the really handy jq tool to parse this json to pull out the id. The modified version of the command, which only returns the job id, looks like this:

/usr/local/bin/curl -s \
  -H "X-Rundeck-Auth-Token:mytokengoeshereblahblah" \
  -H "Accept:application/json" \
  -XPOST "https://my.rundeck.server/api/17/job/12345678-abcd-abcd-abcd-012345678901/run" \
  | jq -r '.id'

Step 2 : Polling until the job is complete #

Now the job is running. I want to wait until the job is complete so I can tell that it actually worked. To do this, we need to poll the Rundeck server, checking the job status until it’s finished.

The name for an instance of a job in Rundeck terminology is an execution. The API endpoint to check the status of an execution is an HTTP GET to /API/17/execution/[ID]. Here is the full command to check the status of an execution. Note, we’re using the ID (1234) from the previous step.

/usr/local/bin/curl -s \
  -H "X-Rundeck-Auth-Token:mytokengoeshereblahblah" \
  -H "Accept:application/json" \
  -XGET "https://my.rundeck.server/api/17/execution/1234"

The output looks something like this.

{
  "id": 1234,
  "href": "https://my.rundeck.server/api/17/execution/1234",
  "permalink": "https://my.rundeck.server/project/Ops/execution/show/1234",
  "status": "running",
  "project": "Ops",
  "user": "gitlab",
  "date-started": {
    "unixtime": 1493313598645,
    "date": "2017-04-27T17:19:58Z"
  },
  "job": {
    "id": "12345678-abcd-abcd-abcd-012345678901",
    "averageDuration": 6147,
    "name": "pull-hiera",
    "group": "Puppet/hiera",
    "project": "Ops",
    "description": "Run a git pull on the puppetmasters in the hiera dir.",
    "href": "https://my.rundeck.server/api/17/job/12345678-abcd-abcd-abcd-012345678901",
    "permalink": "https://my.rundeck.server/project/Ops/job/show/12345678-abcd-abcd-abcd-012345678901"
  },
  "description": "#!/usr/bin/env bash\r\n\r\necho \"Updating /var/db/hiera on @node.name@\"\r\ncd /var/db/hiera\r\nsudo git pull",
  "argstring": null
}

We can see that the status is running in this response. We need to keep polling the API, every few seconds, with this request until the job has finished running.

Again, we can use jq to pull out the value of status from the responses by piping the output to jq -r '.status'.

Step 3 : Fetch the execution log #

Now that the job is finished, we want to see what the log output from the job was. The execution output API endpoint will tell us this. This is a GET request to /api/17/execution/[ID]/output.

/usr/local/bin/curl -s \
  -H "X-Rundeck-Auth-Token:mytokengoeshereblahblah" \
  -H "Accept:application/json" \
  -XGET "https://rundeck.afilias.tech/api/17/execution/1234/output"

This is what a response might look like.

{
  "id": "1234",
  "offset": "1320",
  "completed": true,
  "execCompleted": true,
  "hasFailedNodes": false,
  "execState": "succeeded",
  "lastModified": "1493313605000",
  "execDuration": 6409,
  "percentLoaded": 99.5475113122172,
  "totalSize": 1326,
  "entries": [
    {
      "time": "17:19:59",
      "absolute_time": "2017-04-27T17:19:59Z",
      "log": "Updating /var/db/hiera on puppet1.example.com",
      "level": "NORMAL",
      "user": "rdeck",
      "command": null,
      "stepctx": "1",
      "node": "puppet1.example.com"
    },
    {
      "time": "17:20:00",
      "absolute_time": "2017-04-27T17:20:00Z",
      "log": "Already up-to-date.",
      "level": "NORMAL",
      "user": "rdeck",
      "command": null,
      "stepctx": "1",
      "node": "puppet1.example.com"
    },
    {
      "time": "17:20:02",
      "absolute_time": "2017-04-27T17:20:02Z",
      "log": "Updating /var/db/hiera on puppet2.example.com",
      "level": "NORMAL",
      "user": "rdeck",
      "command": null,
      "stepctx": "1",
      "node": "puppet2.example.com"
    },
    {
      "time": "17:20:04",
      "absolute_time": "2017-04-27T17:20:04Z",
      "log": "Already up-to-date.",
      "level": "NORMAL",
      "user": "rdeck",
      "command": null,
      "stepctx": "1",
      "node": "puppet2.example.com"
    }
  ]
}

Parsing the output is a little more complicated this time. There is a list called entries in the response. We want to pull out just log for each of the entries in this list. The jq command to do this is jq -r '.entries[].log.

Lets put this all together in one final script.

#!/usr/bin/env bash

JOB='12345678-abcd-abcd-abcd-012345678901'
TOKEN='mytokengoeshereblahblah'
RDECK='my.rundeck.server'

echo "Running Puppet/hiera/pull-hiera rundeck job [${TOKEN}]..."

# Run the rundeck job
execution_id=$(/usr/local/bin/curl -s \
    -H "X-Rundeck-Auth-Token:${TOKEN}" \
    -H "Accept:application/json" \
    -XPOST "https://${RDECK}/api/17/job/${JOB}/run" | jq -r '.id')

echo -n "Waiting for job ${execution_id} to complete"

# Wait until the job finishes
STATUS=$(/usr/local/bin/curl -s \
    -H "X-Rundeck-Auth-Token:${TOKEN}" \
    -H "Accept:application/json" \
    -XGET "https://${RDECK}/api/17/execution/${execution_id}" \
    | jq -r '.status')

until [ $STATUS != "running" ]; do
    STATUS=$(/usr/local/bin/curl -s \
        -H "X-Rundeck-Auth-Token:${TOKEN}" \
        -H "Accept:application/json" \
        -XGET "https://${RDECK}/api/17/execution/${execution_id}" \
        | jq -r '.status')
    echo -n "."
    sleep 3
done

echo ${STATUS}

if [[ $STATUS != "succeeded" ]]; then
    echo "WARNING : the rundeck job seems to have failed."
fi

# Get output
RESULT=$(/usr/local/bin/curl -s \
    -H "X-Rundeck-Auth-Token:${TOKEN}" \
    -H "Accept:application/json" \
    -XGET "https://${RDECK}/api/17/execution/${execution_id}/output" \
    | jq -r '.entries[].log')

echo "Rundeck log output was ... "
echo "<~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~->"
echo "${RESULT}"
echo "<~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~->"

Finally, we need to put the git hook in place. In my case, we’re using Gitlab. In Gitlab the git hooks live in /usr/home/git/repositories/PROJECT/REPO.git/custom_hooks/. We want the hook to run after the commit is complete, so we create a file called post-receive. The contents of this file are the script above.

Lets do a commit to check that everything works.

chrskly@laptop ~/example % git commit -m "test" test.yaml && git push
[master 36b2ed7] test
 1 file changed, 1 deletion(-)
Counting objects: 3, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 284 bytes | 0 bytes/s, done.
Total 3 (delta 2), reused 0 (delta 0)
remote:
remote: Running Puppet/hiera/pull-hiera rundeck job [12345678-abcd-abcd-abcd-012345678901]...
remote: Waiting for job 1234 to complete...succeeded
remote: Rundeck log output was ...
remote: <~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~->
remote: Updating /var/db/hiera on puppet1.example.com
remote: From gitlab.example.com:project/test
remote:    b669a34..36b2ed7  master    -> origin/master
remote: Already up-to-date.
remote: Updating /var/db/hiera on puppet2.example.com
remote: From gitlab.example.com:project/test
remote:    b669a34..36b2ed7  master    -> origin/master
remote: Updating b669a34..36b2ed7
remote: Fast-forward
remote: <~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~-~->
To gitlab.example.com:project/test.git
   0480da3..36b2ed7  master -> master

And that’s it. Following this pattern you can trigger Rundeck jobs from a git hook.