A changelog for my blog posts

The idea to append a list of updates and errata (in the form of commit messages) to my blog posts has been bouncing around my head for a few months now. It started when I saw Evan Travers' comment to Dave Rupert's Some unsolicited blogging advice:

[…] I've been thinking a lot about how we could reflect a changelog on our blogposts, just to give us permission to adjust and post while "not totally ready".

I'm doing a really simple git log at the bottom of my static site pages for this. It's given me a lot of freedom to get posts out of my perfection clog that was keeping everything in drafts.

Evan Travers

I strongly agree with the sentiment (my drafts folder is full of unfinished, not-deemed-perfect-enough ideas). Plus, this looked like a nice project to dig into, especially now that I'm running Eleventy and feel somewhat comfortable poking around the engine. All posts are created from markdown files under version control, so it should be possible.

My plan was deceptively simple:

1. Grab a git log and transform it to JSON (so I can consume it easily with Javascript).
2. Write an Eleventy filter that checks every page's source URL for a match with my commits-as-JSON (I just learned this technique when I implemented webmentions).
3. Output a list of commits containing the subject line and a timestamp.

As all plans do, this one quickly hit a few snags when it met reality.

Git log to JSON#

So I wanted to get my commit history as JSON. At first I tried to find if I could use someone else's work. A Google search returned a few npm packages that claimed to convert a git log to JSON, but it turned out my requirements were a bit, shall we say, off the beaten path. None of them included references of the changed files which I needed to map commits to blog posts.

The good news: I learned that you can actually use git's in-built --pretty=format:<string> to format git log into JSON. There is a list of placeholders that allow you to specify which information you want to log (all explained in the official git docs) as well as ones that expand to single literal characters, like %n for newline. So you can do this:

git log -1 --pretty=format:'{%n  "commit_hash": "%H",%n  "date": "%aD",%n  "subject": "%s",%n  "author": "%aN" %n}'

and get nice, clean JSON:

{
  "commit_hash": "3fe019607382e28a39117b8fda5e436beb3afb5f",
  "date": "Tue, 3 Sep 2019 20:20:11 +0200",
  "subject": "Add changelog to blog posts",
  "author": "Søren Birkemeyer"
}

However, it turns out there is no placeholder for files. D'uh. I returned to Google and found docs for git log --name-only and git log --name-status, the latter being my saviour. It outputs a commit with a list of files and status changes. Here's the same commit from before returned by git log -1 --name-status:

Author: Søren Birkemeyer <author@mailhost.com>
Date:   Tue Sep 3 20:20:11 2019 +0200

    Add changelog to blog posts

    Display a list of commit messages (subject lines only) at the bottom of
    blog posts if the commit(s) modified the source markdown file.

M       .eleventy.js
M       .gitignore
A       _cache/gitlog.json
A       git-log-merge.sh
A       git-log-namestatus.sh
A       git-log-pretty.sh
M       gulpfile.js
M       src/_assets/scss/06_trumps/_text.scss
A       src/_data/commits.js
M       src/_includes/layouts/blogpost.njk
A       src/_includes/snippets/changelog.njk

If you look closely, you may be able to see where this is heading. When I searched for "git log --name-status to json" I struck gold with a Gist by Noah Sussman. Apparently, he had been working on the same problem and solved it with shell scripts and some Perl voodoo I only half understand – which didn't matter, because the result was exactly what I needed. 🎉

Perl in a shell#

Riffing off his work, I created three shell scripts:

git-log-pretty.sh#

collects the commit data I can get via format placeholders and writes a JSON file to a temp folder.

git log \
    --pretty=format:'{%n  ßßßhashßßß: ßßß%hßßß,%n  ßßßauthorßßß: ßßß%aN <%aE>ßßß,%n  ßßßdateßßß: ßßß%cIßßß,%n  ßßßsubjectßßß: ßßß%sßßß %n},' \
    $@ | sed 's/"/\\"/g' | sed 's/ßßß/"/g' | \
    perl -pe 'BEGIN{print "["}; END{print "]\n"}' | \
    perl -pe 's/},]/}]/' > ./_tmp/git-log.json

git-log-namestatus.sh#

outputs an array of changed files and their status for each commit into a second JSON file. The commits are made identifiable by --format='%h' which translates to the commit hash.

git log \
    --name-status \
    --format='%h' \
    $@ | \
    perl -lawne '
        if (defined $F[1]) {
            print qq# {"status": "$F[0]", "path": "$F[1]"},#
        } elsif (defined $F[0]) {
            print qq#],\n"$F[0]": [#
        };
        END{print qq#],#}' | \
    tail -n +2 | \
    perl -wpe 'BEGIN{print "{"}; END{print "}"}' | \
    tr '\n' ' ' | \
    perl -wpe 's#(]|}),\s*(]|})#$1$2#g' | \
    perl -wpe 's#,\s*?}$#}#' > ./_tmp/git-name-status.json

git-log-merge.sh#

joins the two files' contents via the unique commit hashes.

jq --slurp '.[1] as $logstat | .[0] | map(.files = $logstat[.hash])' ./_tmp/git-log.json ./_tmp/git-name-status.json > ./_cache/gitlog.json

As I said, I don't speak Perl – but through trial and error I was able to make the few cosmetic changes I needed. There is probably a much nicer way to do this without sed, or with awk or some clever Node.js streams that avoid the creation of temporary files altogether. Do you have a better solution? Tell me, or even better: blog about it!

Commits in my merged JSON log look like this now:

{
    "hash": "3fe0196",
    "author": "Søren Birkemeyer <author@mailhost.com>",
    "date": "2019-09-03T20:20:11+02:00",
    "subject": "Add changelog to blog posts",
    "files": [
      {
        "status": "M",
        "path": ".eleventy.js"
      },
      {
        "status": "M",
        "path": ".gitignore"
      },
      {
        "status": "A",
        "path": "_cache/gitlog.json"
      },
      {
        "status": "A",
        "path": "git-log-merge.sh"
      },
      {
        "status": "A",
        "path": "git-log-namestatus.sh"
      },
      {
        "status": "A",
        "path": "git-log-pretty.sh"
      },
      {
        "status": "M",
        "path": "gulpfile.js"
      },
      {
        "status": "M",
        "path": "src/_assets/scss/06_trumps/_text.scss"
      },
      {
        "status": "A",
        "path": "src/_data/commits.js"
      },
      {
        "status": "M",
        "path": "src/_includes/layouts/blogpost.njk"
      },
      {
        "status": "A",
        "path": "src/_includes/snippets/changelog.njk"
      }
    ]
  }

I already use Gulp to compile my static assets (CSS/JS) so it seemed reasonable to add a (synchronous) series of tasks to execute my shell scripts. For this first iteration I plan to run these locally and commit the generated JSON. Believe me, the irony of committing a git log is not lost on me. 🤦‍♂️ It's a clutch until I find the time to test if and how I can run the scripts during a Netlify deployment.

Let's rather look at step two: processing the changelog with Eleventy.

Filter commits by changed files and status "modified"#

I had decided to treat the changelog like my webmentions data and store it in a global _cache folder, so I needed a Javascript data file to extract it and make it available to Eleventy.

Here's my _data/commits.js which exposes the changelog as a global commits variable I can use in my templates:

const fs = require('fs');
const CACHE_DIR = '_cache';

function readFromCache() {
    const filePath = `${CACHE_DIR}/gitlog.json`;

    if (fs.existsSync(filePath)) {
        const cacheFile = fs.readFileSync(filePath);
        return JSON.parse(cacheFile);
    }
    return '{[]}';
}

module.exports = async function() {
    return readFromCache();
};

Now I needed to figure out how to get only the commits I was interested in. Eleventy has a nifty way to configure filter extensions for templates which let you manipulate data at build time. I wrote down what I wanted to achieve: "Get a list of commits for every blog post that changed the source markdown file with a status of modified." I was specifically interested in the "modified" status because I didn't want the publish commit (status "added") to appear in my changelog.

I tried a long time to write the filter with Array.prototype.filter(), but the deeply-nested array/object structure baffled me. In the end, I stuck to the tools I know and wrote two loops that check each file in each commit for a URL and status match:

// Git Commit Filter
eleventyConfig.addFilter('changelogForUrl', (commits, url) => {
    let changelog = [];

    commits.forEach(commit => {
        commit.files.forEach(file => {
            if (file.path === url && file.status === 'M') {
                changelog.push(commit);
            }
        });
    });

    return changelog;
});

Do you know how to refactor this with .filter()? Perhaps with .some() and a callback? I'd be happy to learn!

Render the changelog#

All that was left to do was to actually show the changelog at the bottom of my blog posts. Because page.inputPath returned a string starting with a ./ I constructed the url to the markdown file by hand. Here's a slimmed down version of my changelog.njk partial:

{%- set url = 'src' ~ page.url ~ 'index.md' | url -%}
{%- set changelog = commits | changelogForUrl(url) -%}

{% if changelog | length %}
<div id="changelog">
    Changelog
    <ul>
        {% for commit in changelog %}
            <li id="{{ commit.hash }}">
                {{ commit.date | dateFromTimestamp | readableDateShort }}
                {{ commit.subject }}
            </li>
        {% endfor %}
    </ul>
</div>
{% endif %}

I added the partial to my blog post template and voilà! I had my very first changelog.

So there it is. I will probably add links to the commits on Github once I am done deciding whether I want to switch the repo to public. Also, there's a lot of room for improvement; I definitely plan to revisit the shell scripts and corresponding Gulp tasks – and maybe I will also get to learn a few new tricks from replies to this blog post?

I'd love that. 👨‍💻