A changelog for my blog posts

The idea to append a list of updates and errata (in the form of commit messages) to my blog posts has been bouncing around my head for a few months now. It started when I saw Evan Travers' comment to Dave Rupert's Some unsolicited blogging advice:

[…] I've been thinking a lot about how we could reflect a changelog on our blogposts, just to give us permission to adjust and post while "not totally ready".

I'm doing a really simple git log at the bottom of my static site pages for this. It's given me a lot of freedom to get posts out of my perfection clog that was keeping everything in drafts.

Evan Travers

I strongly agree with the sentiment (my drafts folder is full of unfinished, not-deemed-perfect-enough ideas). Plus, this looked like a nice project to dig into, especially now that I'm running Eleventy and feel somewhat comfortable poking around the engine. All posts are created from markdown files under version control, so it should be possible.

My plan was deceptively simple:

  1. Grab a git log and transform it to JSON (so I can consume it easily with Javascript).
  2. Write an Eleventy filter that checks every page's source URL for a match with my commits-as-JSON (I just learned this technique when I implemented webmentions).
  3. Output a list of commits containing the subject line and a timestamp.

As all plans do, this one quickly hit a few snags when it met reality.

Git log to JSON#

So I wanted to get my commit history as JSON. At first I tried to find if I could use someone else's work. A Google search returned a few npm packages that claimed to convert a git log to JSON, but it turned out my requirements were a bit, shall we say, off the beaten path. None of them included references of the changed files which I needed to map commits to blog posts.

The good news: I learned that you can actually use git's in-built --pretty=format:<string> to format git log into JSON. There is a list of placeholders that allow you to specify which information you want to log (all explained in the official git docs) as well as ones that expand to single literal characters, like %n for newline. So you can do this:

git log -1 --pretty=format:'{%n  "commit_hash": "%H",%n  "date": "%aD",%n  "subject": "%s",%n  "author": "%aN" %n}'

and get nice, clean JSON:

{
"commit_hash": "3fe019607382e28a39117b8fda5e436beb3afb5f",
"date": "Tue, 3 Sep 2019 20:20:11 +0200",
"subject": "Add changelog to blog posts",
"author": "Søren Birkemeyer"
}

However, it turns out there is no placeholder for files. D'uh. I returned to Google and found docs for git log --name-only and git log --name-status, the latter being my saviour. It outputs a commit with a list of files and status changes. Here's the same commit from before returned by git log -1 --name-status:

Author: Søren Birkemeyer <author@mailhost.com>
Date: Tue Sep 3 20:20:11 2019 +0200

Add changelog to blog posts

Display a list of commit messages (subject lines only) at the bottom of
blog posts if the commit(s) modified the source markdown file.

M .eleventy.js
M .gitignore
A _cache/gitlog.json
A git-log-merge.sh
A git-log-namestatus.sh
A git-log-pretty.sh
M gulpfile.js
M src/_assets/scss/06_trumps/_text.scss
A src/_data/commits.js
M src/_includes/layouts/blogpost.njk
A src/_includes/snippets/changelog.njk

If you look closely, you may be able to see where this is heading. When I searched for "git log --name-status to json" I struck gold with a Gist by Noah Sussman. Apparently, he had been working on the same problem and solved it with shell scripts and some Perl voodoo I only half understand – which didn't matter, because the result was exactly what I needed. 🎉

Perl in a shell#

Riffing off his work, I created three shell scripts:

git-log-pretty.sh#

collects the commit data I can get via format placeholders and writes a JSON file to a temp folder.

Note: I decided to use the un-sanitized commit subject %s because I want my changelog to look nice. Because I sometimes use double quotes in commit messages and they would invalidate the JSON, I grabbed an escape-hack (the ßßß) from a comment by Alexandre Baizeau and hope that'll be enough.

git log \
--pretty=format:'{%n ßßßhashßßß: ßßß%hßßß,%n ßßßauthorßßß: ßßß%aN <%aE>ßßß,%n ßßßdateßßß: ßßß%cIßßß,%n ßßßsubjectßßß: ßßß%sßßß %n},' \
$@ | sed 's/"/\\"/g' | sed 's/ßßß/"/g' | \
perl -pe 'BEGIN{print "["}; END{print "]\n"}' | \
perl -pe 's/},]/}]/' > ./_tmp/git-log.json

git-log-namestatus.sh#

outputs an array of changed files and their status for each commit into a second JSON file. The commits are made identifiable by --format='%h' which translates to the commit hash.

git log \
--name-status \
--format='%h' \
$@ | \
perl -lawne '
if (defined $F[1]) {
print qq# {"status": "$F[0]", "path": "$F[1]"},#
} elsif (defined $F[0]) {
print qq#],\n"$F[0]": [#
};
END{print qq#],#}'
| \
tail -n +2 | \
perl -wpe 'BEGIN{print "{"}; END{print "}"}' | \
tr '\n' ' ' | \
perl -wpe 's#(]|}),\s*(]|})#$1$2#g' | \
perl -wpe 's#,\s*?}$#}#' > ./_tmp/git-name-status.json

git-log-merge.sh#

joins the two files' contents via the unique commit hashes.

jq --slurp '.[1] as $logstat | .[0] | map(.files = $logstat[.hash])' ./_tmp/git-log.json ./_tmp/git-name-status.json > ./_cache/gitlog.json

As I said, I don't speak Perl – but through trial and error I was able to make the few cosmetic changes I needed. There is probably a much nicer way to do this without sed, or with awk or some clever Node.js streams that avoid the creation of temporary files altogether. Do you have a better solution? Tell me, or even better: blog about it!

Commits in my merged JSON log look like this now:

{
"hash": "3fe0196",
"author": "Søren Birkemeyer <author@mailhost.com>",
"date": "2019-09-03T20:20:11+02:00",
"subject": "Add changelog to blog posts",
"files": [
{
"status": "M",
"path": ".eleventy.js"
},
{
"status": "M",
"path": ".gitignore"
},
{
"status": "A",
"path": "_cache/gitlog.json"
},
{
"status": "A",
"path": "git-log-merge.sh"
},
{
"status": "A",
"path": "git-log-namestatus.sh"
},
{
"status": "A",
"path": "git-log-pretty.sh"
},
{
"status": "M",
"path": "gulpfile.js"
},
{
"status": "M",
"path": "src/_assets/scss/06_trumps/_text.scss"
},
{
"status": "A",
"path": "src/_data/commits.js"
},
{
"status": "M",
"path": "src/_includes/layouts/blogpost.njk"
},
{
"status": "A",
"path": "src/_includes/snippets/changelog.njk"
}
]
}

I already use Gulp to compile my static assets (CSS/JS) so it seemed reasonable to add a (synchronous) series of tasks to execute my shell scripts. For this first iteration I plan to run these locally and commit the generated JSON. Believe me, the irony of committing a git log is not lost on me. 🤦‍♂️ It's a clutch until I find the time to test if and how I can run the scripts during a Netlify deployment.

Let's rather look at step two: processing the changelog with Eleventy.

Filter commits by changed files and status "modified"#

I had decided to treat the changelog like my webmentions data and store it in a global _cache folder, so I needed a Javascript data file to extract it and make it available to Eleventy.

Here's my _data/commits.js which exposes the changelog as a global commits variable I can use in my templates:

const fs = require('fs');
const CACHE_DIR = '_cache';

function readFromCache() {
const filePath = `${CACHE_DIR}/gitlog.json`;

if (fs.existsSync(filePath)) {
const cacheFile = fs.readFileSync(filePath);
return JSON.parse(cacheFile);
}
return '{[]}';
}

module.exports = async function() {
return readFromCache();
};

Now I needed to figure out how to get only the commits I was interested in. Eleventy has a nifty way to configure filter extensions for templates which let you manipulate data at build time. I wrote down what I wanted to achieve: "Get a list of commits for every blog post that changed the source markdown file with a status of modified." I was specifically interested in the "modified" status because I didn't want the publish commit (status "added") to appear in my changelog.

I tried a long time to write the filter with Array.prototype.filter(), but the deeply-nested array/object structure baffled me. In the end, I stuck to the tools I know and wrote two loops that check each file in each commit for a URL and status match:

// Git Commit Filter
eleventyConfig.addFilter('changelogForUrl', (commits, url) => {
let changelog = [];

commits.forEach(commit => {
commit.files.forEach(file => {
if (file.path === url && file.status === 'M') {
changelog.push(commit);
}
});
});

return changelog;
});

Do you know how to refactor this with .filter()? Perhaps with .some() and a callback? I'd be happy to learn!

Render the changelog#

All that was left to do was to actually show the changelog at the bottom of my blog posts. Because page.inputPath returned a string starting with a ./ I constructed the url to the markdown file by hand. Here's a slimmed down version of my changelog.njk partial:

{%- set url = 'src' ~ page.url ~ 'index.md' | url -%}
{%- set changelog = commits | changelogForUrl(url) -%}

{% if changelog | length %}
<div id="changelog">
Changelog
<ul>

{% for commit in changelog %}
<li id="{{ commit.hash }}">
{{ commit.date | dateFromTimestamp | readableDateShort }}
{{ commit.subject }}
</li>
{% endfor %}
</ul>
</div>

{% endif %}

I added the partial to my blog post template and voilà! I had my very first changelog.

A screenshot of the changelog for my blog post "On Tinkering" showing one entry from September 02, 2019: "Fix link to 11ty.io"
Fortunately I had a real correction to test with (I had forgotten to add https:// to a URL).

So there it is. I will probably add links to the commits on Github once I am done deciding whether I want to switch the repo to public. Also, there's a lot of room for improvement; I definitely plan to revisit the shell scripts and corresponding Gulp tasks – and maybe I will also get to learn a few new tricks from replies to this blog post?

I'd love that. 👨‍💻

Changelog

  • Dec 30, 2019 Highlight automated changelogs as a featured blog post
  • Oct 20, 2019 Split blog articles and bookmarks
  • Sep 05, 2019 Fix issues in code examples with Nunjucks' "raw" tag
Post a comment If you post a tweet with a link to this page, it will appear here as a webmention (updated daily).

Webmentions

  1. Max Böck Max Böck
    this is some really good stuff, thanks for writing it! btw: you can include nunjucks syntax in your code samples by wrapping the block in {% raw %}{% endraw %} 😉
40 likes
  1. liked by Chris on Code
  2. liked by Nicanor Perera
  3. liked by Nolan Franklin
  4. liked by Abraham Williams
  5. liked by Alex Carpenter
  6. liked by Eleventy
  7. liked by Juha Liikala 👾
  8. liked by Anneke Sinnema
  9. liked by dan leatherman
  10. liked by Ire Aderinokun
  11. liked by Peter Antonius
  12. liked by Ibe
  13. liked by Dr. Richie C.
  14. liked by Frederic Marx
  15. liked by jackyalcine 🔜 TwitchCon 2019
  16. liked by Wade Dominic
  17. liked by Max Böck
  18. liked by Matt Biilmann
  19. liked by Dan Ciupuliga
  20. liked by Liam Fiddler
  21. liked by John Arthur
  22. liked by Ryan Frazier
  23. liked by PSD
  24. liked by Karan Ganesan
  25. liked by Maxim
  26. liked by KimSia Sim 🇸🇬 💻 📗
  27. liked by ehaffson
  28. liked by Corinna Baldauf
  29. liked by Henry Zeitler 🌵
  30. liked by Brett Jankord
  31. liked by Lukas Grebe
  32. liked by Max Kohler
  33. liked by Sebastian Kugler
  34. liked by AJ Klein
  35. liked by Matthias 🦄
  36. liked by ross
  37. liked by Ondřej Kašpar
  38. liked by Matthias Pigulla
  39. liked by Jimmy Hugge
  40. liked by Tom Gallacher
7 reposts
  1. reposted by Eleventy
  2. reposted by Frederic Marx
  3. reposted by Matt Biilmann
  4. reposted by Henry Zeitler 🌵
  5. reposted by Sebastian Kugler
  6. reposted by ross
  7. reposted by Matthias Pigulla