With so much information out there today, having a system to process and manage them is crucial to not be overwhelmed. For me this involves having a good digital information capture and retrieval workflow, a.k.a note taking. I classify notes into three categories: micro text notes (<500 characters), notes consisting primarily text (including code), and notes that have embedded media like images. Micro text covers consists of fleeting thoughts and requires almost immediate offloading from the mind otherwise it will be forgotten. The latter two requires deliberate effort to plan and give structure to the content. Some software are better suited for one category than others, but the best will depend on one’s workflow. Last weekend, I reviewed my workflow and here are my reflections on all the software I’ve tried and used over the last decade in chronological order. Each section includes a verdict aimed at the reader thinking of using the software. Hopefully it is useful to curious readers looking for note taking software options.
Tiddlywiki classic
My first knowledge management software that I used a little more than a decade ago was Tiddlywiki classic, a single page HTML wiki software. For a single HTML page, it is surprisingly good: it supported multimedia notes and has a search functionality built-in. Back then, I enjoyed solving math and logic puzzles and used Tiddlywiki to collect and curate interesting puzzles I encountered.
Tiddlywiki classic was great as a offline wiki system and was straightforward to put online for reading and information retrieval. However, making changes to existing entries on the website requires a bit more effort and can be sluggish when the wiki gets large because it needs to upload the whole file for every change, no matter how small.
The community came up with a solution for this by developing phone apps, e.g. this android app, essentially wrapping the single file in a local server to allow for editing on the go, along with a performant way of saving large wiki files. It was a cute idea, but I found that editing on the small screen was not ideal back then.
My verdict: It is great as an offline text wiki for one user. If there are many notes with media, or if saving over the internet is a requirement, it can get a bit sluggish and awkward over time.
ZimWiki
At university, I decided to invest in a digital knowledge management system for taking lecture notes, and tried several software to achieve this. I considered using Tiddlywiki but it had a few rough edges which made me look around for other options. I tried RedNote, CherryTree, MediaWiki, Evernote, OneNote, but they all felt clumsy, inflexible and didn’t really gel with me. Except one, ZimWiki. I like it because it is simple, the data is portable - notes are just text files - and it is open source, which meant that I could write extensions to it to scratch my own itch.
That said, the main selling point to me was the ease of adding media, such as screen shots, and latex equations to notes entries which made taking study notes pleasant. Adding pdf attachments made it easy to link to lecture notes, and I used that feature quite extensively.
The plain text format makes it great for reading on the go too because most system can read text files without additional software. Using file syncing services like Dropbox allowed me to access my notes on the phone, and I could even edit them too. ZimWiki also has a export to HTML solution for turning the collection into a website for read only access but I didn’t explore automating so it didn’t become my workflow.
My verdict: This is perfect for creating notes with many media attachment. There is also journalling support out of the box which makes it a viable candidate for micro notes. The plain text format makes it portable for other systems. However, there is no easy way to write notes on the go.
Tiddlywiki as a commonplace book
A few years later, Tiddlywiki received an upgrade and it looked shiny. At that time, I was also very interested in digital archiving, so I explored using this new version as a commonplace book, essentially a scrapbook for ideas and knowledge.
The upgraded version retained the philosophy of being a single page wiki which was perfect for hosting it as a website. It also addressed the awkward and sluggish saving problem by having a new server mode which meant that changes to existing entries only affected that single entry, saving unnecessary data transfer over the internet on change.
My main gripe with it was adding attachments was a bit awkward when compared to ZimWiki. Adding an image for example would increase the single HTML size considerably and that wasn’t scalable for the amount of images I was adding for study notes.
My verdict: The new server mode makes it ideal for editing on the go, and definitely makes it more scalable, but it doesn’t support distributed local syncing. So if you happen to have a thought to capture but there is no internet, you will be out of luck.
Using static site generators
Around the same time, static site generators (SSG) also became quite trendy. SSG software like Jekyll made it easy to turn text files into websites. This wasn’t something new given that my existing solutions could also make notes available online, but the idea that all I needed were text files instead of special software got me started thinking whether I could simplify the software stack.
I explored turning my note collection into a Jekyll website, but there was one problem: the format of the notes were in markdown, and my existing notes were in wiki format used by Tiddlywiki or ZimWiki. Given I was so heavily invested in ZimWiki and Tiddlywiki at that time, I abandoned the migration.
Nonetheless, the idea of data portability in the form of text files resonated strongly with me. Markdown, a popular plain text format, caught my attention as a portable data format. I loved the idea that I could lightly annotate text files to give it structure for machine parsing while still keeping it readable (unlike HTML and XML formats). After looking around, I saw that many sites I used also supported markdown format. I also loved the decoupling of the text format from the presentation layer; the same text file can be fed into different renderers and the final output can be styled to taste. See Github’s renderer back then.
All these convinced me that migrating to markdown was worth the effort and I started writing new notes in markdown.
My verdict: using SSG for taking micro notes is a bad idea. It can be great for longer text notes though. For writing notes with media attachment, the flow is as good as the editors out there, but chances are you’ll have to deal with the SSG’s syntax for attaching media.
Using markdown for notes
After switching to markdown, I realized I was writing way less. I attributed that to the chore of creating notes and the burden of managing and reading notes.
Writing in markdown reduced my productivity when I needed to add media to my notes. With wiki software, the software takes care of embedding pictures and equations for me, but with markdown files, I had to work with pretty basic text editors (I was using Sublime Text with some markdown plugin) and manually add links to media files using markdown syntax.
Without any software, it was also difficult to have a bird’s eye view of my universe of notes or an easy way to read the contents with media. I’d much rather see a graph image instead of . I considered rendering them using Jekyll but the effort it takes to start a Jekyll post for a note was simply too much. For example, each entry prompts for metadata like tags, description, etc. I was worried about tagging my notes wrongly, so I ended up not writing it at all.
---
layout: post
title: A random note
tags: oh dear
categories: panic
---
I explored markdown friendly solutions to manage and create notes, such as Laverna, a self hosted Evernote replacement and it worked for a while until I forgotten to save the data and cleared my browser’s storage and all my notes got deleted. I got a bit frustrated and gave up.
Because of these obstacles, I didn’t fully migrate over and still kept my ZimWiki workflow for the ease of attaching files to the note entry.
My verdict: Using the markdown format is a huge win for data portability. However, finding an editor system to create, view and manage the files that scratch your itch is a hit and miss.
Mixing notes and blog posts with Hugo
When I graduated and started my first job programming professionally with Golang, I found out about Hugo. Hugo is a SSG written in Golang, and is known for its performance. I tried it and loved it immediately. It felt like Jekyll on steroids so I decided to attempt note taking with Hugo, with the vision that I could easily turn notes into blog post this way. That was also when I started this blog.
However, like Jekyll, I eventually found myself writing fewer notes because of high inertia and decision paralysis when starting a new note. I thought using some content management software (CMS) on top of Hugo (like Hokus CMS) could improve the workflow but it didn’t really work for me.
I still like Hugo, and still use it to update this blog, but I use it mainly for synthesizing blog posts I think are worth sharing instead of using it as the main notes taking software. Using hugo as a knowledge management tool was a little like forcing a square peg into a round hole. I’d probably wouldn’t use it that way anymore.
My verdict: SSG is a growing trend, but I don’t think it should be used for (nor was it meant for) taking micro notes. It can be great for turning the notes collection into a website though. A good markdown editor is recommended for that.
Using third party services
At this point I was quite exhausted with finding the perfect text file workflow for capturing fleeting thoughts and notes, and settled for whatever was convenient. I turned to the plethora of mobile note taking apps and services. Google’s “Keep” was my go-to choice for a while. I realized that after graduation, most of my notes were micro notes, and I didn’t really require the full feature set of markdown. By switching to third party services, I lost control over the notes as they were no longer text files I control, but I was definitely jotting down more of my thoughts. Alas, that was the price for convenience.
My verdict: If you don’t mind losing a bit of control of your notes and data, and are not too worried about data portability or whether the service will be there for the long run, then by all means use it for its convenience. Personally, I’d recommend only using it to take micro notes and feed it to your own notes collection for processing.
Self-hosted Nextcloud
About a year ago, with an increasing focus on data privacy, I decided to fork out some money and opt for a self-hosted version of NextCloud software suite, which comes with a notes app, and migrated to that for writing micro notes (in markdown). The best thing is I now regain control over my micro notes and could easily turn them into longer markdown notes if desired.
At the same time, I also went looking for markdown editors, and thanks to the explosion of people developing electron apps, there were so many to choose from.
This was really when my investment in writing notes as markdown files paid off. Most editors make it easy to attach media files, effectively matching up to the one thing that ZimWiki does well. A good number of them also tries to be a knowledge management solution, such as allowing notes to be tagged and searched.
My verdict: I believe that investing in a portable data format and owning your own data is the way to go. The open source community is amazing and can only continue to grow. If you’ve invested enough time to know what workflow works best for you, it might be a good time to look at self hosted solution to scratch your own itch.
My workflow today
Today, I primarily write markdown notes and my text editors of choice are Boostnote, Zettlr (for trying out Zettelkasten system), and VSCode. All of them happen to be Electron apps 😉. For note taking in particular, I use the first two because they have very low barrier to start writing and have built-in renderer to give instant visual feedback. There are several close runner ups for open source markdown editors that are worth giving a shout out to here: vnote, notable and QOwnNotes.
While away from my computer, I write my thoughts and notes on Android using Nextcloud notes app and Markor, using Nextcloud to sync them to my desktop for further processing.
Most of the time, these notes are micro notes and plain text format usually suffice. What’s more important in this flow is making sure they flow into the same central pool of notes and are searchable. On my Windows computer, I have a AHK script that brings up a prompt for quick nano note taking. I have a daily reminder in the evening to review notes I’ve taken.
I hope this has been useful and thanks for reading. Let me know if you have comments or tips to share!