In Pursuit of an Efficient Org-Agenda
Table of Contents
After beginning my Emacs and Org-Mode adventures I quickly realized the insufficiency of the standard agenda. It is wonderful at producing a list, but I am quickly approaching over 1000 notes. The agenda tooling is incapable of searching this many files efficiently, and it took over thirty seconds to generate an agenda.
This was intolerable. I had two options:
- Place all my TODO items into a single file.
- Narrow the number of files the agenda mechanism needs to search
I find the first option undesireable for reasons I mentioned in my post about zettelkasten tools. I like to have my todo items mixed with the context where they were born. A student of the Getting Things Done methodology might ask, "Doesn't this violate the central todo-list principle?" Yes, it does. Org-agenda allows me to have my cake and eat it too. I can create a centralized todo-list out of all my todo items, and then immediately jump into the context of my next task.
It took several iterations of configuration to reach a seamless workflow. If you'd like to jump straight to the final setup, here's the link.
1. First Iteration
The first piece of tooling I used came from this post and this Gist.
Essentially it creates a function vulpea-project-files that can easily query for a tag, and then sets the org-agenda files to all the files containing this tag.
Coupled with a helper-function that adds the tag hastodos to any file containing a TODO entry, it functioned well.
(defun vulpea-project-files () "Return a list of note files containing 'hastodos' tag." ; (seq-uniq (seq-map #'car (org-roam-db-query [:select [nodes:file] :from tags :left-join nodes :on (= tags:node-id nodes:id) :where (like tag (quote "%\"hastodos\"%"))])))) (setq org-agenda-files (vulpea-project-files))
This solution is great because it is pure elisp.
Anywhere you're running Emacs it should function just fine.
It also works well with one of my needs: transparent file encryption.
Emacs has extensions for both gpg and age encryption that allow files to be transparently encrypted and decrypted.
This solution can make use of that where a standard grep could not.
2. Second Iteration
Eventually, I grew tired of seeing the entire solution in my config file. All told, it is about 170 lines. This is a lot for a small utility!
I decided to try using ripgrep to fix this:
(defun set-org-agenda-files-ripgrep () (setq org-agenda-files (split-string (shell-command-to-string "rg -torg -l TODO /home/user/org"))))
This worked nicely. It takes just a few lines, runs just as fast–probably faster–than pure elisp, and is much easier to read and understand.
The only difficulty is encryption.
ripgrep does not operate on open emacs buffers, instead it operates on what is saved to disk.
Naturally, this means that it only sees the encrypted files.
3. Final
The solution to file encryption is ripgrep preprocessing. I owe this solution to this reddit post.
Ripgrep can run a command (or shell script!) to files before processing.
While the original poster was using gpg to encrypt files, just a few modifications allowed me to use age.
#!/usr/bin/env zsh
case "$1" in
*.age)
# The -s flag ensures that the file is non-empty.
if [ -s "$1" ]; then
exec /usr/bin/age --decrypt -i ~/.age/personal $1
else
exec cat
fi
;;
*)
;;
esac
This script operates on all .age files, decrypting them to stdout.
When using a preprocessor, ripgrep will simply search the output of the command.
I often have a mix of files in my notes directory, so I use filetypes to restrict ripgrep.
This means I have to add a new filetype to allow .age files through the filter.
rg --type-add 'aorg:*.org.age' \ -torg -taorg \ --pre ~/age-preprocessor.zsh --pre-glob '*.age' -l TODO /home/user/org
Finally, we have our completed function:
(defun set-org-agenda-files-ripgrep () (setq org-agenda-files (split-string (shell-command-to-string "rg --type-add \'aorg:*.org.age\' -torg -taorg --pre ~/age-preprocessor.zsh --pre-glob \'*.age\' -l TODO /home/user/org "))))
Hope this helps!