43 Folders

Back to Work

Merlin’s weekly podcast with Dan Benjamin. We talk about creativity, independence, and making things you love.

Join us via RSS, iTunes, or at 5by5.tv.

”What’s 43 Folders?”
43Folders.com is Merlin Mann’s website about finding the time and attention to do your best creative work.

Academic notes in one big text file + tag clouds

Back when I first started writing my dissertation (seems like another lifetime now), I used FileMaker Pro to store all my notes. After hundreds of dollars and a lot of headaches, I now realize that I could easily have used the command line and text files. And I would have ended up with a much more portable, robust, and versatile data bank than anything FileMaker could offer.

I present my quick and dirty note storage solution here as a command line newbie, so I would be grateful for any tips or help. Thought it might be of interest to all the grad students and academics out there.

1) I take notes on index cards. This limits me to "atomic" bits of information, thwarting my bad habit of keeping a massive stream of unbroken notes. (When I used FileMaker, some of my entries were several pages long, clearly defeating the purpose of a database).

2) When I process my inbox, I append these new notes to a growing text file. Each note occupies a single line and gets a unique date/time stamp. It also gets a bibliographic code (author-date of publication - e.g., Wallace03), which I can use to search a separate .bib file. (I save the index cards as a paper backup.)

3) I tag each note with several keywords, using "kw" to set them apart. E.g., kwrenaissance kwprinter kwstatistics kwflorence kw1490s- these might be tags for a note on the number of printers in Renaissance Florence.

4) I can now quickly search my notes using grep. If I want to look only for keywords, then I make sure to begin my search query with kw. So let's say I wanted to see my notes tagged "Beethoven." Well then I would type "grep kwbeethoven notes.txt." (I leave all my keywords uncapitalized.) Immediately all the relevant lines pop up. If I want a broader search, then I leave out the kw (and be sure to turn on case insensitive [-i]).

5) If I'm not sure whether I've used a keyword before (or if I'm lazy), I can always count on VIM's autocomplete feature. Let's say I want to enter a keyword for Roman Law but can't remember whether I've previously used kwlawroman or kwromanlaw. Well, then I simply type kwrom followed by CTRL-N or CTRL-P, which allows me to flip through all the other words beginning with kwrom in the document. I can do the same thing for kwlaw.

6) tr -cs A-za-z '\012' < notes.txt | sort | uniq -c | grep kw | sed 's/kw//' > keywords.txt

And finally, this ugly little command will pull out all the words beginning with kw, strip them of that unsightly kw, count how many times they appear in my notes, and put the results in a text file. I'm sure there would be a prettier way to do this, but it works OK for me. So I get a nice little tag cloud of all my keywords together with frequency, which looks like this:

2 1830s
3 1920s
3 abacus
2 ardvark
5 barbarians
2 beethoven
[and so on...]

I guess my only question now is how big my notes file can become before it gets too slow and/or unwieldy to work with.

I'm thrilled to discover the simplicity of text. It's free, universal, and so much more secure than my other solutions. And with grep and redirection, it's so easy to pull together a new, smaller text file of notes on a particular subject.

The nice thing is that I can always go from here to other platforms. At some point, I plan to split my big text file into single-entry files and import all these little files into DevonThink, so that I can take advantage of that program's "See Also" feature.

But I like the text file. Besides, I can access my database from anywhere using SSH.

TOPICS: Life Hacks
mdl's picture

Sorry to keep responding to...

Sorry to keep responding to my own posts, but I've tweaked my notetaking system even further. Though this has required the somewhat tedious task of going through my old entries, I think this new method is even more robust than the previous one. I wanted to present it to show everyone (as if you don't already know!) how flexible and powerful a text based database can be.

Previously, I've been tagging my entries with keywords (the keyword with the prefix "kx"). This is the simplest way to go. But I'm finding that I need to keep a distinction between different types of keywords. For instance, I want to distinguish between entries that deal specifically with "history" as a topic (e.g., what is the nature of history?) and entries that fall within the field of history.

I also found that I was not consistently labeling my entries. For example, some entries on modern French history would get three separate keywords: "kxhistory" and "kxfrance" and "kxmodern." Other entries would get a keyword such as "kxhistorymodernfrance." So now, I have multiple types of tags. I've tried to avoid letter combinations that appear in English words. (I tried "nx" for names, but then realized that "nx" is found in "anxious."):

fx = field tags (history, literature, science, etc.)
lx = location (place) tags
px = people (name) tags
tx = time (period) tags (e.g., modern, earlymodern, medieval, ancient)
dx = date tags (dx1560, dx1990s, etc.)
bx = bibliography keywords (these go in front of bibliographical codes)
kx = keyword tags

Keeping these distinct helps me keep my tagging and searching more consistent.

I put a text file called "commands" in my notes folder. It contains the following lines:

alias kx "tr -cs A-za-z0-9 '\012' < notes | grep kx | sort | uniq -c | sed 's/kx//' > keywords"
alias fx "tr -cs A-za-z0-9 '\012' < notes | grep fx | sort | uniq -c | sed 's/fx//' > subjects"
alias dx "tr -cs A-za-z0-9 '\012' < notes | grep dx | sort | uniq -c | sed 's/dx//' > dates"
alias px "tr -cs A-za-z0-9 '\012' < notes | grep px | sort | uniq -c | sed 's/px//' > names"
alias lx "tr -cs A-za-z0-9 '\012' < notes | grep lx | sort | uniq -c | sed 's/lx//' > places"
alias bx "tr -cs A-za-z0-9 '\012' < notes | grep bx | sort | uniq -c | sed 's/bx//' > biblio"
alias tx "tr -cs A-za-z0-9 '\012' < notes | grep tx | sort | uniq -c | sed 's/tx//' > periods"
alias grep "grep -i"

When I enter my notes folder, I type "source commands" and I have a bunch of shortcuts ready to generate different types of indexes of my notes.




An Oblique Strategy:
Honor thy error as a hidden intention


Subscribe with Google Reader

Subscribe on Netvibes

Add to Technorati Favorites

Subscribe on Pageflakes

Add RSS feed

The Podcast Feed


Merlin used to crank. He’s not cranking any more.

This is an essay about family, priorities, and Shakey’s Pizza, and it’s probably the best thing he’s written. »

Scared Shitless

Merlin’s scared. You’re scared. Everybody is scared.

This is the video of Merlin’s keynote at Webstock 2011. The one where he cried. You should watch it. »