Diff revision: diff-based revision of text notes, using spaced repetition

A few years ago I started getting frustrated with forgetting material I studied for side projects and interests. I knew of flashcard spaced repetition, but it didn’t really suit my need. So I built a tool to help review text notes, to better remember their content. It serves many goals, notably studying a large number of subjects over longer periods of time, by taking advantage of text notes one would normally write anyway. The core idea is to review differences between versions of note files using spaced repetition intervals. I’ve been using this for about 3 years myself.

Warning: the concept is very general, and how you actually use the tool may change your experience entirely. (See “usage tips”)

Here’s a presentation (as a video) showing a typical use of the tool. (Note that “drev” and “diffrevision.py” are both aliases for “python path/to/diffrevision.py” on my machine)

What it is

Here’s the typical usage scenario:

  • Say you write notes in a bunch of text (.txt) files in a directory, or in a text-based wiki program (WikidPad, vimwiki, orgmode, etc.)
  • Every time you make a “significant” change in some files, you run the tool’s “add” and “diff” commands and it will know which files are new, which parts were modified in existing files, and which parts didn’t change.
  • You then review the parts that changed today in 2 days, then 6 days, then even further in the future, etc.
    • Every time you can remember longer until you need to review. That’s the spaced repetition idea, here applied to the differences in notes.
    • The idea is to highlight (in color) the parts that changed. You still see the whole text, including the old parts, when you review the new parts.

I made a command-line tool to support doing this. It’s a rewrite of something I had been using for two years, and I’ve been using the rewritten version for about a year.


Warning to spaced repetition fans:

  • there’s no quantification for the moment, no “grading yourself” from, e.g., 0 to 5 upon review. Intervals are fixed. I’m looking at ways of adding this, but it’s not obvious to do right.
  • there’s no “hiding the answer” (as with flashcards, or with cloze deletion) either. Again it might be an interesting feature to add.

Trying the tool

The tool is command-line based. The review is done in a web browser, by producing a set of HTML files, one for every “diff” to review. It works with plain text files, and if anything changed on a line, the whole line is highlighted when reviewing.

The code is on Github, or here’s a .zip of an hopefully stable version. See below for usage instructions, or just run the program without arguments (“python diffrevision.py”). It’s Python (no dependency on anything but core Python packages), so you’ll need the Python interpreter. I know it to work on Ubuntu Linux with Python 2.6 and 2.7, and I tested it partially under Cygwin (Windows, Python 2.5).

By the way I have other versions written, notably a web app version, but I have a feeling that this will appeal mostly to programmers or power users, so I’m releasing the open source version. To the business-minded readers who think one could charge for this: first read this, this and this, on the marketing of spaced repetition products.

 

Concepts

Unix ‘diff’

The Unix ‘diff’ tool is a program which takes as input two files, and tells you what changed from one to the other. This is used extensively by programmers to know what parts of a source code changed from one version to the next.

Spaced repetition (with flashcards)

Spaced repetition programs such as Supermemo, Anki and Mnemosyne help you remember facts. I’ve described them in more details here. But in essence you create a question and answer flashcard, say Q: what does DNA stand for? A: Deoxyribonucleic acid. The program will ask in a day or two to review it. Then, depending on how well you remembered it, you give yourself a score. That score is then used to adjust the next interval, so maybe next review will then be in a week. This repeats itself, each time stretching the interval more and more, as you remember better every time (hopefully).

The benefits (why use this?)

Why you would want to do this kind of review, in addition to, or instead of, using a flashcard spaced repetition program? Succintly, and cutting corners:

  • Putting knowledge in the form of flashcards is a lot of work (a lot of time for a relatively small quantity of knowledge).
  • Also, the result is not an easily “browsable”, readable knowledge store.
  • On the other hand, taking continuous-text notes is very natural, easily done, and helps making ideas clearer.
    • And you’re free to organize pieces of knowledge the way you want.
    • Also, personally, I just do it anyway, whether I review or not. In itself it helps understanding, and remembering (generation effect).
  • But, contrary to flashcards, reviewing text that changes over time as we learn more on a topic, is a challenge: which parts should be reviewed, if old parts are well remembered, but novel bits, less familiar, are here and there all over the document?
    • Hence reviewing the changes helps being efficient in your review of what is also a useful reference text.

Also, sometimes you don’t mind forgetting some parts. You just want the main ideas to be “fresh in your memory”. I find this form of review especially well suited for this.

Going further, sometimes you simply don’t want to forget something even exists, or that you have notes somewhere on X or Y topic. I’m often confounded when someone tells me we discussed this or that and I’ve absolutely no recollection of the episode… I’m sure it also happens with some things I read, but I simply have no one to remind me of those.

Another benefit is that you can review “in context”: you see the whole text, with new parts being highlighted. So you see the old parts as well, the “context”.

Using the tool

I programmed a command-line utility in Python to support this. It builds a database of versions of note files.

In essence, you first configure the program in “core/config.py” (see explanations at the top of this file). You then write your notes in text files in a single directory (hierarchy could in principle be supported, but it’s not done yet). Say you’ve added your first two .txt files. You then run

python diffrevision.py show_new

and you’ll see a list of the new files in the directory of your notes (specified in the config). You then add them with:

python diffrevision.py add

which will simply add them to the list of files to be watched for changes.

When you’ve added new files or you’ve made some changes and want to construct reviews from differences, you run:

python diffrevision.py diff

This will look for changes in files that the system is tracking, and record new versions for files that changed since the last time that command was run. There are also commands to see new files, add them for tracking changes, or ignore some files.

When you want to review changes that are to be reviewed at this point in time, you use:

python diffrevision.py today

This will create a temporary directory and populate it with one HTML file for each “diff” to be reviewed that day. You then load the “index.html” file in a Web browser, and open tabs for each file. When finished, you run

python diffrevision.py finished 1234 2345 3456 …

where “1234 2345 3456” are the space-separated IDs of each of the diffs. To simplify this, the command is given to you in “index.html” based on the reviews (diffs) you clicked on.

Suggestions for Linux users: in your $HOME/.bashrc, write an alias such as “alias drev=’python path/to/diffrevision.py’ to make it easier to run the program.

Limitations

  • This is limited to plain text files, with almost no support for formatting.
    • Formatting of notes only supports WikidPad syntax for the moment. Writing another formatter is a bit complicated, but doable provided you’re a Python programmer and you have well-isolated Python code to export wiki syntax to HTML. See wikidpad_formatter.py and surrounding files for an example.
  • Don’t use non-ASCII characters (accents etc.) in filenames; at first it’ll smile at you and things will work, but when you least expect it it’ll stab you in the back. For the content of the files, just configure the character encoding properly in configuration.py.

Usage tips

As said in the intro, how you use the tool may change your experience from “bleh” to “how did I ever live without this?” (that’s my case anyway).

  • The most important point: make every change small, 5-10 lines or less before you run “diff”.
    • Try to make changes be coherent, a single “idea”. Otherwise I find myself botching the review.
    • The tool is made for “diff” to be fast to run (checking file modification time), to allow doing exactly that.
  • Use with a personal wiki that stores notes in plain text (WikidPad, vimwiki, orgmode…). I wrote about why this is a good idea.
  • Don’t delay review too much, as reviews accumulate fast, and it can get daunting. Ideally it’s done every day, just like with other spaced repetition programs.

 

Main use cases (who might benefit from this?)

With time, I might write more use cases here. See also my follow-up post on very specific personal benefits I see with this. The main use cases I see are:

  • Studying a large diversity of topics over an extended period of time. This is typical of knowledge work, such as programming. To take an example:
    • Say a programming language is of interest to me, but I hear about it piece by piece (news about Erlang, say)
    • At first I’ll read about it on Wikipedia, take 10 minutes or so and write down what’s different about it in some bullet points (I like bullet points, if you hadn’t figured that out 😛 ).
    • That’ll be in a file named “Programming — Erlang.txt”
    • But then a month later I might see an interesting article about a project that benefited from it
    • I’ll add notes about this to the .txt file.
    • But in the meantime if I hadn’t reviewed my notes, as I’m pretty busy and studying Erlang is low in my priorities, I might very well have forgotten.
  • Studying topics that change over time, such as the state of the world, or any academic field.
    • For topics such as politics, I’ll usually have a “trends” and “news” sections to my pages.
  • Generally having a better mental model of the structure of your notes (no need to review the details if this is the only goal)

Cases where flashcards are better

This point is important: this is not a flashcard-review replacement. For factual information you want to remember very precisely, flashcards are much better. This includes vocabulary review, dates, precise technical terms, etc. I’ll probably expand on this list at some point.

The other very important point is that there’s no way of scoring your performance on a review, and even if I wanted to implement this, it’s just not clear what would be the best way to do it. The absence of score makes it hard to measure/quantify things such as retention rate, forgetting curves, etc. There’s a lot of room for improvement on scheduling, and it’s the next feature I want to add (quantification and better scheduling).

About “memorization”

Memorizing is a very, er, touchy subject. When someone comes up with the topic, many people go up in arms with “but memorizing is useless; creativity, usage and comprehension are much more important!” etc. Yet I still think that for some types of tasks memorizing, to varying degrees of precision, is useful. Hence this tool.

I’ll later present the way I roughly divide studying tasks and the memorization/study method I think appropriate for each category. I’m no pedagogy or spaced repetition expert, by the way, so all this is just the opinions of a programmer.

Personal history with this

I’ve been using this for 3 years, since the start of my Masters degree (which I finished a year ago). I’ve accumulated some (qualitative) experience with the process, some of which I’ll gradually share here.

I have no proof that this works (hard data, graphs). My qualitative experience is that it does help me remember better. To be specific I’ve written a follow-up post on some benefits I’ve seen personally, using very (very) specific examples, as suggested by gwern in the comments.

I hope this may be useful to others. I won’t have much time to work on improving the tool, but as I use it everyday, my natural itch-scratch cycle will probably lead to some new features. Also, feel free to improve on the tool. If your extensions are clean, useful and modular (ie. does not change behavior unless a user’s config specifies it) I’ll gladly merge them with the main Git branch.

Wishlist / TODO for the tool

  • Support for telling the program how well you remembered each diff, like with spaced repetition systems: grading yourself from 1 to 5 and adjusting the interval accordingly. Also, that will help experimenting and keeping statistics. However it’s not as simple as with flashcards, as there may be more than one single piece of information in a review, and you may remember some parts better than others. I’m open to ideas here.
  • No support for file hierarchy for the moment.
  • Adding other formatters would be very useful, notably for vimwiki, ReStructured Text and markdown.
  • For the moment, each version is stored in its entirety, which will sound stupid to anyone knowing anything about version control. But it was faster/simpler. The Storage object is well isolated: you want to make it more efficient, have a go at it 😛

There are many more tasks to be done, of course, but these are the main features I’d like to add.

6 Comments

  1. gwern:

    That’s quite unusual a system.

    I think it’s pretty silly to not being using a wiki or revision control system which you could extract diffs from, but I guess if it works for you…

    I’m not really convinced it works. I think I get plenty of review by just editing the text files: you have to read to update and modify material, and often I will get ideas or think of things to do from that. What would reviewing diffs really add to that?

    More case studies would be much more convincing.

  2. Francois:

    It’s indeed unusual, I hope that’s not a problem in itself 🙂

    It’s funny you mention revision control systems (Git etc.), because it’s also the first comment someone else made. I guess programmers wonder that right away, so I’ll try to be more thorough in my answer. I didn’t integrate with a version control system, either for code or from the editor itself, because:

    – it’s actually simpler that way: I simply use a Python module, extract line numbers the module gives me;
    – if I ever want to change the diff strategy or tool, for example to make diffs in PDFs, .docs, or HTML, I have control over this;
    – it does not force the user to use a VCS (ie. no dependency);
    – the Wiki system I use (WikidPad) has no version control that I know of, and I suspect many other notetaking tools don’t either. I wanted a general tool.

    >> “I think I get plenty of review by just editing the text files”

    You’re quite right, as long as you actually edit all parts worth remembering, at adequate intervals. What I observed is that often I don’t go back on notes I’ve taken for a long time. Even if I edit, usually it’s to add new content, and I won’t really re-read what was already there (though of course I do sometimes re-read, otherwise there’d be little point in keeping the wiki clean). Maybe it’s my usage pattern: I mostly add content when reading on new material I don’t know about (vs, for example, managing notes on evolving opinions). This in turn ends up as new bullet points, new sections, in existing documents.

    >> “More case studies would be much more convincing.”

    Absolutely. I want to start tracking my performance, but as I mentioned it’s not as straightforward as with flashcard SRS. Accumulating stats will be long, though.

    Also someone else suggested I do an explanatory video, and I’m working on this.

    Thanks for reading and for your comments!

  3. gwern:

    > Absolutely. I want to start tracking my performance, but as I mentioned it’s not as straightforward as with flashcard SRS. Accumulating stats will be long, though.

    I didn’t mean stats, necessarily, but examples or case studies: ‘on the fifth, I remembered that shell backticks need to be escaped because the day before I read a revision ranting about how shell is the worst programming language in the world’; ‘on the 20th, I ordered a great present x for my brother-in-law because I saw a note from last year mentioning X as a really cool looking gadget which might be worth getting when it comes out in a few months’; etc.

  4. Francois:

    Oh, sure, that might help, and it’s pretty easy to add. I’ll try to think about some concrete examples, and take note when it actually serves me in practice. However it’ll be hard to tell if I would have remembered without reviewing.

    Right away, however, a “side effect” that comes back often is it reminds me to check things that might have come out / evolved since the time I created the revision. E.g. Kickstarter projects I want to keep an eye on, and that is clearly a consequence of reviewing (though not the original intent!).

  5. Francois:

    Oh and by the way that second example of yours happened to me 3 hours ago 🙂 ie. I had taken note of some gift to get for someone a few months back, and that’s probably what I’ll be getting him (but to be fair that’s just the benefit of taking notes in a consistent manner)

  6. Francois:

    I’ve made a new blog entry on specific examples as you suggested, gwern:

    http://www.fsavard.com/flow/20.....-examples/

    It’s not _all_ that I do with it, but I hope it helps clarifying how it might be useful to others.

Leave a comment