The organization scheme that comes naturally, at first, is folders. Those have been there since the early beginnings with Adam and Eve when the Web was young and domain names were aplenty in the tree of TLDs. That’s what I relied on for about six thousand years, and it became a huge mess. I still have old folders from my antique “classification system” I never look at anymore except for quick-and-easy nostalgia.

Tags and multiple axis for classification

So it quickly becomes obvious more axis are needed to classify. The most self-evident case is when doing a project: you’ll quickly accumulate a bunch of links which are contextually related because of the project, but otherwise would end up in different categories.

For example, if you’re creating a web site on video games, you accumulate links on, say, Nintendo, HTML, marketing and Ramen noodles, but in the Grand Scheme of Things (ie. Dewey classification, or some directory like dmoz.org), these are not usually put together.

So you end up trying to set up some keyword/tag system, hack together for 20 hours some frail Firefox extension, and then Firefox 3 comes along and does it for you anyway. The end result is you can create a tag for your project, yet also tag with proper general categories.

With a personal wiki

But in my experience that still doesn’t work, based on the fact that I never look at the bookmarks again, except when I have a very precise idea of what I’m looking for. And then there’s Google anyway.

In fact, currently, I only use local bookmarks for

1. Transition until I put them in my wiki;
2. pages and sites I use all the time, so I need quick access (online tools etc.).

Why is it that my old bookmarks were still condemned to live unfullfillingly in the dark for eons? There are many reasons, but the main one, for me, was that bookmarks don’t offer any formatting options and their context is not rich enough, even with tags or folders.

When you write a blog post or wiki entry, you can use context, explanations, and the links make sense. They’re part of the text, and when you look back at the entry, you don’t just see a list of equally-created bookmarks, but each has its place and the content you summarize, the description you make create a whole, and of course it’s text so you can have sections, bullet points, images and whatnot.

So my current system is one where I put my bookmarks in wiki entries related to their topic, with some summary explaining why it’s there and what I extracted from it, if I read it. If I need more axis, I’ve developed a WikidPad extension to tag a part of a page.

It seems to work: I reuse the links much more often, and it actually creates value for me as the content slowly grows with the links and knowledge instead of just being an anonymous bunch of pointers.

Of course it requires a bit more work for each link, but in the end if you’re not willing to spend 30 seconds writing a quick note, perhaps it wasn’t worth bookmarking anyway.

This blog is about structuring your personal knowledge. Code snippets and, more generally, programming language information, are interesting in that everyone and their cubicle neighbor seem to have their own approach to organizing them. Here I survey some interesting software and approaches I’ve read about, their features, and present my own method based on my personal wiki.

This post is an example where wiki features come in handy (by opp. to a thorough survey of Code Snippet Management as, err, an academic field of study).

Software and approaches

A code snippet manager is a piece of software which allows you to organize short pieces of code to reuse later. Yet I’m also seeking the ability to integrate general information about the language (explanations, elements of theory, etc.): in my experience, snippets are often examples of a notion I’m learning.

In researching a bit on existing systems, I’ve found a few feature families:

• Code features
  • Syntax highlighting
  • Management of multiple files (a plus if you want to add entire libs to your snippet database)
  • More specific:
    • automatic indentation on insertion
    • dependency management
    • IDE integration
    • (other noteworthy?)
• Organization and retrieval features
  • Hierarchical: by language, by functionality/algorithm
  • Tags
    • Tags are particularly useful here (vs pure hierarchical) because I’ll often stumble on situations like:
      • I need a snippet in whatever language for a quick sort algorithm.
      • I need a C++ snippet with an iterator loop.
  • Full-text/regular expression search
    • Regular expressions are especially useful since you often seek specific constructs and regular text indexing won’t cut it.
  • Hyperlinks (well, hallmark of wikis here)
  • Date and other general fields
• Sharing

There are lots of different approaches and systems. Specialized software exists that allows you to organize your snippet library in a standalone and dedicated manner. Google Snippely is an example:

Screenshot of Google Snippely

A whole bunch of sharewares exist that do similar jobs. Some IDEs come with a snippet manager integrated, as is the case with Visual Studio. Most of these local programs offer a basic outline for organization with more or less search capabilities. If you’re looking for an online version with tagging, check out Snipplr, which, being online, also allows you to share and search others’ submissions.

Snipplr homepage screenshot

In the homebrew solution department, this thread is interesting. Some people talk of filesystem based solutions. A few even use a custom database. Personal wikis (as I use, see bellow) and general outliner software clearly need mention too. For example, this blogger says she uses Microsoft OneNote to organize her snippets.

Getting a bit less personal, it should be noted a quite a few bloggers describe their blog as being a “repository for them to search later”. Therefore blogs and websites somehow count as personal snippet libraries (I did a bit of this with my old me-me-me blog over yonder). These score high on integrating other information (ie. free-form formatted text) with the snippets, and of course on the sharing aspect. Community wikis (ie. not personal) are also a great way to organize and share snippets and knowledge (examples here, here).

As a side note, it’s pretty clear we won’t only rely on our own snippets when coding. “The Web + Google” describes my most often used “system” when searching for coding solutions. Yet there are specialized search engines for this job: Google Code Search (you can use regexps on the whole DB!), Koders, and quite a few more.

My approach

Given earlier posts, this doesn’t need a drum roll introduction: I use my personal wiki to organize my snippets and my programming language learning. Of course, this solution allows for inclusion of formatted text. I admit I have a strong tendency to use my snippets for learning more than for reuse, so that factor might weight more than usual in my choice.

A wiki will allow for many different types of retrieval. For example, using the right combination of plugins, with WikidPad I have hierarchical organization, tags/keywords, full text and regular expression search, and, of course, linking. Most popular wiki systems will have plugins to allow for syntax highlighting, and WikidPad is no exception.

Code snippet screenshot in WikidPad
(using the PrettyCode extension)

Where that solution might be lacking is in the IDE integration department, and in the management of multiple files. In the last case, I have a separate personal code (file system) directory to which I may refer using file:// links.

Those that use Firefox have certainly encountered extensions such as AdBlock Plus and Flashblock, which help in making web pages look less like a stress test for epilepsy. More general (cross-browser) solutions exist by using a proxy mechanism to filter incoming content, such as Privoxy.

Yet one can go even further to isolate the text of an article. Some sites offer a “print” version of their articles (usually a single, clean page), but that’s not the general case. That’s where the Aardvark extension comes in. It allows you to delete elements from a page and rearrange it quickly so you only keep the part you want.

Overview of Aardvark’s modification commands

Once installed, you navigate to a page you want to clean up and you launch Aardvark (Tools -> Start Aardvark). You then see a red rectangle over elements when your mouse pointer hovers over them. You  press keys to activate different editing operations for the selected element (press ‘h’ to get the list of commands).

Aardvark’s help (list of commands)

It helps here to understand how web pages are coded (HTML), but in essence a page is made of rectangular zones inside bigger zones (ex: an image in a paragraph), forming a hierarchy. As your mouse pointer hovers over a given rectangle (say a paragraph title), you may want to select its parent in the hierarchy (the paragraph itself). To do it, you press ‘w’ to ‘widen’ the selection. The inverse operation is ‘n’ for ‘narrow’.

Example of Aardvark’s rectangle selection

You can delete elements in essentially two ways. The first is the straightforward one: you select an element and press ‘r’ to remove it. The other is the opposite: you press ‘i’ to isolate the selected element, ie. keep only this one, remove all the rest. ‘i’ is very useful to select the page main element that contains the whole text, and then you can work the details with ‘r’.

If the isolated text is too narrow (doesn’t fill the page horizontally), you can press ‘d’ to ‘de-widthify’, which means that the ‘width’ attribute (which prevents the block from filling the page) is removed from it. You may have to fiddle a bit until you find the element on which the ‘width’ is applied, though.

Saving the result with ScrapBook

When the modifications are over, I save the page in its modified state using ScrapBook (which I covered in another blog post). I can then read in the format I want, and add notes and highlights. (The ScrapBook extension does have a “delete” feature, but it’s not as featureful as Aardvark’s.)

If an article is spread over multiple pages, you can use ScrapBook “Combine Wizard” (in the SB sidebar: Tools -> Combine Wizard) feature to merge them in a single page.

Information overload has numerous causes, and one of them is plain old repetition, e.g.: two sources delivering the same information, with superficial differences. It’s natural to repeat information for various reasons.

As an example, when students take notes on a teacher’s lecture, they all duplicate basically the same information. If they all decide to put their notes online, bam, 30 new versions of “Notes on Heisenberg uncertainty principle”. Same goes for journals and bloggers reporting on a given event.

Of course there might be additional value to each version, different points being made, but for someone doing research on recent events, he still gets to read again and again the same basic facts.

Clearly there’s no simple solution. In fact I might mention here that discussion in the blogosphere does create repetition, but makes that information evolve. Something similar happens for students exchanging notes. In this light, repetition appears as a necessary evil.

If we really want to get philosophical, let’s just say repetition is unavoidable from the very start, as production of repetitive information is just the consequence of information flowing in the social graph and of different human beings going through similar experiences and train of thoughts. And clearly it’s not because one of them has eaten apple pie that humanity can move on and experience other stuff.

Gratuitous picture of humanity’s bane (source)

(Ah, of course, the irony here is that this very article is just some remix of ideas told a zillion times over).

My WikidPad extension

Yet, being aware of the problem, you can at least work on making your own set of notes as repetition-free as possible. That’s another core reason why I love personal wikis. Instead of rewriting information on two pages, as you’d do in paper notes because you don’t have your old notebooks handy, you simply link to the other page and voilà! you just avoided adding a little more repetition to this world (why not add some grandiose here? ).

Yet there are cases where where linking is not enough. Say I’m taking notes on the differences between two programming languages, C# and Java. I have a page on C#, a page on Java. Where do I put the notes? I could create a page dedicated to that topic, but I don’t have enough material for the moment to justify that. So say I put them in the page about Java. Consequence: when on C# page I have to navigate to the other page to read the info.

Diagram explaining the extension

What my extension does is grab the info on the Java page (and any other page) and dynamically bring the relevant sections in the C# page. Technically, you give the extension a keyword, and it will search your whole wiki to find pages that contain it. Then, in those pages, it searches for precisely the lines that contain your keyword and some context around it (”sections”). It then prints a list of those sections.

Now it doesn’t matter as much where I put the notes. As long as I label the sections correctly, I can centralize them in the relevant pages when needed, and I don’t need manual copy anymore.

Grab the code & read details here: http://www.fsavard.com/flow/wikidpad-dynamic-search-results/

A screenshot of Faster!

Go ahead and try it out. Nothing to download, just click “Play that text”.

In the rest of the post I elaborate on similar software and the effectiveness of speed reading in general.

Comparing it to other available software

One (free & open source) program which has similar features, but more options for rhythm, is Dictator. You must download it though. I created my script to have something approaching that functionality, but available through the web and easy to modify for people who know JavaScript (and, *hrm*, because it made a cute little programming project )

Screenshot of Dictator

Other Web versions (see Spreeder and ZAPReader) are nice but lack some features which make Dictator and commercial programs more appealing, notably following the position in the text. Spreeder does have some more advanced rhythm options, though.

There are many commercial programs which offer even more features, such as integration with Office programs. Instead of trying to list them all here, I’ll point you to this comparison table of free and commercial RSVP programs.

I also want to make my version available for integration in other sites (say, for readers to read your blog posts), but I’m waiting for a few interested webmasters to contact me before I take the time to do it.

Why would one use RSVP? and the reality of speed reading

As mentioned briefly in my last post, some people believe this can help increase your reading speed. The idea is to gradually increase the reading speed, the number of words you read in a group, etc. by tweaking parameters in the software as you progress.

Recently, RSVP was also proposed as an alternative way to read on small displays such as cellphones. Some people seem to actually see RSVP as an alternative reading technique rather than simply a way to increase your “normal reading” speed.

My personal experience with the speed reading aspect is that in can indeed help you learn to focus more and reread back less of the text you read a line before. Also, it forces you to decrease subvocalization — hear the text in your head, or even whisper it. I personally never ended the in-head habit completely, though. Phrasal blurbs seem to overlap rather than disappear, if you will.

All these improvements translate in an increased reading speed for some material. They’re basic goals of “classic” speed reading techniques. A good reference for these techniques can be found at Wikibooks.

Speed reading software also has critics; see the bottom of the Wikibooks page.

Types of reading material

If you’re reading some advanced math textbook, taking the time to understand what you’re reading, sentence by sentence, naturally decreases your reading speed. Basically, speed reading is definitely no panacea for Will Hunting your way through a textbook every 30 minutes.

Where it can really help, though, in my experience (and as said by others), is when reading familiar material, say when reading the daily news or reading multiple opinion pieces on a given event.

Speed reading is to be combined with, but not confused with, skimming. Skimming is about not reading at all some parts. Speed reading is about reading faster the whole thing.

The app: http://www.fsavard.com/code/speedread/

You simply click “Play that text” and you should grasp the principle real quick.

This is thought by some to increase your reading speed if used consistently. It teaches to not go back while reading, not “subvocalize” (ie. hear the words in your head, or even whisper them as you read) and by displaying more than one word at a time, to read more in one glance.

If you like it and want to come back to it, there’s a bookmarklet in the “About & download” tab that will allow you to select text in any web page and use it as input. Or you can simply copy & paste the text in the text area.

I’ll post something more lengthy on speed reading eventually.

When digging deeper on how to maximize the efficiency of flashcard review, the “when to review such or such card?” question naturally comes to mind. It turns out some people have been researching that very question and have come up with interesting tools and theories on how to best use your time when reviewing.

SuperMemo and spaced repetition

One such person is Piotr Wozniak, a rather eccentric Polish profressor who, since the early 1980s, has been studying and perfecting what he calls “spaced repetition“. His software, SuperMemo, implements this technique.

Essentially, once a flashcard is made and reviewed for the first time, it is then scheduled to be reviewed some time in the future, say in 2 days. Then, depending on how well you remember it in 2 days, it is rescheduled for a next review, but this time further in the future, say in a week. The process is then repeated, spacing repetitions further and further until a point where you won’t forget about it anymore.

SuperMemo 2006 screenshot

The core idea behind spaced repetition in SuperMemo is to present the flashcard just when you’re about to forget about it. Dr Wozniak has developped and refined models of how memorization and forgetting happens in time. The end goal, of course, is to repeat a minimum number of times a given flashcard, so you can repeat more flashcards total.

The SuperMemo website details his findings and tips concerning memorization, and can make for hours of reading. In particular, there are techniques to make better flashcards (how to formulate the question, etc.) and to use the SuperMemo software, of course. There is also an affiliated site where you can find ready-made flashcard sets available for purchase.

EDIT: On the website, you can notably find more detailed information on the mathematical details of the algorithm used by the program and its evolution. As mentioned in a comment below by FlashcardDB creator, the Leitner system is another spaced repetition system which doesn’t rely on computers (it was created in the ’70s), but on moving real cards in real decks, as explained on the FlashcardDB site itself.

Alternative software

Altough it seems Piotr Wozniak has been a pioneer in creating spaced repetition software, other programs have recently appeared that follows a similar model.

If you want something cross-platform (I run Linux so that’s my case), you can go for Mnemosyne. It’s much simpler than SuperMemo (much less options, formatting in flashcards, etc.), but it’s open source and free.

Screenshot of Mnemosyne

There are also Web versions. SuperMemo itself has an online version. Very recently, websites SpicyElephant, Mind Picnic and Flashcard DB (mentioned above) have appeared that follow the spaced repetition model. Being online communities, they allow you to share flashcards and reuse those made by others.

References

• WikiPedia: spaced repetition

EDIT: Side note: my own experience or lack thereof

Unfortunately, I do not have a lot of personal experience with this. When I was in high school I had programmed something similar for myself (my formula was real simple and very inflexible). That’s why I found it amusing to discover spaced repetitions a few months ago.

But these days I don’t deal with much information that would apply well to spaced repetition (I study software engineering). Encoding the Q/As is too much effort for the gain. Learning through application works better right now.

I’m trying to come back to it with other learning trails, though, as the principle is just so cool

As you may know, on the Web there are clear incentives to get good search engine ranks to drive traffic to your site, in some cases generating revenue from ads. I just recently realized some of the specifics of this sometimes involve generating massive amounts of content, details of which I found rather startling.

Some background on “Internet marketing”

Search Engine Optimization (SEO), is the work done to make a site stand out in search engine results, for obvious marketing purposes. Some people specialize in doing this. No, bar that: there’s a whole industry centered around this.

Some SEO techniques are entirely ethical and simply make good content easier to find: they’re referred to as “white hat”. Other techniques, “black hat”, use devious ways to route people to less desirable content.

There are ways to benefit directly from search engine rankings without really creating any value in the process. One of those ways is the pathological MFA site, Made For AdSense site (AdSense/Adwords referring to Google ads).

These are those sites you end up on when you mistype a domain name and bam, get a page which is essentially a fresco of Google ads with some teeny-weeny content lost in there somewhere. If for some rather popular keyword someone may get a good spot in Google results for his MFA page, then he’s won the game and reaps some profit when wandering visitors click his ads. At least that’s one scenario.

The techniques of content generation

I knew about this, vaguely, but the other day I stumbled upon this article over at TheNextWeb concerning DataPresser. In a nutshell, this is a tool that allows you to generate content automatically, by following rules.

With DataPresser, for example, you can generate all sorts of variations on “Find cool wallpapers of _______”, where ______ is a blank filled from a database. Not only can you replace the blank, you can even change the way the sentence is worded, using synonyms or even grammatical constructions. That’s to avoid being flagged as “duplicate content” by Google, who obviously tries to eliminate such sites from its results. As you can see, it’s a game of cat and mouse.

The general keyword for this activity is “Content Generation”. There are many techniques and tools. Some will simply generate new pages by republishing from RSS feeds found elsewhere, with ads slapped around. Others will accumulate tons of text and mix bits of sentences from here and there to create text that doesn’t make sense but appears correct to search engines. You can even buy whole databases of content, say game cheatcodes, rather cheaply.

Why mass-produce?

Obviously, it’s profitable to mass-produce text to cover many topics, therefore many keywords. But there’s another reason: search engines give more credit to sites with links pointing to them. That’s why some content generation involve the creation of many sites linking to each other (the whole thing is called a link farm). Another “link-building” technique concerns message boards and blog comments, with posts being made solely to create links that add to a site search-engine karma.

Conclusion

I think this phenomenon plays a big role in understanding the huge “size” of the Web (number of pages). A very simple technique to generate more revenue for a given site, for example, is to split an article in multiple pages so more ads can be displayed. But with these MFA sites, we’re talking about generating thousands of pages at the click of a button!

In the end, it all boils down to spam and background noise. Whatever the service, if it can generate buzz, chances are it’ll be exploited.

References

• The Basics of Content Generation: Methods, Coherrence, and Unique Content
• Madlibbing your way to riches with Datapresser

In the end, perhaps unsurprisingly, the magic relies on being really quick at judging an article from its title, its overall look and other cues. There are a couple of technical tips, though, about using the Google Reader interface efficiently, like relying on keyboard shortcuts.

Here’s the video:

Main points

WikidPad is a desktop application, so you have to install it locally, as opposed to wikis accessed through the Web. It works on Windows, Linux and Mac OS X. You can download it here, and here’s a quick start guide.

The killer point, for me, is that WikidPad allows for quick edition and navigation. Speed is crucial: if you feel slowed down, chances are you’ll just give up or at least won’t use it with all benefits of a personal wiki.

WikidPad screenshot

Wikidpad main features are:

• Easy and quick edition of entries, with on-the-fly formatting and linking (explained bellow)
• Outline view (on the left) and other navigation helpers (ex: history of last accessed entries, useful keyboard shortcuts)
• More ways to structure your entries
  • TODO lists produced by gathering TODOs on all entries
  • Entries attributes, which allow you to list all entries that have a given attribute
• It’s open source, written in Python and can be extended with modules
• The data is stored in plain text, one .wiki file per entry

The “programmer’s point of view” in the title stems from the fact that the speed of edition might be achieved with more conventional edition behavior (WYSIWYG or Word’s style edition), but then one loses the wiki syntax which to a programmer (or simply nerd) is very appealing for being similar to plain text formats. That’s the unique mix of WikidPad, from what I gather.

Edition behavior: details

WikidPad has a sort of dual-mode of edition/viewing. An example is best to explain: if you mark a word be in *bold* (with asterisks), you don’t have to switch back to “formatted view” mode for it to be showed in bold: it’s shown that way while editing, on-the-fly.

Example of on-the-fly formatting
(++ is for title, *for bold*, _for italics_, etc.)

That turns out to be a real time saver, for me. With all other wiki software I have used, edition is done differently, in either of two ways.

In one case, there’s a constant switching back-and-forth between “formatted view” and “edit view”. TiddlyWiki operates in this way, for example. The thing is, most of the activity done in a wiki involves small modifications here and there, so this “switch” happens often enough to make it cumbersome, to my experience.

In the other case, the WYSIWYG (What You See Is What You Get) behavior, you don’t have to switch, but you format using buttons or keyboard shortcuts, just like in Word (ie. you select the text and click “B” to make it bold). Most people will probably find this a better option, but as a programmer (or just a nerd?), I love the fact that there’s nothing hidden: what I write is all there is in the .wiki file that represents the entry (ie. you work directly at “source” level).

Side note: programmers will recognize WikidPad’s behavior as being the one of an IDE (integrated development environment), esp. for the code coloring aspect. That’s why WikidPad motto is “an IDE for your thoughts”. It also features auto-completion of WikiWords, as another similarity.

Outline view

Example outline (here for the help, which is itself a wiki)

A killer principle of wikis (vs traditional note organizers/outliners) is their very flexible structure: you’re not restricted to having a note be a “child” of one and only one note. You can link from any entry to any other. Yet hierarchies are very intuitive, since we’re used to placing files in folders, and folders in other folders, etc.

WikidPad gives you both. Sure, you can link from any entry to any other one. But an entry linking to another one automatically becomes a “parent” in the hierarchy (outline) displayed to the left. And yes: an entry may have multiple parents that way. (Surprisingly, this does not cause any universe-shattering paradoxes)

This outline gives you yet another way to navigate in the wiki and speed things up.

Extensibility

WikidPad is open source and written in Python, a language which many programmers like to work with for personal projects. WikidPad allows you to write and load your own extensions.

Some extensions you can readily use allow you to visualize the topology of your wiki (with GraphViz) or include mathematical formulas or graphs (MimeTex, Ploticus).

Links and references

• WikidPad home page
• “First start” quick guide for WikidPad
• Someone doing medical research and using WikidPad to take notes (mentioned in other post)