Structure in the flow

When collaborating with others, we often set up rules on how we’re going to proceed: we work from 8 to 5. We do meetings once a week. Etc. Standards, laws, rules, call them as you wish, we have a natural tendency to create them to make the environment more predictable and stable (among so many other reasons).

Yet there’s no reason you shouldn’t adopt them when working on your own documents, even if you’re not going to share them. Having evolved my own personal wiki for a few years now, I certainly see their benefits. Here I’ll share a few of my own standards and explain the reasons behind them.

They all have a common root: making the structure stable through time, so you can make a page evolve over many years, as slowly as it needs. It’s always fun to revisit a list I’ve started two years ago, add a new point, close the page, and know the structure is still as valid now as it was back then. No need to start everything over because I left the page in the dark for months. No need to reread everything either: I can easily find the place where I wanted to insert the new point, or get the information I needed.

You can think of it as the you-of-now collaborating with the you-of-the-future, in a sense.

The meta-conventions

The first standard I’ll mention is the “wiki meta” page. This is the page where I document my own conventions. This might seem overkill, but it actually helps in shaping the rules and thinking about them (which is a core reason for writing things down: it makes them clearer). In there, I have

• The goals of the wiki; its scope.
• A list of common page sections
• Format and syntax rules
• Guiding principles
• Special keywords and their meaning

I’ll give examples of the content of each of these sections bellow

Common page sections

Perhaps the most useful structural principle I use is the *common sections*. These are similar to common sections you’ll find in a book: table of contents, intro, chapters, conclusion, references, etc. Encyclopedias and Wikipedia page also have patterns in sections. Here’s a list of common sections in my pages, in the order they usually appear on a page:

• Related entries
• To do
• Changelog
• Glossary
• Sources of information
• News
• Events and people
• Experience, common traps and mistakes

These are sections which apply to almost any entry. Of course, other patterns emerge in specific areas of the wiki. But you get the idea.

Guiding principles

These are high-level, perhaps even a bit gnomic or philosophical, statements to guide changes in the wiki. Examples of these are:

• Be precise in meaning, be factual. List references behind noted information to know where it came from, who stands behind it.
• Have a clear top-down structure for division in subtopics in a page, as much as possible.
• Constantly think of better ways to organize the entries.
• Don’t hesitate to link related entries, to move about easily and see structure more clearly.
• Split entries that get too long into subtopics.

Format standards and precise syntactic rules

These are rules I try to follow regarding the meaning of text markup or special notation. For example:

• Highlighting an important point is done by bold (vs capitals, italics, highlighting etc.)
• A new concept or keyword is noted by BOLD AND CAPITAL
• When an information is uncertain or doubtful, mark it with (??)
• Use “–” to indicate subsumption in entry titles (ie. “Mammals — Bear”)

I also have a special syntax for:

• Dates: I date almost every section and bullet point I write. That way, I can clearly see when something appeared, how the page evolved. WikidPad has a keyboard shortcut to insert a date formatted the way you like, by the way.
• Tags: Wiki links do a good job for most structure needs, but I nevertheless use tags here and there for tranversal considerations and classifying subpoints in multiple pages (cf. my related WikidPad extension).

Further reading

I haven’t read it myself, but the Manual of Style of Wikipedia seems interesting if you want some more inspiration for convention ideas.

Bookmarks have been around since the earliest browsers. With years I’ve accumulated thousands of them, and I’ve heard of similar numbers from other people. As it grows, it becomes obvious some organization is needed.

The organization scheme that comes naturally, at first, is folders. Those have been there since the early beginnings with Adam and Eve when the Web was young and domain names were aplenty in the tree of TLDs. That’s what I relied on for about six thousand years, and it became a huge mess. I still have old folders from my antique “classification system” I never look at anymore except for quick-and-easy nostalgia.

Tags and multiple axis for classification

So it quickly becomes obvious more axis are needed to classify. The most self-evident case is when doing a project: you’ll quickly accumulate a bunch of links which are contextually related because of the project, but otherwise would end up in different categories.

For example, if you’re creating a web site on video games, you accumulate links on, say, Nintendo, HTML, marketing and Ramen noodles, but in the Grand Scheme of Things (ie. Dewey classification, or some directory like dmoz.org), these are not usually put together.

So you end up trying to set up some keyword/tag system, hack together for 20 hours some frail Firefox extension, and then Firefox 3 comes along and does it for you anyway. The end result is you can create a tag for your project, yet also tag with proper general categories.

With a personal wiki

But in my experience that still doesn’t work, based on the fact that I never look at the bookmarks again, except when I have a very precise idea of what I’m looking for. And then there’s Google anyway.

In fact, currently, I only use local bookmarks for

1. Transition until I put them in my wiki;
2. pages and sites I use all the time, so I need quick access (online tools etc.).

Why is it that my old bookmarks were still condemned to live unfullfillingly in the dark for eons? There are many reasons, but the main one, for me, was that bookmarks don’t offer any formatting options and their context is not rich enough, even with tags or folders.

When you write a blog post or wiki entry, you can use context, explanations, and the links make sense. They’re part of the text, and when you look back at the entry, you don’t just see a list of equally-created bookmarks, but each has its place and the content you summarize, the description you make create a whole, and of course it’s text so you can have sections, bullet points, images and whatnot.

So my current system is one where I put my bookmarks in wiki entries related to their topic, with some summary explaining why it’s there and what I extracted from it, if I read it. If I need more axis, I’ve developed a WikidPad extension to tag a part of a page.

It seems to work: I reuse the links much more often, and it actually creates value for me as the content slowly grows with the links and knowledge instead of just being an anonymous bunch of pointers.

Of course it requires a bit more work for each link, but in the end if you’re not willing to spend 30 seconds writing a quick note, perhaps it wasn’t worth bookmarking anyway.

(This post is geared towards programmers.)

This blog is about structuring your personal knowledge. Code snippets and, more generally, programming language information, are interesting in that everyone and their cubicle neighbor seem to have their own approach to organizing them. Here I survey some interesting software and approaches I’ve read about, their features, and present my own method based on my personal wiki.

This post is an example where wiki features come in handy (by opp. to a thorough survey of Code Snippet Management as, err, an academic field of study).

Software and approaches

A code snippet manager is a piece of software which allows you to organize short pieces of code to reuse later. Yet I’m also seeking the ability to integrate general information about the language (explanations, elements of theory, etc.): in my experience, snippets are often examples of a notion I’m learning.

In researching a bit on existing systems, I’ve found a few feature families:

• Code features
  • Syntax highlighting
  • Management of multiple files (a plus if you want to add entire libs to your snippet database)
  • More specific:
    • automatic indentation on insertion
    • dependency management
    • IDE integration
    • (other noteworthy?)
• Organization and retrieval features
  • Hierarchical: by language, by functionality/algorithm
  • Tags
    • Tags are particularly useful here (vs pure hierarchical) because I’ll often stumble on situations like:
      • I need a snippet in whatever language for a quick sort algorithm.
      • I need a C++ snippet with an iterator loop.
  • Full-text/regular expression search
    • Regular expressions are especially useful since you often seek specific constructs and regular text indexing won’t cut it.
  • Hyperlinks (well, hallmark of wikis here)
  • Date and other general fields
• Sharing

There are lots of different approaches and systems. Specialized software exists that allows you to organize your snippet library in a standalone and dedicated manner. Google Snippely is an example:

Screenshot of Google Snippely

A whole bunch of sharewares exist that do similar jobs. Some IDEs come with a snippet manager integrated, as is the case with Visual Studio. Most of these local programs offer a basic outline for organization with more or less search capabilities. If you’re looking for an online version with tagging, check out Snipplr, which, being online, also allows you to share and search others’ submissions.

Snipplr homepage screenshot

In the homebrew solution department, this thread is interesting. Some people talk of filesystem based solutions. A few even use a custom database. Personal wikis (as I use, see bellow) and general outliner software clearly need mention too. For example, this blogger says she uses Microsoft OneNote to organize her snippets.

Getting a bit less personal, it should be noted a quite a few bloggers describe their blog as being a “repository for them to search later”. Therefore blogs and websites somehow count as personal snippet libraries (I did a bit of this with my old me-me-me blog over yonder). These score high on integrating other information (ie. free-form formatted text) with the snippets, and of course on the sharing aspect. Community wikis (ie. not personal) are also a great way to organize and share snippets and knowledge (examples here, here).

As a side note, it’s pretty clear we won’t only rely on our own snippets when coding. “The Web + Google” describes my most often used “system” when searching for coding solutions. Yet there are specialized search engines for this job: Google Code Search (you can use regexps on the whole DB!), Koders, and quite a few more.

My approach

Given earlier posts, this doesn’t need a drum roll introduction: I use my personal wiki to organize my snippets and my programming language learning. Of course, this solution allows for inclusion of formatted text. I admit I have a strong tendency to use my snippets for learning more than for reuse, so that factor might weight more than usual in my choice.

A wiki will allow for many different types of retrieval. For example, using the right combination of plugins, with WikidPad I have hierarchical organization, tags/keywords, full text and regular expression search, and, of course, linking. Most popular wiki systems will have plugins to allow for syntax highlighting, and WikidPad is no exception.

Code snippet screenshot in WikidPad
(using the PrettyCode extension)

Where that solution might be lacking is in the IDE integration department, and in the management of multiple files. In the last case, I have a separate personal code (file system) directory to which I may refer using file:// links.

Digression on repetition

Information overload has numerous causes, and one of them is plain old repetition, e.g.: two sources delivering the same information, with superficial differences. It’s natural to repeat information for various reasons.

As an example, when students take notes on a teacher’s lecture, they all duplicate basically the same information. If they all decide to put their notes online, bam, 30 new versions of “Notes on Heisenberg uncertainty principle”. Same goes for journals and bloggers reporting on a given event.

Of course there might be additional value to each version, different points being made, but for someone doing research on recent events, he still gets to read again and again the same basic facts.

Clearly there’s no simple solution. In fact I might mention here that discussion in the blogosphere does create repetition, but makes that information evolve. Something similar happens for students exchanging notes. In this light, repetition appears as a necessary evil.

If we really want to get philosophical, let’s just say repetition is unavoidable from the very start, as production of repetitive information is just the consequence of information flowing in the social graph and of different human beings going through similar experiences and train of thoughts. And clearly it’s not because one of them has eaten apple pie that humanity can move on and experience other stuff.

Gratuitous picture of humanity’s bane (source)

(Ah, of course, the irony here is that this very article is just some remix of ideas told a zillion times over).

My WikidPad extension

Yet, being aware of the problem, you can at least work on making your own set of notes as repetition-free as possible. That’s another core reason why I love personal wikis. Instead of rewriting information on two pages, as you’d do in paper notes because you don’t have your old notebooks handy, you simply link to the other page and voilà! you just avoided adding a little more repetition to this world (why not add some grandiose here? ).

Yet there are cases where where linking is not enough. Say I’m taking notes on the differences between two programming languages, C# and Java. I have a page on C#, a page on Java. Where do I put the notes? I could create a page dedicated to that topic, but I don’t have enough material for the moment to justify that. So say I put them in the page about Java. Consequence: when on C# page I have to navigate to the other page to read the info.

Diagram explaining the extension

What my extension does is grab the info on the Java page (and any other page) and dynamically bring the relevant sections in the C# page. Technically, you give the extension a keyword, and it will search your whole wiki to find pages that contain it. Then, in those pages, it searches for precisely the lines that contain your keyword and some context around it (“sections”). It then prints a list of those sections.

Now it doesn’t matter as much where I put the notes. As long as I label the sections correctly, I can centralize them in the relevant pages when needed, and I don’t need manual copy anymore.

Grab the code & read details here: http://www.fsavard.com/flow/wikidpad-dynamic-search-results/

In my introductory post on personal wikis, I briefly explained their use to organize personal knowledge. I also said I use WikidPad, personally. Here’s the rationale behind my choice.

Main points

WikidPad is a desktop application, so you have to install it locally, as opposed to wikis accessed through the Web. It works on Windows, Linux and Mac OS X. You can download it here, and here’s a quick start guide.

The killer point, for me, is that WikidPad allows for quick edition and navigation. Speed is crucial: if you feel slowed down, chances are you’ll just give up or at least won’t use it with all benefits of a personal wiki.

WikidPad screenshot

Wikidpad main features are:

• Easy and quick edition of entries, with on-the-fly formatting and linking (explained bellow)
• Outline view (on the left) and other navigation helpers (ex: history of last accessed entries, useful keyboard shortcuts)
• More ways to structure your entries
  • TODO lists produced by gathering TODOs on all entries
  • Entries attributes, which allow you to list all entries that have a given attribute
• It’s open source, written in Python and can be extended with modules
• The data is stored in plain text, one .wiki file per entry

The “programmer’s point of view” in the title stems from the fact that the speed of edition might be achieved with more conventional edition behavior (WYSIWYG or Word’s style edition), but then one loses the wiki syntax which to a programmer (or simply nerd) is very appealing for being similar to plain text formats. That’s the unique mix of WikidPad, from what I gather.

Edition behavior: details

WikidPad has a sort of dual-mode of edition/viewing. An example is best to explain: if you mark a word be in *bold* (with asterisks), you don’t have to switch back to “formatted view” mode for it to be showed in bold: it’s shown that way while editing, on-the-fly.

Example of on-the-fly formatting
(++ is for title, *for bold*, _for italics_, etc.)

That turns out to be a real time saver, for me. With all other wiki software I have used, edition is done differently, in either of two ways.

In one case, there’s a constant switching back-and-forth between “formatted view” and “edit view”. TiddlyWiki operates in this way, for example. The thing is, most of the activity done in a wiki involves small modifications here and there, so this “switch” happens often enough to make it cumbersome, to my experience.

In the other case, the WYSIWYG (What You See Is What You Get) behavior, you don’t have to switch, but you format using buttons or keyboard shortcuts, just like in Word (ie. you select the text and click “B” to make it bold). Most people will probably find this a better option, but as a programmer (or just a nerd?), I love the fact that there’s nothing hidden: what I write is all there is in the .wiki file that represents the entry (ie. you work directly at “source” level).

Side note: programmers will recognize WikidPad’s behavior as being the one of an IDE (integrated development environment), esp. for the code coloring aspect. That’s why WikidPad motto is “an IDE for your thoughts”. It also features auto-completion of WikiWords, as another similarity.

Outline view

Example outline (here for the help, which is itself a wiki)

A killer principle of wikis (vs traditional note organizers/outliners) is their very flexible structure: you’re not restricted to having a note be a “child” of one and only one note. You can link from any entry to any other. Yet hierarchies are very intuitive, since we’re used to placing files in folders, and folders in other folders, etc.

WikidPad gives you both. Sure, you can link from any entry to any other one. But an entry linking to another one automatically becomes a “parent” in the hierarchy (outline) displayed to the left. And yes: an entry may have multiple parents that way. (Surprisingly, this does not cause any universe-shattering paradoxes)

This outline gives you yet another way to navigate in the wiki and speed things up.

Extensibility

WikidPad is open source and written in Python, a language which many programmers like to work with for personal projects. WikidPad allows you to write and load your own extensions.

Some extensions you can readily use allow you to visualize the topology of your wiki (with GraphViz) or include mathematical formulas or graphs (MimeTex, Ploticus).

Links and references

• WikidPad home page
• “First start” quick guide for WikidPad
• Someone doing medical research and using WikidPad to take notes (mentioned in other post)