Structure in the flow

This is a “weekly report” for the lab I study in, mostly intended for other lab members. See the first one for further explanations.

Lab-related projects

As said in the last post, for my MLP experiments I had started work on classification of the Iris set. Performance was pretty poor, so Pierre-Antoine suggested I use a softmax activation function for outputs. I did it this week, using the proper gradient (he sent me a document concerning this). It worked almost right away, with training/test error very near what I obtained with other methods (histogram-based, Bayes’ classifier with multivariate gaussians) in lab work for IFT6390.

I also started working on implementing other tricks found in Efficient backprop (LeCun), but I had little time and I realized I’d better work on visualization and/or find better ways to see by myself why the tricks work (why using softmax produced such results, for example). Blind implementation is not going to help me much for learning purposes. I’ll have to think about it.

Readings

• I read most of “Extracting and composing robust features with denoising autoencoders” by Pascal Vincent et al. I spent quite a while pondering over some of the alternative mathematical perspectives (information theory, stochastic operator), trying to understand them correctly. I asked a few questions to Pascal directly, and after some more brain cycles I think it’s pretty clear.
• I read the first half of “Loss Functions for Discriminative Training of Energy-Based Models” by LeCun and Huang. My desire was to understand what energy-based models are, and the advantage offered by that perspective on learning.
• I read rather thoroughly some LISA’s projects descriptions sent to me by Yoshua.

General course work

• I began the first homework for the Machine learning class. I’m using it as an occasion to learn to work with LaTeX (I know basic math syntax, but not much more).
• We’ve got a work assignment in the NLP class where we need to predict letters one by one in SMS text (Shannon’s game) by developing a suitable language model. I started work on this with a teammate.

Plan for next week

In terms of projects, I’ll be working mostly on the Shannon’s game program, trying to get as far as I can on this front. Papers to read: to be done.