Machine Learning Summary

A grand tool box for vision people

chap 2
Linear Regression with Maximum likelihood estimation
Normal distribution
Student T distribution : SGD(Stochastic Gradient Descent), EM(with Gaussian Scale Mixture)
Lapalce distribution: Linear Programming, EM(with Gaussian Scale Mixture), or Huber Loss function

Censor regression (Kevin: not a big deal, since it only move the line a slightly up, why there are hundreds paper on it?)

chap 3
Logistic Regression
Objective: convex
Parameter Estimation: no close form solution
1. Newton method(IRLS)
2. minfunc in Matlab

always get optimal solution

Multidim Regression: no big deal
Probit Regression: convex objective, use Gradient Descent(minfunc) or EM(slow) to fit it.

chap 4
Model Selection
1. Baysian Approach: P(D|M). Average the all possible theta to protect from overfitting. (need concrete example)
2. BIC approximation. dof(M) can be estimated by minimum encoding of the model( information theory). Good if there are many models and there are some ways to get dof(m) from anther model dof(m')
3. cross validation: not suitable when there are many candidate models, takes too much time

L2 regularization:
QR
SVD
Gradient

L1 regularization(Lasso):
Problem: Laplace is not differentiable at origin
Sol: soft threshold to the point near origin
problem of sol: not a unbiased estimator anymore
sol of above: reestimate the nonzero w with Least Square( a unbiased estimator)

Linear Programming(not editor of choice)
LARS
SCAD( not editor of choice): just a adhoc approach. cannot be put into baysian framework
NEG: best but slow

chap5
Neural networks
Non-convex
Cascade linear and non-linear model( it has to be, or the different layer will collapse and become single linear layer)
Use gradient descent to do estimation( back propagation algorithm)


chap 12
Generative model
PI->Yi->Xi

Discriminant Analysis
p(x,y) pic here

Discriminative method(logistic regression)
p(x|y) pic here

chap 13
Feature selection:
Forward Feature Selection:
Greedy put one feature in( editor of choice, simple and better than stochastic approach like genetic algorithm, simulated annealing, ...)

More prior:
Normal Gamma: more spiky at origin, and flatter tail than Laplace

chap 14
Mixture Model
PI->Zi->Xi
Different from chap 12, since Zi is hidden(need to be inferred from EM), but Yi is given.

Hacking SuperTab.vim

I have been unhappy about the longest common text of auto-complete provided by VIM for a long long time.
I try to find a good solution on the Internet, but all I got is
Put
completeopt += menuone,longest
in your .vimrc
It does provide a longest common text of auto-complete FOR THE FIRST TIME. After I type one more character and press <tab>, it just starts to move on the popup menu. No more longest common text of all matches anymore.
To work around this, I hacked the SuperTab to provide longest common text.
Here is how:

1. open supertab.vim in your /plugin directory
2. find function! s:SuperTab(command)
3. change the code

    " handle 'context' completion.
    if b:complType == 'context'
      let complType = s:ContextCompletion()
      if complType == ''
        exec "let complType = \"" .
          \ escape(g:SuperTabContextDefaultCompletionType, '<') . "\""
      endif
        return complType . key
    endif
to
    " handle 'context' completion.
    if b:complType == 'context'
      let complType = s:ContextCompletion()
      if complType == ''
        exec "let complType = \"" .
          \ escape(g:SuperTabContextDefaultCompletionType, '<') . "\""
      endif
         if pumvisible()
             return "\<ESC>a" . complType . key
         else
            return complType . key
        endif
    endif
The idea is simple. Just close the popup menu each time users press <tab>, since VIM will do longest common text before the popup menu appears.

Change Firefox Default Search Engine

1. Type about:config in Firefox address bar
2. Search for keyword.URL
3. Change the URL to any search engine you want
like the following
data:text/plain,keyword.URL=http://www.google.com/search?hl=zh-TW&source=hp&q=

The plugin list of VIM

It's time to organize my VIM plugins

My VIM Plugin Lists

1. SnipMate – provide TextMate like snippet. Use neocomplcache instead
2. SuperTab – use tab to complete words or sentence  Use neocomplcache instead    Can be configured to use tab key to trigger the snippet and word complete at the same time. Very cool~~~
3. FuzzyFinder – find files and functions in an instant 
4. L9 – required for new version of FuzzyFinder
5. EnhancedCommentify– comment and uncomment a piece of codes with single keystroke
6. DoxygenToolkit – generate function comment
7. a.vim – switch between .cpp and .h files
8. visual_studio.vim – compile VS files in VIM Not compatible with VS2010, 
9. omnicppcomplete – complete c++ files Use neocomplcache instead
10. surround.vim – make hello_world to "hello_world" with few key strokes
11. repeat.vim – make "surround" operation repeatable(with ".")
12. vim-latex – latex syntax support
13. matlab_run.vim – run matlab script in vim
14. matlab plugin for vim – provide syntax highlighting and indention of matlab
15. tlib – provides utilities for viki Use AsciiDoc instead
16. viki – support viki ( a kind of wiki language ) Use AsciiDoc instead
17. neocomplcache – it has been actively developed during recent months. It's mature enough.   It's interesting to see the author reply every blog which has questions about this plugin. It seems that the author really cares about his work.
18. neocomplcache-snippets-complete – since neocomplcache V7, the snippets have been separated from neocomplcache. You need to download it individually. 
19. delimitMate – parenthesis completion, very handy
20. cscope_map – provide keyboard mapping for cscope

Required Software:

1. Deplate – convert viki to html
2. Python 2.4.4 – required by Visual_Studio.vim
3. Ruby 1.86 – required by Deplate
4. ctags – required for omni-complete 
5. cscope 15.7 – a better ctags for python. The zip file contains windows binary.
6. Python for Windows Extension – required by Visual_Studio.vim

Abandoned plugins:

1. taglist – use FuzzyFinder instead
2. project – use FuzzyFinder instead
3. RltvNmbr – too slow included in VIM 7.3
   
4. snippetsEmu – too old, use SnipMate instead
5. neocomplcache – too new. It is a combination of omnicppcomplete, SnipMate, AutoComplPop and SuperTab. It doesn't not work properly if you set the same trigger key to complete and snippet. However, it supports nested snippet and longest common string complete. Very promising.

Plugins needs modified:

1. visual_studio.vim
   
   1. needs to update error message format for visual studio 2008 to work (I set it to set errorformat= %#%f(%l) : %#%t%[A-z]%# %m )
   2. needs to change key binding to avoid conflicts with viki.vim ( I changed them from vc to ic)
2. SnipMate.vim
   
   1. needs to update snippets (of course) 

*Update 2010/12/11 for Vim 7.3
*Update 2011/03/10 for delimitMate  
*Update 2011/07/11 for cscope 
*Update 2012/04/24 for neocomplcache 

Add VIM to the right-click menu

The default VIM 7.2 right-click menu will open the new file in the current buffer.
It's quite annoying because I want to open the new file in a new tab most of the time.

To solve the problem, just import the following setting to the windows registery


Windows Registry Editor Version 5.00

[HKEY_CLASSES_ROOT\*\shell\Edit with &Vim\command]
@="\"c:\\Program Files\\Vim\\vim72\\gvim.exe\" -p --remote-tab-silent \"%1\""


SkyMall -- the pleasure of Delta Airline

One of the enjoyful part when I took Delta airline is to browse the skymall shopping magazine.
Theere are always some crazy stuff there.

Like R2D2

And Programmable Gaming Robot

Its pets' solutions

Want to shut your dog up?
Get this

Bark Free Dog Barking Controls

Recent Fun

I recently spent some time playing an old RPG game I got from 2005

Lost Labyrinth!!! 
 
It always gives me a lot of fun to play it.
It is a simple game, but hard enough to be beaten by computers all the time.