Linear Topic Modeling

Apr 17th, 2013

The interest in learning some kind of topics from a corpus of documents started from the publication of Latent Semantic Analysis (LSA) [Deerwester90], also called Latent Semantic Indexing (LSI) in the context of information retrieval.

LSA is a linear method based on the factorization of the document-word matrix $\mathit{X}$, where $x_{dw}$ is the count of occurrences of word $w$ in document $d$. The goal is to find a low-rank approximation $\tilde{\mathit{X}}$ of $\mathit{X}$ factorizing it into two matrices, one representing the documents, and the other the topics.

In this model, documents are converted to the bag-of-words format which ignores the ordering of words and only keeps the word counts.

Singular Value Decomposition.

The most common way is to use the Singular Value Decomposition (SVD) of

$$\mathit{X}=\mathit{U}\mathbf{\Sigma }{\mathit{V}}^T$$

where $\mathit{U}$ and $\mathit{V}$ are orthonormal matrices and $\mathbf{\Sigma }$ is diagonal. By selecting only the $K$ largest singular values from $\mathbf{\Sigma }$ and the corresponding vectors in $\mathit{U}$ and ${\mathit{V}}^T$, we get the best rank $K$ approximation of matrix $\mathit{X}$ according to the loss $\Vert \mathit{X}-\tilde{\mathit{X}}{\Vert }_2^2$. Rows of $\mathit{U}$ represent documents in a $K$-dimensional space, and columns of ${\mathit{V}}^T$ represents the topics in the same space. Each document can be expressed as a linear combination of topics.

Non-negative Matrix Factorization.

Another common approach is to use use Non-negative Matrix Factorization (NMF) on the document-word matrix $\mathit{X}$ [Lee99] [Pauca04]. Here, $\tilde{\mathit{X}}=\mathit{U}\mathit{V}$ where $\mathit{U}$ represents the topics and $\mathit{V}$ the documents, both being non-negative matrices.

Comments