Matrix decomposition is a foundational tool for many machine learning problems and is widely used in data compression, dimensionality reduction, and sparsity learning, to name but a few. In many cases, for purposes of approximating a data matrix by a low-rank structure, Singular Value Decomposition (SVD) is often verified as the best choice. However, the accurate and efficient SVD of large-scale data matrices is computationally challenging. To resolve the SVD in this situation, there are many algorithms have been developed by applying randomized linear algebra. One of the most important algorithms is randomized SVD, which is competitively efficient for decomposing any large matrix with a relatively low rank.

Figure 1: A timeline of major SVD developments. (The picture is from [2])

This post will introduce the preliminary and essential idea of the randomized SVD. To help readers gain a better understanding of randomized SVD, we also provide the corresponding Python implementation in this post. In addition, Jupyter notebook of this post is available at here.

---

Preliminary

SVD Formula

As you may already know, SVD is one of the most important decomposition formula in linear algebra. For any given matrix A, SVD has the form of

A = U_Σ_V^T

where the matrices U and**_ V _**consist of left and right singular vectors, respectively. The diagonal entries of Σ are singular values.

A Small Matrix Example

Take a 3-by-3 matrix for example, we can compute the SVD by using numpy.linalg.svd() in Python. Let us have a look:

import numpy as np

A = np.array([[1, 3, 2],
              [5, 3, 1],
              [3, 4, 5]])
u, s, v = np.linalg.svd(A, full_matrices = 0)
print('Left singular vectors:')
print(u)
print()
print('Singular values:')
print(s)
print()
print('Right singular vectors:')
print(v)
print()

In this case, the singular values are 9.3427, 3.2450, and 1.0885.

Left singular vectors:
[[-0.37421754 0.28475648 -0.88253894]
 [-0.56470638 -0.82485997 -0.02669705]
 [-0.7355732 0.48838486 0.46948087]]
Singular values:
[9.34265841 3.24497827 1.08850813]
Right singular vectors:
[[-0.57847229 -0.61642675 -0.53421706]
 [-0.73171177 0.10269066 0.67383419]
 [ 0.36051032 -0.78068732 0.51045041]]

#linear-algebra #signal-processing #machine-learning #matrix-decomposition #numpy