Calculate Eigenvalues and Eigenvectors of Matrices by Hand – Fusion of Engineering, Control, Coding, Machine Learning, and Science

In this linear algebra tutorial, we will learn how to calculate eigenvalues and eigenvectors of matrices by hand. It is true that today almost all calculations are done by using computers. However, knowing how to calculate a linear algebra object or a quantity by hand is very important since only in that way we can truly obtain a long-lasting and solid understanding of numerical linear algebra. Moreover, by learning how to compute quantities by hand, we will obtain enough insights into how linear algebra algorithms work behind the scenes. This knowledge will enable us to develop new algorithms. The YouTube video accompanying this tutorial is given below.

Definition of Eigenvalues, Eigenvectors, and Characteristic Polynomial

Let us consider an $n\times n$ matrix $A$ .

Definition of eigenvalues and eigenvectors: A (complex) number $\lambda$ is an eigenvalue of the matrix $A$ , if there exists a non-zero vector $\mathbf{v}\in \mathbb{R}^{n}$ such that

(1) $\begin{align*}A\mathbf{v}=\lambda \mathbf{v}\end{align*}$

The vector $\mathbf{v}$ is called the eigenvector of the matrix $A$ .

Consider this equation

(2) $\begin{align*}A\mathbf{x}\end{align*}$

where $\mathbf{x}$ is an arbitrary n-dimensional vector. We can say that the matrix $A$ acts on the vector $x$ . This means the following. Depending on the direction of $\mathbf{x}$ , the matrix $A$ can rotate and scale the vector. If the eigenvalue is real, then the eigenvector can be interpreted like this. The eigenvector is the vector that is only being scaled by the action of the matrix with the possibility of changing the sign! That is, if $\mathbf{v}$ is an eigenvector and if $\lambda$ is real, then if we multiply $A\mathbf{v}$ we will obtain $\lambda \mathbf{v}$ and that is precisely the scaled version of the vector $\mathbf{v}$ . The action line of the vector $\mathbf{v}$ did not change by the action of the matrix $A$ .

The eigenvalues can be real or complex. Here, it should also be kept in mind that the entries of eigenvectors can be complex numbers! Also, it should be observed that the eigenvector $\mathbf{v}$ is the vector that belongs to the null space of the matrix $A-\lambda I$ .

Let us now explain how to compute the eigenvectors and eigenvalues. The equation (1) can be written in the equivalent form

(3) $\begin{align*}\big( A-\lambda I \big) \mathbf{v}= 0 \end{align*}$

here $I$ is $n\times n$ identity matrix. We are searching for a non-zero solution $\mathbf{v}$ of this homogeneous system of equations. That is, we are searching only for non-trivial solutions. This system will have non-zero solution $\mathbf{v}$ if and only if the determinant of the matrix $A-\lambda I$ is equal to zero. That is, the system has non-trivial solutions if and only if

(4) $\begin{align*}\det \big( A-\lambda I \big) = 0\end{align*}$

The equation (4) is a polynomial in $\lambda$ . The equation (4) is called the characteristic polynomial or the characteristic equation of the matrix $A$ . The solution of the characteristic polynomial, or equivalently, its roots are the eigenvalues of the matrix $A$ . Generally speaking, the polynomial (4) is of order $n$ , and consequently, we will have $n$ eigenvalues. Note here that some eigenvalues can be identical. Let the eigenvalues be denoted by $\lambda_{1},\lambda_{2},\ldots, \lambda_{n}$ . Let us for the time being assume that the eigenvalues are distinct. That is, to every eigenvalue, we can associate an eigenvector. Let such eigenvectors be denoted by $\mathbf{v}_{1},\mathbf{v}_{2},\ldots, \mathbf{v}_{n}$ . Once the eigenvalues are computed by computing the roots of the characteristic polynomial (4), the eigenvectors are determined by solving the corresponding systems of linear equations

(5) $\begin{align*}\big( A-\lambda_{i} I \big) \mathbf{v}_{i}=0,\;\;\; i=1,2,\ldots, n\end{align*}$

Numerical Example of Computing Eigenvectors and Eigenvalue of 3 by 3 Matrix

Now that we understand the definition of the eigenvectors and eigenvalues, let us do a numerical example in order to illustrate this definition and calculation steps. We consider the following matrix

(6) $\begin{align*}A=\begin{bmatrix} 1 & 2 & 1 \\ 6 & -1 & 0 \\ -1 & -2 & -1 \end{bmatrix}\end{align*}$

and our goal is to compute the eigenvectors and eigenvalues of this matrix. First, we need to compute the eigenvalues. The eigenvalues are given as the roots of the characteristic polynomial:

(7) $\begin{align*}\det \big( A-\lambda I \big) = 0 \\\det \Big( \begin{bmatrix} 1 & 2 & 1 \\ 6 & -1 & 0 \\ -1 & -2 & -1 \end{bmatrix} -\lambda \begin{bmatrix} 1 & 0 & 0 \\ 0 & 1 & 0 \\ 0 & 0 & 1 \end{bmatrix} \Big) =0 \\\det \Big( \begin{bmatrix} 1-\lambda & 2 & 1 \\ 6 & -1-\lambda & 0 \\ -1 & -2 & -1-\lambda \end{bmatrix} \Big)=0\end{align*}$

We can compute the determinant by using the coefficients from the third column. We do that since the computations will simplify due to the zero in the position $(2,3)$ . The determinant is given by

(8) $\begin{align*}& \det \big( A-\lambda I \big) =1\cdot \big(6(-2)-(1+\lambda) \big) +(-1)\cdot (1+\lambda)\big( (-1)(1-\lambda)(1+\lambda)-12 \big) =0 \\& \det \big( A-\lambda I \big)=-\lambda^{3}-\lambda^{2}+12\lambda =0\\& \det \big( A-\lambda I \big)=-\lambda(\lambda^{2}+\lambda -12)=0\end{align*}$

That is, the characteristic polynomial is

(9) $\begin{align*}-\lambda(\lambda^{2}+\lambda -12)=0\end{align*}$

Obviously, the eigenvalues are

(10) $\begin{align*}\lambda_{1}=0,\;\; \lambda_{2}=-4,\;\; \lambda_{3}=3\end{align*}$

Now, let us compute the corresponding eigenvectors.

First, we compute the eigenvector $\mathbf{v}_{1}$ by solving the following system of equations

(11) $\begin{align*}& \big( A-\lambda_{1} I \big) \mathbf{v}_{1}=0 \\& \big( A- 0 I \big) \mathbf{v}_{1}=0 \\& \begin{bmatrix} 1 & 2 & 1 \\ 6 & -1 & 0 \\ -1 & -2 & -1 \end{bmatrix}\mathbf{v}_{1}=\begin{bmatrix} 0 \\ 0 \\ 0 \end{bmatrix}\end{align*}$

Let the components of the vector $\mathbf{v}_{1}$ be given by

(12) $\begin{align*}\mathbf{v}_{1}=\begin{bmatrix}v_{11} \\ v_{12} \\ v_{13} \end{bmatrix}\end{align*}$

Then the system of equations (11) can be written like this

(13) $\begin{align*}\begin{bmatrix} 1 & 2 & 1 \\ 6 & -1 & 0 \\ -1 & -2 & -1 \end{bmatrix}\begin{bmatrix} v_{11}\\ v_{12} \\ v_{13} \end{bmatrix}=\begin{bmatrix} 0 \\ 0 \\0 \end{bmatrix}\end{align*}$

We can expand this system of equations like this

(14) $\begin{align*}1v_{11}+2v_{12}+v_{13}=0\\6v_{11}-v_{12}+0v_{13}=0\\-1v_{11}-2v_{12}-v_{13}=0\end{align*}$

We want to solve this system by using the Gaussian elimination. First, we multiply the first equation by $-6$ and we add the result to the second equation. Then we add the first equation to the third equation. As a result, we obtain

(15) $\begin{align*}1v_{11}+2v_{12}+v_{13}=0\\0v_{11}-13v_{12}-6v_{13}=0 \\0v_{11}+0v_{12}+0v_{13}=0\end{align*}$

From the second equation, we obtain

(16) $\begin{align*}v_{13}=-\frac{13}{6}v_{12}\end{align*}$

By substituting this equation into the first equation of (15), we obtain

(17) $\begin{align*}v_{11}-\frac{1}{6}v_{12}=0,\;\; v_{11}=\frac{1}{6}v_{12}\end{align*}$

Obviously, the original system of equations has an infinite number of solutions. However, from (16) and (17), we can observe that the solutions are actually parametrized by the following vector

(18) $\begin{align*}\mathbf{v}_{1}=\begin{bmatrix}v_{11} \\ v_{12} \\ v_{13} \end{bmatrix} =\begin{bmatrix}\frac{1}{6}v_{12} \\ v_{12} \\ -\frac{13}{6}v_{12}\end{bmatrix}\end{align*}$

where we have complete freedom to select the scalar $v_{12}$ . For example, we can select $v_{12}=12$ . As the result, we obtain the following eigenvector

(19) $\begin{align*}\mathbf{v}_{1}=\begin{bmatrix}2 \\ 12 \\ -26 \end{bmatrix} \end{align*}$

By selecting any other value of $v_{12}$ we will also obtain another eigenvector. That is, if we multiply an eigenvector by a constant scalar, the result is also an eigenvector!

Next, we compute the eigenvector $\mathbf{v}_{2}$ by solving the following system of equations

(20) $\begin{align*}& \big( A-\lambda_{2} I \big) \mathbf{v}_{2}=0 \\& \big( A+4 I \big) \mathbf{v}_{2}=0 \\& \begin{bmatrix} 5 & 2 & 1 \\ 6 & 3 & 0 \\ -1 & -2 & 3\end{bmatrix}\mathbf{v}_{2}=\begin{bmatrix} 0 \\ 0 \\ 0 \end{bmatrix}\end{align*}$

By using the parametrization, we obtain

(21) $\begin{align*}\mathbf{v}_{2}=\begin{bmatrix}v_{21} \\ v_{22} \\ v_{23} \end{bmatrix}\end{align*}$

The system (20) can be written as

(22) $\begin{align*}\begin{bmatrix} 5 & 2 & 1 \\ 6 & 3 & 0 \\ -1 & -2 & 3\end{bmatrix}\begin{bmatrix}v_{21} \\ v_{22} \\ v_{23} \end{bmatrix}=\begin{bmatrix} 0 \\ 0 \\0 \end{bmatrix}\end{align*}$

or in the expanded version

(23) $\begin{align*}5v_{21}+2v_{22}+v_{23}=0 \\6v_{21}+3v_{22}+0v_{23}=0 \\-1v_{21}-2v_{22}+3v_{23}=0 \end{align*}$

By multiplying the third equation by (-1) and placing it as the first equation, we obtain

(24) $\begin{align*}1v_{21}+2v_{22}-3v_{23}=0 \\5v_{21}+2v_{22}+v_{23}=0 \\6v_{21}+3v_{22}+0v_{23}=0 \end{align*}$

By multiplying the first equation with (-5) and adding the result to the second equation, and by multiplying the first equation with (-6) and by adding the result to the third equation, we obtain

(25) $\begin{align*}1v_{21}+2v_{22}-3v_{23}=0 \\0v_{21}-8v_{22}+16v_{23}=0 \\0v_{21}-9v_{22}+18v_{23}=0 \end{align*}$

By multiplying the second equation by $-9/8$ and adding the resulting equation to the third equation, we obtain

(26) $\begin{align*}1v_{21}+2v_{22}-3v_{23}=0 \\0v_{21}-8v_{22}+16v_{23}=0 \\0v_{21}+0v_{22}+0v_{23}=0 \end{align*}$

From the second equation, we obtain

(27) $\begin{align*}-8v_{22}+16v_{23}=0, \\ v_{23}=\frac{8}{16}v_{22}, \\v_{23}=\frac{1}{2}v_{22}\end{align*}$

By substituting this equation in the first equation of (26), we obtain

(28) $\begin{align*}v_{21}+\frac{1}{2}v_{22}=0 \\v_{21}=-\frac{1}{2}v_{22}\end{align*}$

By using (27) and (28), we obtain the parametrized eigenvector

(29) $\begin{align*}\mathbf{v}_{2}=\begin{bmatrix}v_{21} \\ v_{22} \\ v_{23} \end{bmatrix} =\begin{bmatrix}-\frac{1}{2}v_{22} \\ v_{22} \\ \frac{1}{2}v_{22} \end{bmatrix}\end{align*}$

By selecting $v_{22}=4$ , we obtain the second eigenvector

(30) $\begin{align*}\mathbf{v}_{2}=\begin{bmatrix} -2 \\ 4 \\ 2 \end{bmatrix}\end{align*}$

Finally, we compute the eigenvector $\mathbf{v}_{3}$ by solving the following system of equations

(31) $\begin{align*}& \big( A-\lambda_{3} I \big) \mathbf{v}_{3}=0 \\& \big( A-3 I \big) \mathbf{v}_{3}=0 \\& \begin{bmatrix} -2 & 2 & 1 \\ 6 & -4 & 0 \\ -1 & -2 & -4\end{bmatrix}\mathbf{v}_{3}=\begin{bmatrix} 0 \\ 0 \\ 0 \end{bmatrix}\end{align*}$

By using the eigenvector

(32) $\begin{align*}\mathbf{v}_{3}=\begin{bmatrix}v_{31} \\ v_{32} \\ v_{33} \end{bmatrix}\end{align*}$

The resulting system of equations is

(33) $\begin{align*}-2v_{31}+2v_{32}+v_{33}=0 \\6v_{31}-4v_{32}+0v_{33}=0 \\-1v_{31}-2v_{32}-4v_{33}=0\end{align*}$

By multiplying the third equation by $-1$ , and placing as the first equation, we obtain

(34) $\begin{align*}1v_{31}+2v_{32}+4v_{33}=0 \\-2v_{31}+2v_{32}+v_{33}=0 \\6v_{31}-4v_{32}+0v_{33}=0 \end{align*}$

By multiplying the first equation by $2$ and adding the result to the second equation, and by multiplying the first equation by $-6$ and by adding the result to the third equation, we obtain

(35) $\begin{align*}1v_{31}+2v_{32}+4v_{33}=0 \\0v_{31}+6v_{32}+9v_{33}=0 \\0v_{31}-16v_{32}-24v_{33}=0 \end{align*}$

By multiplying the second equation by $16/6$ and by adding the result to the third equation, we obtain

(36) $\begin{align*}1v_{31}+2v_{32}+4v_{33}=0 \\0v_{31}+6v_{32}+9v_{33}=0 \\0v_{31}+0v_{32}+0v_{33}=0 \end{align*}$

From the second equation, we obtain

(37) $\begin{align*}6v_{32}+9v_{33}=0 \\v_{33}=-\frac{6}{9}v_{32} \\v_{33}=-\frac{2}{3}v_{32}\end{align*}$

By substituting this result in the first equation of (36), we obtain

(38) $\begin{align*}v_{31}-\frac{2}{3}v_{32}=0\\v_{31}=\frac{2}{3}v_{32}\end{align*}$

Consequently, the parametrized vector is

(39) $\begin{align*}\mathbf{v}_{3}=\begin{bmatrix}v_{31} \\ v_{32} \\ v_{33} \end{bmatrix} =\begin{bmatrix}\frac{2}{3}v_{32} \\ v_{32} \\ -\frac{2}{3}v_{32} \end{bmatrix}\end{align*}$