Covariance is a measure of how much two variables change together. A covariance matrix measures the covariance between many pairs of variables.
When the variables tend to show similar behavior, the covariance is positive. That is when greater values of one variable mainly correspond with the greater values of the other variable, or lesser values of one variable correspond with lesser values of the other variable. When the variables tend to show opposite behavior, the covariance is negative. That is when the greater values of one variable mainly correspond to the lesser values of the other and vice-versa.
The magnitude of the covariance is not meaningful to interpret. However, the standardized version of the covariance, the correlation coefficient, indicates by its magnitude the strength of the relationship.
A covariance matrix measures the covariance between many variables. Because the covariance of a variable with itself is that variable's variance, the diagonal of the covariance matrix is simply the variance of each variable.
