Determinant of a 3x3 matrix