glm::inverseTranspose for mat2 returns the plain inverse, the transpose step is missing. The mat3 and mat4 overloads are correct.
#include <glm/glm.hpp>
#include <glm/gtc/matrix_inverse.hpp>
glm::mat2 const M(1.f, 3.f, 2.f, 4.f);
glm::mat2 const A = glm::inverseTranspose(M);
glm::mat2 const R = glm::transpose(glm::inverse(M));
// A != R, but A == glm::inverse(M)
Cause: in glm/gtc/matrix_inverse.inl, the mat<2, 2, T, Q> overload constructs the same matrix as inverse(mat2) (func_matrix.inl), the two middle cofactors are never swapped.
Not a recent regression: the same implementation appears in the original glm/gtx/inverse_transpose.inl (created 2006) and was carried unchanged through the GTC promotion (cd183ed, 2010). Present on master and in 1.0.3. Configuration-independent (pure arithmetic); observed with clang on macOS/aarch64.
PR to follow with the one-line fix plus a regression test in test/gtc/gtc_matrix_inverse.cpp.
glm::inverseTransposeformat2returns the plain inverse, the transpose step is missing. Themat3andmat4overloads are correct.Cause: in
glm/gtc/matrix_inverse.inl, themat<2, 2, T, Q>overload constructs the same matrix asinverse(mat2)(func_matrix.inl), the two middle cofactors are never swapped.Not a recent regression: the same implementation appears in the original
glm/gtx/inverse_transpose.inl(created 2006) and was carried unchanged through the GTC promotion (cd183ed, 2010). Present onmasterand in 1.0.3. Configuration-independent (pure arithmetic); observed with clang on macOS/aarch64.PR to follow with the one-line fix plus a regression test in
test/gtc/gtc_matrix_inverse.cpp.