Access failed to fetch information from IdP

I’m using ADFS as my IdP but unfortunately no matter how I configured it, the error

Failed to fetch user/group information from the identity provider
  "name": "saml",
  "idpType": "saml",
  "originalError": "none"

shows up. The error message doesn’t seem to be helpful since the email claim does exist when I decode one of the SAML response.

I found the issue. Apparently, there is a bug that causes the metadata parser uses the encryption certificate instead of signing certificate.