What this usually means
At its core, SAML SSO failure after a seemingly working setup is almost always one of three things: certificate/key rotation without corresponding update on the other side (most common), clock skew between IdP and SP exceeding the allowed tolerance (typically 5 minutes), or a mismatch in the NameID format or attribute mapping that causes the SP to reject the assertion. I've also seen cases where the IdP updates its metadata (adds a new signing certificate) but the SP only validates against the old certificate fingerprint. The SAML XML itself may be valid, but the trust chain is broken. Another frequent cause is the relay state being lost or tampered with, especially when proxies or load balancers strip cookies or rewrite URLs.
The first ten minutes — establish facts before touching code.
- 1Check the SP's SAML debug log: tail -f /var/log/tomcat/saml.log | grep -i 'error\|fail\|invalid'
- 2Decode the SAML response from the browser: use SAML-tracer extension or copy the SAMLResponse parameter from the POST form, base64-decode it, and view the XML for NotBefore/NotOnOrAfter and issuer
- 3Compare the certificate fingerprint in the SAML response with the SP's configured IdP certificate: openssl x509 -in idp.crt -noout -fingerprint -sha256
- 4Verify clock skew: ntpq -p on both IdP and SP servers, ensure within 30 seconds
- 5Check the SP's metadata cache: force a refresh by restarting the SP or clearing the cache directory (e.g., rm -rf /var/cache/shibboleth/*)
The specific files, logs, configs, and dashboards that usually own this bug.
- searchSP logs: /var/log/shibboleth/shibd.log, /var/log/tomcat/saml.log, /var/log/httpd/error_log
- searchIdP logs: /opt/shibboleth-idp/logs/idp-process.log, /var/log/shibboleth-idp/messages.log
- searchSAML response (via browser dev tools): Network tab -> filter by 'SAMLResponse' -> copy value -> base64 decode
- searchSP metadata file: /etc/shibboleth/shibboleth2.xml or /opt/sp/conf/metadata.xml
- searchIdP metadata file: /opt/shibboleth-idp/metadata/idp-metadata.xml
- searchSystem time: timedatectl status on both servers
- searchCertificate files: /etc/shibboleth/idp-signing.crt, /etc/shibboleth/sp-signing.crt
Practical causes, not theory. These are the things you will actually find.
- warningIdP rotated its signing certificate but SP still validates against the old certificate fingerprint
- warningClock skew between IdP and SP exceeds the allowed lifetime of the assertion (typically 5 minutes)
- warningNameID format mismatch: SP expects 'emailAddress' but IdP sends 'unspecified' or 'persistent'
- warningRelay state parameter is lost or modified due to URL encoding issues or load balancer rewriting
- warningSP metadata entity ID mismatch: IdP sends assertion for wrong SP (e.g., staging vs. production)
- warningIdP metadata cache is stale: SP hasn't fetched updated metadata after IdP changed endpoints or certificates
Concrete fix directions. Pick the one that matches your root cause.
- buildUpdate the SP's IdP certificate: copy the new signing certificate from the IdP metadata and restart the SP service
- buildSynchronize clocks: set up NTP on both sides and ensure time drift is under 30 seconds
- buildFix NameID format: in the SP config, specify the expected format (e.g., urn:oasis:names:tc:SAML:1.1:nameid-format:emailAddress) and ensure IdP sends that
- buildPreserve relay state: if using a reverse proxy, ensure it doesn't strip or modify the RelayState parameter; use POST binding instead of redirect to avoid URL truncation
- buildForce metadata refresh: configure SP to fetch IdP metadata periodically or trigger a manual refresh after IdP changes
- buildIncrease assertion lifetime: temporarily set NotOnOrAfter to 30 minutes in IdP config while debugging, but revert after fix
A fix you cannot prove is a guess. Close the loop.
- verifiedPerform a full SAML login flow and capture the SAMLResponse; decode and check that NotBefore < current time < NotOnOrAfter and that signature validates with SP's configured IdP certificate
- verifiedRun a time sync check: ntpdate -q pool.ntp.org on both servers and confirm difference < 1 second
- verifiedLog in as a test user and confirm the SP receives the expected attributes (e.g., email, roles) by checking SP logs or a test endpoint
- verifiedUse a SAML validator tool like samltool.io to verify the response structure and signature
- verifiedSimulate a clock skew by setting the SP server time 10 minutes ahead and confirm login fails, then correct time and confirm success
Things that make this bug worse or harder to find.
- warningDon't blindly replace the entire SAML config; change only the certificate or the specific attribute mapping
- warningDon't ignore the audience restriction: ensure the SP entity ID in the metadata exactly matches what the IdP sends in the assertion
- warningDon't disable signature validation for testing and then forget to re-enable it in production
- warningDon't assume base64 decode is enough: the SAML response may be deflated (compressed) – look for < in the decoded output; if not, it's likely deflated and needs gunzip
- warningDon't copy certificates from a browser view; always get them from the official IdP metadata XML to avoid truncation or encoding issues
- warningDon't forget to check the SAML response for 'AuthnContextClassRef' if the SP requires a specific authentication method
Certificate rotation takes down SSO for 3000 users
Timeline
- 08:15IdP team rotates signing certificate as part of quarterly key rotation.
- 08:20First user reports SSO failure: 'The page is blank after login.'
- 08:25PagerDuty alert: SAML errors in SP logs (signature validation failed).
- 08:30I check the IdP metadata; notice new signing certificate and old one marked as expired.
- 08:35I decode a SAML response: NotBefore 08:20, NotOnOrAfter 08:25, signature uses the new certificate.
- 08:40I open SP's shibboleth2.xml: the idp.crt file points to old certificate. I copy the new certificate from IdP metadata.
- 08:45Restart shibd and Apache. Test login with my account: works.
- 08:50Monitor logs: errors stop. All 3000 users regain access within 5 minutes.
- 09:00Postmortem: lack of automated certificate rotation check in SP; manual step was missed.
We had a quarterly certificate rotation policy. The IdP team ran their script, which updated the signing certificate and published new metadata. They didn't notify the SP team because they assumed the SP would automatically pick up the new certificate from the metadata. Our SP was configured to validate against a static file copy, not the live metadata feed.
When the first user reported the blank page, I initially suspected a network issue. But the error rate spiked in our monitoring dashboard. I pulled up the SP logs and saw the same error repeated: 'SAML response signature validation failed'. I decoded a SAML response using samltool.io and immediately saw the certificate serial number was different from what our SP had.
I copied the new certificate from the IdP metadata XML into our SP's certificate file, restarted shibd and Apache, and tested. Login worked. The fix took 15 minutes, but the outage lasted 25 minutes because we didn't have an automated check. I added a cron job that compares certificate fingerprints daily and alerts if they differ.
Root cause
SP was configured with a static IdP signing certificate and did not automatically fetch updated metadata after the IdP rotated its certificate.
The fix
Updated the SP's idp.crt file with the new certificate from the IdP metadata, then restarted shibd and Apache. Added metadata refresh automation.
The lesson
Always enable automatic metadata refresh or have a process to sync certificates. A simple fingerprint comparison alert would have caught this in minutes.
The SP validates a SAML response in three steps: (1) It checks the signature using the IdP's public key; (2) It verifies the assertion's NotBefore and NotOnOrAfter timestamps; (3) It confirms the audience (the SP's entity ID) matches. At each step, a mismatch causes a failure.
The most common break point is step 1: the SP has an old certificate. The IdP signs with the new private key, but the SP tries to validate with the old public key. The signature check fails with 'signature validation failed' in the logs. Step 2 fails when clocks drift. Even a 5-minute skew can cause 'SAML response expired' errors. Step 3 fails if the SP entity ID in the metadata doesn't match the one in the assertion, often due to copy-paste errors.
Open browser dev tools, go to Network tab, filter by 'SAMLResponse', and copy the value from the POST body. It's a base64-encoded string. Decode it with: echo '<base64>' | base64 -d. If the output is binary or starts with a strange character, it's likely deflated (compressed). In that case, use: echo '<base64>' | base64 -d | python3 -c 'import sys,zlib; sys.stdout.buffer.write(zlib.decompress(sys.stdin.buffer.read()))'.
Once you have readable XML, look for the <Signature> element and extract the <X509Certificate> value. Compare its fingerprint with what your SP has: openssl x509 -in /path/to/idp.crt -noout -fingerprint -sha256. They must match. Also check the <SubjectConfirmationData> NotBefore and NotOnOrAfter – they should be in UTC and cover the time of login.
Sometimes the SAML response is valid but the SP still rejects the login because the NameID format is wrong. For example, the IdP sends <NameID Format="urn:oasis:names:tc:SAML:1.1:nameid-format:unspecified"> but the SP expects emailAddress. Check the SP config for <NameIDPolicy> or attribute mapping – in Shibboleth, it's in attribute-map.xml.
Another subtle issue: the IdP may send attributes with wrong names (e.g., 'mail' instead of 'email'). The SP will silently drop them if not mapped. Look for 'Attribute not recognized' warnings in the SP logs. Fix by updating attribute-map.xml to map the incoming attribute name to the internal name.
The RelayState parameter is used to redirect the user back to the original page after login. If a load balancer or reverse proxy strips or modifies query parameters, the relay state can become invalid. Symptoms: user logs in successfully but gets redirected to a generic page or an error page saying 'Invalid request'.
To debug: capture the RelayState value from the initial redirect to the IdP and compare with the value returned in the POST. They should match. If not, the proxy is likely the culprit. Use POST binding for SP-initiated login to avoid URL length limits. Also ensure the SP's session cookie is set correctly and not blocked by the proxy.
Frequently asked questions
Why does SAML login fail for some users but not others?
If login works for some users but not others, the issue is likely attribute mapping or NameID format. For example, if the IdP sends a persistent NameID for new users but a transient one for old users, the SP may reject one format. Another possibility: the SP has a whitelist of allowed issuers or audience restrictions that vary per user group. Check the SAML response for each user type to compare.
How do I fix 'Audience mismatch' error in SAML?
The audience mismatch error means the SP's entity ID in the SAML assertion does not match what the SP expects. To fix, compare the entityID in the IdP metadata (for the SP) with the entityID configured in the SP's metadata. They must be identical, including trailing slashes. Edit the metadata on either side to match, then restart the SP.
What does 'SAML response expired' mean even though I just logged in?
This error usually indicates clock skew between the IdP and SP. The SAML assertion has a NotBefore and NotOnOrAfter timestamp, typically a 5-minute window. If the SP's clock is more than 5 minutes off from the IdP's clock, the assertion appears expired. Fix by syncing both servers to the same NTP source. You can also temporarily increase the assertion lifetime in the IdP config for testing.
How to check if the IdP certificate is the correct one?
Download the IdP metadata XML from the IdP's metadata URL. The signing certificate is inside the <ds:X509Certificate> element. Copy that entire base64 string into a file, then decode to PEM format and compare the fingerprint with your SP's certificate file. Use 'openssl x509 -in idp.crt -noout -fingerprint -sha256' on both.
Should I use HTTP-POST or HTTP-Redirect binding for SAML?
HTTP-POST is generally preferred for production because it can handle larger SAML responses without URL truncation. HTTP-Redirect is simpler but limited to about 8KB. If you use redirect, the RelayState can be lost or truncated. I recommend POST for SP-initiated login and Redirect only for IdP-initiated where the assertion is sent via a form.