Haider,
Hopefully you found the solution by yourself. I had exact same problem yesterday at a customer's place for back-channel communication between ADFS 3.0 and SAP, just like yours.
It ended up being a networking issue of asynchronous routing. Due to different paths taken by the request and response packets during SSL handshake, the SOAP communication failed.
We could reproduce the error by testing the SAML specific HTTP RFC connection. Once we fixed the network paths ( simply by maintaining HOSTS file in SAP GW server host) everything fell in place.
-Subhra