Hi Matthias,
Perfect .... Your error is coming from the Websphere server and NOT SAP.
You can see from the openssl command I asked to be executed that the Websphere server is the one throwing the SSL handshake error.
It looks to me that either you are using self signed certificates on this server or you don't have the complete certificate chain on this server. Can you get the admin of this server to confirm this.
Kindest Regards,
Amerjit