That audience restriction is generated because we originated at the ADFS logon screen and then chose our Relying Party from the pull down.
But our design has been changed since submitting my post. We changed our config to do WS-Trust since it's a java app making web service calls to SAP at the SOA layer.
You can setup your SAML debug at:
http://host:port/sap/bc/webdynpro/sap/sec_diag_tool?sap-client=####