域名反向代理外部HTTPS网站报错，提示transport failure reason: TLS error

[limityu]

我正在进行测试，例如域名 harbor.litye.cn我想让他反向代理到外部网站 3pi.xxxx.com，但是提示upstream connect error or disconnect/reset before headers. reset reason: connection failure, transport failure reason: TLS error: 33554536:system library:OPENSSL_internal:Connection reset by peer
版本: v1.3.3
具体操作如下：
higress生成的ingress:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    higress.io/backend-protocol: HTTPS
    higress.io/destination: 3pi.dns
    higress.io/enable-rewrite: "true"
    higress.io/ignore-path-case: "false"
    higress.io/ssl-redirect: "true"
    higress.io/upstream-vhost: 3pi.stg.starbucks.com.cn
  creationTimestamp: "2024-04-24T03:31:23Z"
  generation: 1
  labels:
    higress.io/domain_harbor.litye.cn: "true"
    higress.io/resource-definer: higress
  name: harbor.litye.cn
  namespace: higress-system
  resourceVersion: "11184920"
  uid: bca45ad1-f4fa-4c1b-9650-9c4648093217
spec:
  ingressClassName: higress
  rules:
  - host: harbor.litye.cn
    http:
      paths:
      - backend:
          resource:
            apiGroup: networking.higress.io
            kind: McpBridge
            name: default
        path: /
        pathType: Prefix
  tls:
  - hosts:
    - harbor.litye.cn
    secretName: litye.cn
status:
  loadBalancer: {}

配置中涉及的3pi.dns截图

从higress-gateway直接访问外部3pi.xxx.com地址

istio-proxy@higress-gateway-55c57cf4-5jbnc:/$ curl -v https://3pi.xxxxx.cn
*   Trying 101.52.x.x:443...
* Connected to 3pi.xxxxx.cn (101.52.x.x) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.2 (OUT), TLS header, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=US; ST=Washington; O=Starbucks Coffee Company; CN=3pi.xxxxx.cn
*  start date: Sep 27 00:00:00 2023 GMT
*  expire date: Sep 26 23:59:59 2024 GMT
*  subjectAltName: host "3pi.xxxxx.cn" matched cert's "3pi.xxxxx.cn"
*  issuer: C=GB; ST=Greater Manchester; L=Salford; O=Sectigo Limited; CN=Sectigo RSA Organization Validation Secure Server CA
*  SSL certificate verify ok.
* Using HTTP2, server supports multiplexing
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* Using Stream ID: 1 (easy handle 0x55d4e8433e90)
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
> GET / HTTP/2
> Host: 3pi.xxxxx.cn
> user-agent: curl/7.81.0
> accept: */*
> 
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
< HTTP/2 404 
< date: Thu, 25 Apr 2024 03:23:28 GMT
< content-type: application/json; charset=utf-8
< content-length: 48
< vary: Origin
< access-control-allow-credentials: true
< x-xss-protection: 1;mode=block
< strict-transport-security: max-age=31536000;inculdeSubDomains
< x-server-by: gds-stg-ext-os
< x-transaction-id: 331e0b1d-b175-4be0-9b76-7c155208a403
< 
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* Connection #0 to host 3pi.xxxxx.cn left intact
{"message":"no Route matched with those values"}

从higress-gateway访问反向代理后的地址：

istio-proxy@higress-gateway-55c57cf4-5jbnc:/$ curl -v https://harbor.litye.cn
*   Trying 10.10.x.x:443...
* Connected to harbor.litye.cn (10.10.x.x) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.2 (OUT), TLS header, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=*.litye.cn
*  start date: Jan 31 00:00:00 2024 GMT
*  expire date: Apr 30 23:59:59 2024 GMT
*  subjectAltName: host "harbor.litye.cn" matched cert's "*.litye.cn"
*  issuer: C=AT; O=ZeroSSL; CN=ZeroSSL ECC Domain Secure Site CA
*  SSL certificate verify ok.
* Using HTTP2, server supports multiplexing
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* Using Stream ID: 1 (easy handle 0x560b94dace90)
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
> GET / HTTP/2
> Host: harbor.litye.cn
> user-agent: curl/7.81.0
> accept: */*
> 
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
< HTTP/2 503 
< content-length: 195
< content-type: text/plain
< date: Thu, 25 Apr 2024 03:26:16 GMT
< server: istio-envoy
< 
* Connection #0 to host harbor.litye.cn left intact
upstream connect error or disconnect/reset before headers. reset reason: connection failure, transport failure reason: TLS error: 33554536:system library:OPENSSL_internal:Connection reset by peer

[CH3CHO]

可能和对方的SNI配置有关。试一下不使用域名，直接用IP连接目标的443端口，或者使用你们的域名作为Host进行连接。

可以参考一下这个：istio/istio#43540

[limityu]

反向代理为目标域名的IP时，已经尝试过了重写主机修改为对应域名或者IP。报错依旧是这样

[CH3CHO]

那你curl的时候加上SNI的配置能不能成功请求呢？

[limityu]

报错用到对方的TLS证书了,我本意为用自己的证书去加密内容而不是对方的

当前higress设置harbor.litye.cn 代理为目标域名的IP，重写主机为目标域名的IP

# 101.x.x.8 表示目标域名的IP
istio-proxy@higress-gateway-8447dbb948-kf7sb:/$ curl -vv  --resolve harbor.litye.cn:443:101.x.x.8  https://harbor.litye.cn 
* Added harbor.litye.cn:443:101.x.x.8 to DNS cache
* Hostname harbor.litye.cn was found in DNS cache
*   Trying 101.x.x.8:443...
* Connected to harbor.litye.cn (101.x.x.8) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.2 (OUT), TLS header, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=US; ST=Washington; O=xxx Company; CN=www.xxx.com.cn
*  start date: Jun 29 00:00:00 2023 GMT
*  expire date: Jun 28 23:59:59 2024 GMT
*  subjectAltName does not match harbor.litye.cn
* SSL: no alternative certificate subject name matches target host name 'harbor.litye.cn'
* Closing connection 0
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.3 (OUT), TLS alert, close notify (256):
curl: (60) SSL: no alternative certificate subject name matches target host name 'harbor.litye.cn'
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

[CH3CHO]

你这个测试方法不太对。

curl 默认是会校验服务端证书的。你请求用的 host 是 harbor.litye.cn，而服务端返回的服务证书不包含这个域名，所以校验失败。

另外，如果 subject: C=US; ST=Washington; O=xxx Company; CN=www.xxx.com.cn 这一行你没有做过任何处理的话，那说明服务端确实是有 SNI，不同的请求域名使用不同的证书。

不过你确实可以参考上面那个 issue 里的建议，“Also for TLS debugging you may need to turn on trace level logging in order to capture more of the error.”，看看能不能抓下什么详细的日志。也可以抓一下 Gateway 目标域名之间的网络包，看看到底是哪儿断的。

[limityu]

感谢回复，具体操作后续我再研究下