Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Unexpected error log #405

Open
hzxuzhonghu opened this issue May 31, 2024 · 5 comments
Open

Unexpected error log #405

hzxuzhonghu opened this issue May 31, 2024 · 5 comments
Labels
kind/bug Something isn't working

Comments

@hzxuzhonghu
Copy link
Member

hzxuzhonghu commented May 31, 2024 •
edited
Loading

What happened:

root@kurator-linux-0001:~/go/src/kmesh.net/kmesh# k logs kmesh-j6tfz  -n kmesh-system
time="2024-05-31T02:51:47Z" level=info msg="FLAG: --bpf-fs-path=\"/sys/fs/bpf\"" subsys=manager
time="2024-05-31T02:51:47Z" level=info msg="FLAG: --cgroup2-path=\"/mnt/kmesh_cgroup2\"" subsys=manager
time="2024-05-31T02:51:47Z" level=info msg="FLAG: --cni-etc-path=\"/etc/cni/net.d\"" subsys=manager
time="2024-05-31T02:51:47Z" level=info msg="FLAG: --conflist-name=\"\"" subsys=manager
time="2024-05-31T02:51:47Z" level=info msg="FLAG: --enable-bypass=\"false\"" subsys=manager
time="2024-05-31T02:51:47Z" level=info msg="FLAG: --enable-mda=\"false\"" subsys=manager
time="2024-05-31T02:51:47Z" level=info msg="FLAG: --help=\"false\"" subsys=manager
time="2024-05-31T02:51:47Z" level=info msg="FLAG: --mode=\"workload\"" subsys=manager
time="2024-05-31T02:51:47Z" level=info msg="FLAG: --plugin-cni-chained=\"true\"" subsys=manager
time="2024-05-31T02:51:47Z" level=info msg="bpf Start successful" subsys=manager
time="2024-05-31T02:51:47Z" level=info msg="service node ztunnel~10.244.0.219~kmesh-j6tfz.kmesh-system~kmesh-system.svc.cluster.local connect to discovery address istiod.istio-system.svc:15012" subsys=controller/config
time="2024-05-31T02:51:47Z" level=info msg="controller Start successful" subsys=manager
time="2024-05-31T02:51:47Z" level=info msg="start write CNI config\n" subsys="cni installer"
time="2024-05-31T02:51:47Z" level=info msg="kmesh cni use chained\n" subsys="cni installer"
time="2024-05-31T02:51:47Z" level=error msg="workload Kubernetes//Pod/default/bookinfo-reviews-istio-waypoint-86b5c9cd7d-fhfkc not found" subsys=workload_controller
time="2024-05-31T02:51:47Z" level=error msg="workload Kubernetes//Pod/default/productpage-v1-675fc69cf-q2h9b not found" subsys=workload_controller
time="2024-05-31T02:51:47Z" level=error msg="workload Kubernetes//Pod/default/reviews-v2-5b667bcbf8-cscb8 not found" subsys=workload_controller
time="2024-05-31T02:51:47Z" level=error msg="workload Kubernetes//Pod/istio-system/istiod-57dfcb6555-sw277 not found" subsys=workload_controller
time="2024-05-31T02:51:47Z" level=error msg="workload Kubernetes//Pod/kmesh-system/kmesh-j6tfz not found" subsys=workload_controller
time="2024-05-31T02:51:47Z" level=error msg="workload Kubernetes//Pod/kube-system/coredns-76f75df574-kcvsb not found" subsys=workload_controller
time="2024-05-31T02:51:47Z" level=error msg="workload Kubernetes//Pod/kube-system/coredns-76f75df574-r8pxv not found" subsys=workload_controller
time="2024-05-31T02:51:47Z" level=error msg="workload Kubernetes//Pod/local-path-storage/local-path-provisioner-7577fdbbfb-lhqqv not found" subsys=workload_controller
time="2024-05-31T02:51:47Z" level=error msg="workload Kubernetes//Pod/default/reviews-v3-5b9bd44f4-vzrvp not found" subsys=workload_controller
time="2024-05-31T02:51:47Z" level=error msg="workload Kubernetes//Pod/default/details-v1-698d88b-m2kh7 not found" subsys=workload_controller
time="2024-05-31T02:51:47Z" level=error msg="workload Kubernetes//Pod/default/reviews-v1-5b5d6494f4-c5n7n not found" subsys=workload_controller
time="2024-05-31T02:51:47Z" level=error msg="workload Kubernetes//Pod/istio-system/istio-ingressgateway-7bf844ffb6-w9rfc not found" subsys=workload_controller
time="2024-05-31T02:51:47Z" level=error msg="workload Kubernetes//Pod/istio-system/ztunnel-5hftj not found" subsys=workload_controller
time="2024-05-31T02:51:47Z" level=error msg="workload Kubernetes//Pod/default/sleep-7656cf8794-fcqp7 not found" subsys=workload_controller
time="2024-05-31T02:51:47Z" level=error msg="workload Kubernetes//Pod/default/ratings-v1-6484c4d9bb-k7r5d not found" subsys=workload_controller
time="2024-05-31T02:51:47Z" level=error msg="workload Kubernetes//Pod/default/httpbin-86b8ffc5ff-czmdb not found" subsys=workload_controller
time="2024-05-31T02:51:49Z" level=error msg="create certificate: rpc error: code = Unauthenticated desc = request impersonation authentication failure reconnect..." subsys=security
time="2024-05-31T02:51:49Z" level=error msg="create certificate: rpc error: code = Unauthenticated desc = request impersonation authentication failure reconnect..." subsys=security
time="2024-05-31T02:51:49Z" level=error msg="fetcheCert spiffe:///ns/kube-system/sa/coredns error: failed to get certChainPEM" subsys=security
time="2024-05-31T02:51:49Z" level=error msg="fetcheCert spiffe:///ns/default/sa/bookinfo-productpage error: failed to get certChainPEM" subsys=security
time="2024-05-31T02:51:50Z" level=info msg="Copied /usr/bin/kmesh-cni to /opt/cni/bin." subsys="cni installer"
time="2024-05-31T02:51:50Z" level=info msg="kubeconfig either does not exist or is out of date, writing a new one" subsys="cni installer"
time="2024-05-31T02:51:50Z" level=info msg="wrote kubeconfig file /etc/cni/net.d/kmesh-cni-kubeconfig" subsys="cni installer"
time="2024-05-31T02:51:50Z" level=info msg="cni config file: /etc/cni/net.d/10-kindnet.conflist" subsys="cni installer"
time="2024-05-31T02:51:50Z" level=info msg="command Start cni successful" subsys=manager
time="2024-05-31T02:51:50Z" level=error msg="create certificate: rpc error: code = Unauthenticated desc = request impersonation authentication failure reconnect..." subsys=security
time="2024-05-31T02:51:50Z" level=error msg="create certificate: rpc error: code = Canceled desc = grpc: the client connection is closing reconnect..." subsys=security
time="2024-05-31T02:51:50Z" level=error msg="fetcheCert spiffe:///ns/default/sa/bookinfo-details error: failed to get certChainPEM" subsys=security
time="2024-05-31T02:51:50Z" level=error msg="create certificate: rpc error: code = Unauthenticated desc = request impersonation authentication failure reconnect..." subsys=security
time="2024-05-31T02:51:50Z" level=error msg="create certificate: rpc error: code = Canceled desc = grpc: the client connection is closing reconnect..." subsys=security
time="2024-05-31T02:51:50Z" level=error msg="fetcheCert spiffe:///ns/kmesh-system/sa/kmesh error: failed to get certChainPEM" subsys=security
time="2024-05-31T02:51:50Z" level=error msg="fetcheCert spiffe:///ns/default/sa/httpbin error: failed to get certChainPEM" subsys=security
time="2024-05-31T02:51:50Z" level=error msg="create certificate: rpc error: code = Unauthenticated desc = request impersonation authentication failure reconnect..." subsys=security
time="2024-05-31T02:51:50Z" level=error msg="fetcheCert spiffe:///ns/local-path-storage/sa/local-path-provisioner-service-account error: failed to get certChainPEM" subsys=security
time="2024-05-31T02:51:51Z" level=error msg="fetcheCert spiffe:///ns/istio-system/sa/istio-ingressgateway-service-account error: failed to get certChainPEM" subsys=security
time="2024-05-31T02:51:51Z" level=error msg="create certificate: rpc error: code = Unauthenticated desc = request impersonation authentication failure reconnect..." subsys=security
time="2024-05-31T02:51:51Z" level=error msg="create certificate: rpc error: code = Unauthenticated desc = request impersonation authentication failure reconnect..." subsys=security
time="2024-05-31T02:51:51Z" level=error msg="fetcheCert spiffe:///ns/istio-system/sa/istiod error: failed to get certChainPEM" subsys=security
time="2024-05-31T02:51:51Z" level=error msg="create certificate: rpc error: code = Unauthenticated desc = request impersonation authentication failure reconnect..." subsys=security
time="2024-05-31T02:51:51Z" level=error msg="fetcheCert spiffe:///ns/default/sa/bookinfo-reviews error: failed to get certChainPEM" subsys=security
time="2024-05-31T02:51:51Z" level=error msg="fetcheCert spiffe:///ns/default/sa/bookinfo-ratings error: failed to get certChainPEM" subsys=security
time="2024-05-31T02:51:51Z" level=error msg="create certificate: rpc error: code = Unauthenticated desc = request impersonation authentication failure reconnect..." subsys=security
time="2024-05-31T02:51:51Z" level=error msg="fetcheCert spiffe:///ns/istio-system/sa/ztunnel error: failed to get certChainPEM" subsys=security
time="2024-05-31T02:51:51Z" level=error msg="create certificate: rpc error: code = Unauthenticated desc = request impersonation authentication failure reconnect..." subsys=security
time="2024-05-31T02:51:51Z" level=error msg="fetcheCert spiffe:///ns/default/sa/bookinfo-reviews-istio-waypoint error: failed to get certChainPEM" subsys=security
time="2024-05-31T02:51:52Z" level=error msg="create certificate: rpc error: code = Unauthenticated desc = request impersonation authentication failure reconnect..." subsys=security
time="2024-05-31T02:51:52Z" level=error msg="fetcheCert spiffe:///ns/default/sa/sleep error: failed to get certChainPEM" subsys=security

What you expected to happen:

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Environment:

  • Kmesh version:
    kmesh-daemon version
    {
    "gitVersion": "1.0-dev",
    "gitCommit": "12b8d0cb37809d17ad49f52d6d0ac8e2c0701f52",
    "gitTreeState": "clean",
    "buildDate": "2024-05-31T02:41:37Z",
    "goVersion": "go1.22.1",
    "compiler": "gc",
    "platform": "linux/amd64"
    }

  • Others:
@hzxuzhonghu hzxuzhonghu added the kind/bug Something isn't working label May 31, 2024
@hzxuzhonghu
Copy link
Member Author

@lec-bit Can you take a look

@hzxuzhonghu
Copy link
Member Author

Istiod log

2024-05-31T03:33:39.615011Z	info	ads	Push debounce stable[10] 1 for config Address//Kubernetes//Pod/kmesh-system/kmesh-756l6: 104.422892ms since last change, 104.422763ms since last push, full=false
2024-05-31T03:33:39.615136Z	info	ads	XDS: Incremental Pushing ConnectedEndpoints:4 Version:2024-05-31T03:30:16Z/3
2024-05-31T03:33:41.058828Z	warn	serverca	impersonation failed for identity spiffe:///ns/istio-system/sa/ztunnel, error: no node authorizer for cluster 	client=10.244.0.252:60882
2024-05-31T03:33:41.667091Z	warn	serverca	impersonation failed for identity spiffe:///ns/istio-system/sa/ztunnel, error: no node authorizer for cluster 	client=10.244.0.252:60896
2024-05-31T03:33:41.669378Z	warn	serverca	impersonation failed for identity spiffe:///ns/default/sa/sleep, error: no node authorizer for cluster 	client=10.244.0.252:60896
2024-05-31T03:33:41.669574Z	warn	serverca	impersonation failed for identity spiffe:///ns/default/sa/bookinfo-productpage, error: no node authorizer for cluster 	client=10.244.0.252:60896
2024-05-31T03:33:41.669738Z	warn	serverca	impersonation failed for identity spiffe:///ns/default/sa/bookinfo-reviews, error: no node authorizer for cluster 	client=10.244.0.252:60896
2024-05-31T03:33:42.355468Z	warn	serverca	impersonation failed for identity spiffe:///ns/default/sa/bookinfo-reviews, error: no node authorizer for cluster 	client=10.244.0.252:60908
2024-05-31T03:33:42.355718Z	warn	serverca	impersonation failed for identity spiffe:///ns/default/sa/httpbin, error: no node authorizer for cluster 	client=10.244.0.252:60908
2024-05-31T03:33:42.355883Z	warn	serverca	impersonation failed for identity spiffe:///ns/default/sa/bookinfo-details, error: no node authorizer for cluster 	client=10.244.0.252:60908
2024-05-31T03:33:42.568300Z	warn	serverca	impersonation failed for identity spiffe:///ns/default/sa/bookinfo-details, error: no node authorizer for cluster 	client=10.244.0.252:60914
2024-05-31T03:33:42.569249Z	warn	serverca	impersonation failed for identity spiffe:///ns/kube-system/sa/coredns, error: no node authorizer for cluster 	client=10.244.0.252:60914
2024-05-31T03:33:42.655705Z	warn	serverca	impersonation failed for identity spiffe:///ns/kube-system/sa/coredns, error: no node authorizer for cluster 	client=10.244.0.252:60916
2024-05-31T03:33:42.657634Z	warn	serverca	impersonation failed for identity spiffe:///ns/istio-system/sa/istiod, error: no node authorizer for cluster 	client=10.244.0.252:60916
2024-05-31T03:33:42.665555Z	warn	serverca	impersonation failed for identity spiffe:///ns/istio-system/sa/istiod, error: no node authorizer for cluster 	client=10.244.0.252:60932

@lec-bit
Copy link
Contributor

lec-bit commented May 31, 2024

The query operation is performed before the add operation. As a result, the query fails and a large number of errors are reported.

@carlory
Copy link

carlory commented Jun 5, 2024

kmesh/pkg/controller/security/caclient.go

Line 91 in 76f66df

resp, err := c.client.CreateCertificate(ctx, req)

The ctx doesn't contain ClusterID key.

FYI https://github.com/istio/istio/blob/master/security/pkg/nodeagent/caclient/providers/citadel/client.go#L107C59-L107C105

@hzxuzhonghu
Copy link
Member Author

Fixed it in #416

@hzxuzhonghu hzxuzhonghu mentioned this issue Jun 7, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
kind/bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@hzxuzhonghu @carlory @lec-bit