Kubernetes - upgrade from v13 to v14 failed

[pixelfields]

Hello everyone,

We're facing an issue during the upgrade from v13.4.26 to v14.0.0/14.0.0.3.

Deployment type: Kubernetes / Helm Chart

Chart values

acme: false
acmeEmail: ""
acmeURI: ""
affinity: {}
annotations:
  certSecret: {}
  config: {}
  deployment: {}
  ingress: {}
  pod: {}
  service:
    service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
    service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing
    service.beta.kubernetes.io/aws-load-balancer-subnets: subnet-[REDACTED],subnet-[REDACTED],subnet-[REDACTED]
    service.beta.kubernetes.io/aws-load-balancer-type: external
  serviceAccount: {}
auth:
  teleportConfig: {}
authentication:
  connectorName: ""
  localAuth: true
  lockingMode: ""
  secondFactor: "on"
  type: local
aws:
  auditLogMirrorOnStdout: false
  auditLogTable: teleport-events
  backendTable: teleport-backend
  backups: true
  dynamoAutoScaling: false
  readMaxCapacity: null
  readMinCapacity: null
  readTargetValue: null
  region: eu-central-1
  sessionRecordingBucket: osp-teleport-sessions
  writeMaxCapacity: null
  writeMinCapacity: null
  writeTargetValue: null
azure:
  auditLogDatabase: teleport_audit
  auditLogMirrorOnStdout: false
  backendDatabase: teleport_backend
  clientID: ""
  databaseHost: ""
  databasePoolMaxConnections: 0
  databaseUser: ""
  sessionRecordingStorageAccount: ""
chartMode: aws
clusterName: teleport.domain.de
createProxyToken: true
enterprise: false
enterpriseImage: public.ecr.aws/gravitational/teleport-ent-distroless
extraArgs: []
extraContainers: []
extraEnv: []
extraVolumeMounts: []
extraVolumes: []
gcp:
  auditLogMirrorOnStdout: false
  auditLogTable: ""
  backendTable: ""
  credentialSecretName: teleport-gcp-credentials
  projectId: ""
  sessionRecordingBucket: ""
global:
  clusterDomain: cluster.local
highAvailability:
  certManager:
    addCommonName: false
    addPublicAddrs: false
    enabled: true
    issuerGroup: cert-manager.io
    issuerKind: Issuer
    issuerName: letsencrypt-teleport-infrastructure
  minReadySeconds: 15
  podDisruptionBudget:
    enabled: false
    minAvailable: 1
  replicaCount: 1
  requireAntiAffinity: false
image: public.ecr.aws/gravitational/teleport-distroless
imagePullPolicy: IfNotPresent
imagePullSecrets: []
ingress:
  enabled: false
  spec: {}
  suppressAutomaticWildcards: false
initContainers: []
kubeClusterName: ""
kubePublicAddr: []
labels: {}
log:
  extraFields:
  - timestamp
  - level
  - component
  - caller
  format: text
  level: INFO
  output: stderr
mongoPublicAddr: []
mysqlPublicAddr: []
nodeSelector: {}
operator:
  enabled: true
  image: public.ecr.aws/gravitational/teleport-operator
  resources: {}
persistence:
  enabled: true
  existingClaimName: ""
  volumeSize: 10Gi
podMonitor:
  additionalLabels:
    prometheus: default
  enabled: false
  interval: 30s
podSecurityPolicy:
  enabled: true
postStart:
  command: []
postgresPublicAddr: []
priorityClassName: ""
probeTimeoutSeconds: 1
proxy:
  teleportConfig: {}
proxyListenerMode: separate
publicAddr: []
rbac:
  create: true
resources: {}
securityContext: {}
separateMongoListener: false
separatePostgresListener: false
service:
  spec: {}
  type: LoadBalancer
serviceAccount:
  create: true
  name: ""
sessionRecording: ""
sshPublicAddr: []
teleport-operator:
  global:
    clusterDomain: cluster.local
teleportVersionOverride: ""
terminationGracePeriodSeconds: 60
tls:
  existingCASecretName: ""
  existingSecretName: ""
tolerations: []
tunnelPublicAddr: []
validateConfigOnDeploy: true

After the upgrade from 13.4.26 to 14.0.0 (but the same for 14.0.3) we are getting 2 types of errors:

1/ coming from operator

operator Original Error: *trace.BadParameterError "db_client" authority type is not supported                                                                                                                                                                          │
│ operator Stack Trace:                                                                                                                                                                                                                                                          │
│ operator     /go/src/github.com/gravitational/teleport/api/client/client.go:4071 github.com/gravitational/teleport/api/client.(*Client).GetCertAuthorities                                                                                                                     │
│ operator     /go/src/github.com/gravitational/teleport/lib/auth/clt.go:171 github.com/gravitational/teleport/lib/auth.(*Client).GetCertAuthorities                                                                                                                             │
│ operator     /go/src/github.com/gravitational/teleport/lib/tbot/impersonated_identity.go:621 github.com/gravitational/teleport/lib/tbot.(*outputRenewalCache).getCertAuthorities                                                                                               │
│ operator     /go/src/github.com/gravitational/teleport/lib/tbot/impersonated_identity.go:640 github.com/gravitational/teleport/lib/tbot.(*outputRenewalCache).GetCertAuthorities                                                                                               │
│ operator     /go/src/github.com/gravitational/teleport/lib/tbot/config/template_tls_cas.go:94 github.com/gravitational/teleport/lib/tbot/config.(*templateTLSCAs).render                                                                                                       │
│ operator     /go/src/github.com/gravitational/teleport/lib/tbot/config/output_identity.go:77 github.com/gravitational/teleport/lib/tbot/config.(*IdentityOutput).Render                                                                                                        │
│ operator     /go/src/github.com/gravitational/teleport/lib/tbot/impersonated_identity.go:586 github.com/gravitational/teleport/lib/tbot.(*Bot).renewOutputs                                                                                                                    │
│ operator     /go/src/github.com/gravitational/teleport/lib/tbot/impersonated_identity.go:70 github.com/gravitational/teleport/lib/tbot.(*Bot).renewOutputsLoop                                                                                                                 │
│ operator     /go/src/github.com/gravitational/teleport/lib/tbot/tbot.go:153 github.com/gravitational/teleport/lib/tbot.(*Bot).Run.func4                                                                                                                                        │
│ operator     /go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75 golang.org/x/sync/errgroup.(*Group).Go.func1                                                                                                                                                         │
│ operator     /opt/go/src/runtime/asm_amd64.s:1650 runtime.goexit                                                                                                                                                                                                               │
│ operator User Message: rendering output: identity (memory)                                                                                                                                                                                                                     │
│ operator     rendering template tls_cas                                                                                                                                                                                                                                        │
│ operator         "db_client" authority type is not supported] tbot/impersonated_identity.go:79                                                                                                                                                                         │
│ operator 2024-06-10T16:50:22Z INFO             Renewing outputs. Attempt 5 of 5. tbot/impersonated_identity.go:65                                                                                                                                                              │
│ operator 2024-06-10T16:50:22Z INFO             Generating output. output:identity (memory) tbot/impersonated_identity.go:548                                                                                                                                                   │
│ operator 2024-06-10T16:50:23Z WARN             Failed to render output identity (memory) error:[                                                                                                                                                                               │
│ operator ERROR REPORT:                                                                                                                                                                                                                                                         │
│ operator Original Error: *trace.BadParameterError "db_client" authority type is not supported                                                                                                                                                                          │
│ operator Stack Trace:                                                                                                                                                                                                                                                          │
│ operator     /go/src/github.com/gravitational/teleport/api/client/client.go:4071 github.com/gravitational/teleport/api/client.(*Client).GetCertAuthorities                                                                                                                     │
│ operator     /go/src/github.com/gravitational/teleport/lib/auth/clt.go:171 github.com/gravitational/teleport/lib/auth.(*Client).GetCertAuthorities                                                                                                                             │
│ operator     /go/src/github.com/gravitational/teleport/lib/tbot/impersonated_identity.go:621 github.com/gravitational/teleport/lib/tbot.(*outputRenewalCache).getCertAuthorities                                                                                               │
│ operator     /go/src/github.com/gravitational/teleport/lib/tbot/impersonated_identity.go:640 github.com/gravitational/teleport/lib/tbot.(*outputRenewalCache).GetCertAuthorities                                                                                               │
│ operator     /go/src/github.com/gravitational/teleport/lib/tbot/config/template_tls_cas.go:94 github.com/gravitational/teleport/lib/tbot/config.(*templateTLSCAs).render                                                                                                       │
│ operator     /go/src/github.com/gravitational/teleport/lib/tbot/config/output_identity.go:77 github.com/gravitational/teleport/lib/tbot/config.(*IdentityOutput).Render                                                                                                        │
│ operator     /go/src/github.com/gravitational/teleport/lib/tbot/impersonated_identity.go:586 github.com/gravitational/teleport/lib/tbot.(*Bot).renewOutputs                                                                                                                    │
│ operator     /go/src/github.com/gravitational/teleport/lib/tbot/impersonated_identity.go:70 github.com/gravitational/teleport/lib/tbot.(*Bot).renewOutputsLoop                                                                                                                 │
│ operator     /go/src/github.com/gravitational/teleport/lib/tbot/tbot.go:153 github.com/gravitational/teleport/lib/tbot.(*Bot).Run.func4                                                                                                                                        │
│ operator     /go/pkg/mod/golang.org/x/[email protected]/errgroup/errgroup.go:75 golang.org/x/sync/errgroup.(*Group).Go.func1                                                                                                                                                         │
│ operator     /opt/go/src/runtime/asm_amd64.s:1650 runtime.goexit                                                                                                                                                                                                               │
│ operator User Message: rendering template tls_cas                                                                                                                                                                                                                              │
│ operator     "db_client" authority type is not supported] tbot/impersonated_identity.go:587                                                                                                                                                                            │
│ operator 2024-06-10T16:50:23Z WARN             5 retry attempts exhausted renewing outputs. Waiting for next normal renewal cycle in 30m0s. tbot/impersonated_identity.go:93

2/ coming from the deployment itself:

teleport 2024-06-10T16:54:01Z WARN [KUBERNETE] Re-init the cache on error error:[                                                                                                                                                                                                  │
│ teleport ERROR REPORT:                                                                                                                                                                                                                                                             │
│ teleport Original Error: *trace.ConnectionProblemError timeout waiting for watcher init                                                                                                                                                                                            │
│ teleport Stack Trace:                                                                                                                                                                                                                                                              │
│ teleport     github.com/gravitational/teleport/lib/cache/cache.go:1112 github.com/gravitational/teleport/lib/cache.(*Cache).fetchAndWatch                                                                                                                                          │
│ teleport     github.com/gravitational/teleport/lib/cache/cache.go:979 github.com/gravitational/teleport/lib/cache.(*Cache).update                                                                                                                                                  │
│ teleport     runtime/asm_amd64.s:1650 runtime.goexit                                                                                                                                                                                                                               │
│ teleport User Message: timeout waiting for watcher init] cache/cache.go:985                                                                                                                                                                                                        │
│ teleport 2024-06-10T16:54:22Z INFO [AUDIT]     user.login addr.remote:10.179.153.87:13362 cluster_name:teleport.domain.de code:T1000I ei:0 event:user.login method:local mfa_device_name:yubi mfa_device_type:WebAuthn mfa_device_uuid:267a8d12-f77c-4df1-8cc4- │
│ 155752a6fe23 success:true time:2024-06-10T16:54:22.066Z uid:7e904988-20c9-41bd-ac17-33c29544dcd1 user:[email protected] user_agent:Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36 events/emitter. │
│ go:271                                                                                                                                                                                                                                                                             │
│ teleport 2024-06-10T16:54:23Z INFO [CA]        Generating TLS certificate 1.3.9999.1.15=#13046e6f6e65,1.3.9999.2.6=#0c012a,1.3.9999.2.5=#0c012a,1.3.9999.1.9=#130d31302e3137392e3135332e3837,1.3.9999.1.7=#132574656c65706f72742e696e6672617374727563747572652e6f6e6573746f7070726 │
│ f2e6465,1.3.9999.1.3=#132574656c65706f72742e696e6672617374727563747572652e6f6e6573746f7070726f2e6465,1.3.9999.1.2=#130d73797374656d3a6d6173746572,[email protected],O=access+O=db-admin+O=editor+O=kube-admin-access+O=terraform-impersonator,POSTALCODE={\"db_names\ │
│ ":[\"*\"]\,\"db_users\":[\"*\"]},STREET=teleport.domain.de,L=-teleport-nologin-641db899-2593-423e-a938-a19435785a21+L=-teleport-internal-join dns_names:[] key_usage:5 not_after:2024-06-11 02:54:23.093909301 +0000 UTC tlsca/ca.go:1114                       │
│ teleport 2024-06-10T16:54:23Z INFO [AUDIT]     cert.create cert_type:user cluster_name:teleport.domain.de code:TC000I ei:0 event:cert.create client_ip:10.179.153.87 database_names:[*] database_users:[*] expires:2024-06-11T02:54:23.093909301Z kubernetes_cl │
│ uster:teleport.infrastructure.onestoppro.de kubernetes_groups:[system:master] logins:[-teleport-nologin-641db899-2593-423e-a938-a19435785a21 -teleport-internal-join] prev_identity_expires:0001-01-01T00:00:00Z roles:[access db-admin editor kube-admin-access terraform-imperso │
│ nator] route_to_cluster:teleport.domain.de teleport_cluster:teleport.domain.de db_names:[*] db_users:[*] user:[email protected] time:2024-06-10T16:54:23.108Z uid:3525084e-663e-453d-a86a-e142510d98a1 events/emitter.go:271       │
│ teleport 2024-06-10T16:54:23Z WARN [AUTH:1:CA] Re-init the cache on error error:[                                                                                                                                                                                                  │
│ teleport ERROR REPORT:                                                                                                                                                                                                                                                             │
│ teleport Original Error: *trace.BadParameterError "db_client" authority type is not supported                                                                                                                                                                              │
│ teleport Stack Trace:                                                                                                                                                                                                                                                              │
│ teleport     github.com/gravitational/teleport/[email protected]/types/trust.go:87 github.com/gravitational/teleport/api/types.CertAuthType.Check                                                                                                                                         │
│ teleport     github.com/gravitational/teleport/[email protected]/types/trust.go:102 github.com/gravitational/teleport/api/types.(*CertAuthID).Check                                                                                                                                       │
│ teleport     github.com/gravitational/teleport/[email protected]/types/authority.go:342 github.com/gravitational/teleport/api/types.(*CertAuthorityV2).CheckAndSetDefaults                                                                                                                │
│ teleport     github.com/gravitational/teleport/lib/services/authority.go:64 github.com/gravitational/teleport/lib/services.ValidateCertAuthority                                                                                                                                   │
│ teleport     github.com/gravitational/teleport/lib/services/authority.go:470 github.com/gravitational/teleport/lib/services.UnmarshalCertAuthority                                                                                                                                 │
│ teleport     github.com/gravitational/teleport/lib/services/local/trust.go:278 github.com/gravitational/teleport/lib/services/local.(*CA).GetCertAuthorities                                                                                                                       │
│ teleport     github.com/gravitational/teleport/lib/cache/collections.go:979 github.com/gravitational/teleport/lib/cache.certAuthorityExecutor.getAll                                                                                                                               │
│ teleport     github.com/gravitational/teleport/lib/cache/collections.go:97 github.com/gravitational/teleport/lib/cache.(*genericCollection[...]).fetch                                                                                                                             │
│ teleport     github.com/gravitational/teleport/lib/cache/cache.go:1456 github.com/gravitational/teleport/lib/cache.(*Cache).fetch.func2                                                                                                                                            │
│ teleport     golang.org/x/[email protected]/errgroup/errgroup.go:75 golang.org/x/sync/errgroup.(*Group).Go.func1                                                                                                                                                                         │
│ teleport     runtime/asm_amd64.s:1650 runtime.goexit                                                                                                                                                                                                                               │
│ teleport User Message: failed to fetch resource: "cert_authority"                                                                                                                                                                                                          │
│ teleport     "db_client" authority type is not supported] cache/cache.go:985                                                                                                                                                                                               │
│ teleport 2024-06-10T16:54:55Z WARN [AUTH]      "\nERROR REPORT:\nOriginal Error: *awserr.baseError RequestCanceled: request context canceled\ncaused by: context canceled\nStack Trace:\n\tgithub.com/gravitational/teleport/lib/backend/dynamo/dynamodbbk.go:927 github.com/gravi │
│ tational/teleport/lib/backend/dynamo.(*Backend).getKey\n\tgithub.com/gravitational/teleport/lib/backend/dynamo/dynamodbbk.go:487 github.com/gravitational/teleport/lib/backend/dynamo.(*Backend).Get\n\tgithub.com/gravitational/teleport/lib/backend/sanitize.go:97 github.com/gr │
│ avitational/teleport/lib/backend.(*Sanitizer).Get\n\tgithub.com/gravitational/teleport/lib/backend/report.go:219 github.com/gravitational/teleport/lib/backend.(*Reporter).Get\n\tgithub.com/gravitational/teleport/lib/services/local/configuration.go:319 github.com/gravitation │
│ al/teleport/lib/services/local.(*ClusterConfigurationService).GetSessionRecordingConfig\n\tgithub.com/gravitational/teleport/lib/cache/cache.go:2245 github.com/gravitational/teleport/lib/cache.(*Cache).GetSessionRecordingConfig\n\tgithub.com/gravitational/teleport/lib/authz │
│ /permissions.go:484 github.com/gravitational/teleport/lib/authz.(*authorizer).authorizeBuiltinRole\n\tgithub.com/gravitational/teleport/lib/authz/permissions.go:325 github.com/gravitational/teleport/lib/authz.(*authorizer).fromUser\n\tgithub.com/gravitational/teleport/lib/a │
│ uthz/permissions.go:265 github.com/gravitational/teleport/lib/authz.(*authorizer).Authorize\n\tgithub.com/gravitational/teleport/lib/auth/apiserver.go:186 github.com/gravitational/teleport/lib/auth.NewAPIServer.(*APIServer).WithAuth.func22\n\tgithub.com/gravitational/telepo │
│ rt/lib/httplib/httplib.go:117 github.com/gravitational/teleport/lib/auth.NewAPIServer.(*APIServer).WithAuth.MakeHandler.MakeHandlerWithErrorWriter.func72\n\tgithub.com/julienschmidt/[email protected]/router.go:399 github.com/julienschmidt/httprouter.(*Router).ServeHTTP\n\tg │
│ ithub.com/gravitational/teleport/lib/httplib/httplib.go:248 github.com/gravitational/teleport/lib/auth.NewAPIServer.RewritePaths.func50\n\tnet/http/server.go:2136 net/http.HandlerFunc.ServeHTTP\n\tgithub.com/gravitational/teleport/lib/auth/middleware.go:691 github.com/gravi │
│ tational/teleport/lib/auth.(*Middleware).ServeHTTP\n\tgithub.com/gravitational/[email protected]/ratelimit/tokenlimiter.go:118 github.com/gravitational/oxy/ratelimit.(*TokenLimiter).ServeHTTP\n\tgithub.com/gravitational/[email protected] │
│ 44680/connlimit/connlimit.go:75 github.com/gravitational/oxy/connlimit.(*ConnLimiter).ServeHTTP\n\tgithub.com/gravitational/teleport/lib/httplib/httplib.go:69 github.com/gravitational/teleport/lib/auth.NewTLSServer.MakeSecurityHeaderHandler.func1\n\tnet/http/server.go:2136  │
│ net/http.HandlerFunc.ServeHTTP\n\tgithub.com/gravitational/teleport/lib/httplib/httplib.go:94 github.com/gravitational/teleport/lib/httplib.MakeTracingHandler.func1\n\tnet/http/server.go:2136 net/http.HandlerFunc.ServeHTTP\n\tgo.opentelemetry.io/contrib/instrumentation/net/ │
│ http/[email protected]/handler.go:212 go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*Handler).ServeHTTP\n\tnet/http/server.go:2938 net/http.serverHandler.ServeHTTP\n\tnet/http/server.go:2009 net/http.(*conn).serve\n\truntime/asm_amd64.s:1650 runtime.goexit\n │
│ User Message: failed to get "/cluster_configuration/session_recording" (dynamo error)\n\tRequestCanceled: request context canceled\ncaused by: context canceled" authz/permissions.go:1149                                                                                 │
│ teleport 2024-06-10T16:54:55Z WARN [AUTH]      "\nERROR REPORT:\nOriginal Error: *awserr.baseError RequestCanceled: request context canceled\ncaused by: context canceled\nStack Trace:\n\tgithub.com/gravitational/teleport/lib/backend/dynamo/dynamodbbk.go:927 github.com/gravi │
│ tational/teleport/lib/backend/dynamo.(*Backend).getKey\n\tgithub.com/gravitational/teleport/lib/backend/dynamo/dynamodbbk.go:487 github.com/gravitational/teleport/lib/backend/dynamo.(*Backend).Get\n\tgithub.com/gravitational/teleport/lib/backend/sanitize.go:97 github.com/gr │
│ avitational/teleport/lib/backend.(*Sanitizer).Get\n\tgithub.com/gravitational/teleport/lib/backend/report.go:219 github.com/gravitational/teleport/lib/backend.(*Reporter).Get\n\tgithub.com/gravitational/teleport/lib/services/local/configuration.go:319 github.com/gravitation │
│ al/teleport/lib/services/local.(*ClusterConfigurationService).GetSessionRecordingConfig\n\tgithub.com/gravitational/teleport/lib/cache/cache.go:2245 github.com/gravitational/teleport/lib/cache.(*Cache).GetSessionRecordingConfig\n\tgithub.com/gravitational/teleport/lib/authz │
│ /permissions.go:484 github.com/gravitational/teleport/lib/authz.(*authorizer).authorizeBuiltinRole\n\tgithub.com/gravitational/teleport/lib/authz/permissions.go:325 github.com/gravitational/teleport/lib/authz.(*authorizer).fromUser\n\tgithub.com/gravitational/teleport/lib/a │
│ uthz/permissions.go:265 github.com/gravitational/teleport/lib/authz.(*authorizer).Authorize\n\tgithub.com/gravitational/teleport/lib/auth/apiserver.go:186 github.com/gravitational/teleport/lib/auth.NewAPIServer.(*APIServer).WithAuth.func32\n\tgithub.com/gravitational/telepo │
│ rt/lib/httplib/httplib.go:117 github.com/gravitational/teleport/lib/auth.NewAPIServer.(*APIServer).WithAuth.MakeHandler.MakeHandlerWithErrorWriter.func82\n\tgithub.com/julienschmidt/[email protected]/router.go:399 github.com/julienschmidt/httprouter.(*Router).ServeHTTP\n\tg │
│ ithub.com/gravitational/teleport/lib/httplib/httplib.go:248 github.com/gravitational/teleport/lib/auth.NewAPIServer.RewritePaths.func50\n\tnet/http/server.go:2136 net/http.HandlerFunc.ServeHTTP\n\tgithub.com/gravitational/teleport/lib/auth/middleware.go:691 github.com/gravi │
│ tational/teleport/lib/auth.(*Middleware).ServeHTTP\n\tgithub.com/gravitational/[email protected]/ratelimit/tokenlimiter.go:118 github.com/gravitational/oxy/ratelimit.(*TokenLimiter).ServeHTTP\n\tgithub.com/gravitational/[email protected] │
│ 44680/connlimit/connlimit.go:75 github.com/gravitational/oxy/connlimit.(*ConnLimiter).ServeHTTP\n\tgithub.com/gravitational/teleport/lib/httplib/httplib.go:69 github.com/gravitational/teleport/lib/auth.NewTLSServer.MakeSecurityHeaderHandler.func1\n\tnet/http/server.go:2136  │
│ net/http.HandlerFunc.ServeHTTP\n\tgithub.com/gravitational/teleport/lib/httplib/httplib.go:94 github.com/gravitational/teleport/lib/httplib.MakeTracingHandler.func1\n\tnet/http/server.go:2136 net/http.HandlerFunc.ServeHTTP\n\tgo.opentelemetry.io/contrib/instrumentation/net/ │
│ http/[email protected]/handler.go:212 go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp.(*Handler).ServeHTTP\n\tnet/http/server.go:2938 net/http.serverHandler.ServeHTTP\n\tnet/http/server.go:2009 net/http.(*conn).serve\n\truntime/asm_amd64.s:1650 runtime.goexit\n │
│ User Message: failed to get "/cluster_configuration/session_recording" (dynamo error)\n\tRequestCanceled: request context canceled\ncaused by: context canceled" authz/permissions.go:1149

Guys, do you have any idea what could be wrong and how to solve it?

Any input will be highly appreciate.

[pixelfields]

Seems to be related to #39809 , but we got this error during the upgrade, not a downgrade.

[webvictim]

This particular issue is caused by a bug in 14.0.3 IIRC (which was fixed shortly afterwards)

When doing major version upgrades, you can (and should) always upgrade to the latest patch version. For a v13 -> v14 upgrade, this means you can go straight from 13.4.26 to 14.3.20.

[pixelfields]

This particular issue is caused by a bug in 14.0.3 IIRC (which was fixed shortly afterwards)

When doing major version upgrades, you can (and should) always upgrade to the latest patch version. For a v13 -> v14 upgrade, this means you can go straight from 13.4.26 to 14.3.20.

Thank you for a swift reply, it helped a lot. Now it went through successfully.