[ISSUE-5705] Support for degraded health #5741

seonWKim · 2024-06-08T07:38:36Z

Motivation:

Add support for degraded health.

Modifications:

Add HealthStatus enum which can represent the health status of a server
Add healthStatus() method in HealthChecker for fine-grained representation of server's status
Add degradedResponse which shows that the server's status is being degraded.

I think adding new health check service which returns HealthStatus(instead of boolean) might be an alternate option. But because this PR is before any reviews, I've just implemented feature on top of HealthCheckService.

Result:

Users can now receive notification for degraded server status like below

{"healthy":true, "degraded":true}

Closes Support for degraded health #5705. (If this resolves the issue.)
Users can now identify degraded server status in addtion to healthy and unhealthy.

github-actions · 2024-06-08T07:52:52Z

🔍 Build Scan® (commit: `a22beb8`)

Job name	Status	Build Scan®
build-windows-latest-jdk-21	✅	https://ge.armeria.dev/s/52rb52ylfe6lk
build-self-hosted-unsafe-jdk-8	✅	https://ge.armeria.dev/s/zh2hyniepvkz4
build-self-hosted-unsafe-jdk-21-snapshot-blockhound	✅	https://ge.armeria.dev/s/yor4bxu6ublrw
build-self-hosted-unsafe-jdk-17-min-java-17-coverage	✅	https://ge.armeria.dev/s/bfg6lepqmetiu
build-self-hosted-unsafe-jdk-17-min-java-11	❌ (failure)	https://ge.armeria.dev/s/azsfjxc35ogkk
build-self-hosted-unsafe-jdk-17-leak	✅	https://ge.armeria.dev/s/nqsmqi3g4747g
build-self-hosted-unsafe-jdk-11	✅	https://ge.armeria.dev/s/2fgzk5ihnijp6
build-macos-12-jdk-21	✅	https://ge.armeria.dev/s/p5ay3sfmptzdm

trustin

Thanks, @seonWKim! 🙇

core/src/main/java/com/linecorp/armeria/server/healthcheck/HealthCheckService.java

core/src/main/java/com/linecorp/armeria/server/healthcheck/HealthCheckServiceBuilder.java

core/src/main/java/com/linecorp/armeria/server/healthcheck/HealthStatus.java

core/src/main/java/com/linecorp/armeria/server/healthcheck/SettableHealthStatusChecker.java

core/src/main/java/com/linecorp/armeria/server/healthcheck/HealthCheckServiceBuilder.java

core/src/main/java/com/linecorp/armeria/server/healthcheck/HealthCheckService.java

jrhee17

Looks good overall 👍 I think some client-side changes will be needed, but I prefer I do this to ensure compatibility with xDS-side requirements. Left some minor comments 🙇

core/src/main/java/com/linecorp/armeria/server/healthcheck/HealthStatus.java

jrhee17 · 2024-06-14T07:19:32Z

core/src/main/java/com/linecorp/armeria/server/healthcheck/HealthCheckService.java

        final boolean useLongPolling;
        if (longPollingTimeoutMillis > 0) {
            final String expectedState =
                    Ascii.toLowerCase(req.headers().get(HttpHeaderNames.IF_NONE_MATCH, ""));
            if ("\"healthy\"".equals(expectedState) || "w/\"healthy\"".equals(expectedState)) {
-                useLongPolling = isHealthy;
+                useLongPolling = healthStatus == HealthStatus.HEALTHY;


note: this change means if HEALTHY or UNHEALTHY isn't set, then long polling never works since armeria client only sends healthy or unhealthy depending on the status code.

armeria/core/src/main/java/com/linecorp/armeria/client/endpoint/healthcheck/HttpHealthChecker.java

Lines 105 to 106 in 579a6d0

headers = builder.add(HttpHeaderNames.IF_NONE_MATCH,

wasHealthy ? "\"healthy\"" : "\"unhealthy\"")

Oh, I see. Then, I think we can change the logic as follows:

When the expectedState is

healthy: use long polling when the status is HEALTHY or DEGRADED

unhealthy: use long polling when the status is UNHEALTHY, UNDER_MAINTENANCE, STOPPING

I understood the objective of this issue to propagate the status to clients.
If we only maintain expectedState, I think clients may not receive the status on update for long polling health check requests.

If we would like to maintain backwards compatibility, we could define a separate syntax for status matching.

e.g.

If-None-Match=[healthy|unhealthy]: propagates a signal whenever a status changes between healthy <-> unhealthy

If-None-Match=status=[HEALTHY|DEGRADED|UNHEALTHY|UNDER_MAINTENANCE|STOPPING]: propagates a signal when the status does not match the status exactly

I've added If-None-Match=status=[HEALTHY|DEGRADED|UNHEALTHY|UNDER_MAINTENANCE|STOPPING] and preserved If-None-Match=[healthy|unhealthy] in order to maintain backward compatibility.

When If-None-Match=[healthy|unhealthy] is present, return a response when the server becomes healthy(HEALTHY, DEGRADED) or unhealthy(STOPPING, UNHEALTHY, UNDER_MAINTENANCE)

When If-None-Match=status=[HEALTHY|DEGRADED|UNHEALTHY|UNDER_MAINTENANCE|STOPPING] is present, return a response when the server's HealthStatus is changed.

jrhee17 · 2024-06-14T07:22:10Z

core/src/main/java/com/linecorp/armeria/server/healthcheck/HealthCheckService.java

@@ -435,24 +435,35 @@ public HttpResponse serve(ServiceRequestContext ctx, HttpRequest req) throws Exc
            if (updateResult != null) {
                switch (updateResult) {
                    case HEALTHY:
-                        serverHealth.setHealthy(true);
+                        serverHealth.setHealthStatus(HealthStatus.HEALTHY);


Note: If the default HealthCheckUpdateHandler is used, only HEALTHY and UNHEALTHY will be returned.

I think users would find it surprising that the server is aware of options such as DEGRADED, etc.. but doesn't understand update semantics of such statuses

Oh you're right. What do you think of adding an implementation of HealthCheckUpdateHandler to handle new status? 🤔

Sounds good to me 👍

Added HealthStatusUpdateHandler 😄

jrhee17 · 2024-06-14T07:32:17Z

core/src/main/java/com/linecorp/armeria/server/healthcheck/HealthCheckServiceBuilder.java

+     *
+     * @return {@code this}
+     */
+    public HealthCheckServiceBuilder degradedResponse(AggregatedHttpResponse degradedResponse) {


note: from the perspective of health checking in envoy, just sending a x-envoy-degraded header to denote an endpoint is probably enough. Checked that this won't interfere with ongoing xDS related changes.

It seems envoy use headers to denote the specific status. What do you think would be more appropriate for this case to denote specific status? The headers or status field in the json?

I think it's fine to let users customize the response for now - later depending on requirements we may just introduce a separate EnvoyHealthCheckService.

We probably should do some research on the behavior of well known server implementations and make the default response covers them all, as a follow-up to this PR.

jrhee17 · 2024-06-14T07:33:21Z

core/src/main/java/com/linecorp/armeria/server/healthcheck/HealthCheckServiceBuilder.java

-                                                                                 "{\"healthy\":false}");
+    private AggregatedHttpResponse healthyResponse =
+            AggregatedHttpResponse.of(HttpStatus.OK, MediaType.JSON_UTF_8,
+                                      "{\"healthy\":true,\"status\":\"HEALTHY\"}");


Note: In case this point was missed, armeria client is ignoring the body at the moment and determining health status via status code.

Oh right. HttpHealthChecker only checks the status code. Should we handle the changes in this PR as well?

I'm unsure of how we want to expose health check information on the client-side at the moment. From an xDS perspective, other signals such as 1) the ratio of failed health checks 2) time since last health check 3) etc.. are exposed.

I think it's better to focus on implementing backwards compatible server-side changes in this PR, and client-side changes are dealt with separately (after a consensus is made on the API direction).

core/src/main/java/com/linecorp/armeria/server/healthcheck/HealthCheckService.java

trustin

Almost there!

core/src/main/java/com/linecorp/armeria/server/healthcheck/HealthStatus.java

seonWKim · 2024-06-22T02:21:24Z

core/src/main/java/com/linecorp/armeria/server/healthcheck/DefaultHealthCheckUpdateHandler.java

+ * Handler which updates the healthiness of the {@link Server}. Supports {@code PUT}, {@code POST} and
+ * {@code PATCH} requests and tells if the {@link Server} needs to be marked as healthy or unhealthy.
+ */
+public enum DefaultHealthCheckUpdateHandler implements HealthCheckUpdateHandler {


I changed the access modifier to public in order to let users select their update handler(DefaultHealthCheckUpdateHandlerorHealthStatusUpdateHandler`)

Question) what do you think of just combining the two HealthCheckUpdateHandler and adding similar logic as HealthCheckService?

e.g.

if jsonNode.contains("healthy") { // previous logic } else if jsonNode.contains("status") { // update to the status } else { // 400 }

Once done, please don't forget to make this class package-private again.

…bleHealthChecker`

…dates

Co-authored-by: jrhee17 <[email protected]>

…cify HealthStatus

- Add javadoc on `HealthStatus` - Checks for IF-NONE-MATCH header for specific `HealthStatus` - Add `HealthStatusUpdateHandler`

seonWKim · 2024-06-22T02:33:01Z

Removed the [ISSUE-5741] prefix in the commit messages.

jrhee17

Looks good overall 👍 Left a minor suggestion

jrhee17 · 2024-06-24T02:39:57Z

core/src/main/java/com/linecorp/armeria/server/healthcheck/DefaultHealthCheckUpdateHandler.java

+ * Handler which updates the healthiness of the {@link Server}. Supports {@code PUT}, {@code POST} and
+ * {@code PATCH} requests and tells if the {@link Server} needs to be marked as healthy or unhealthy.
+ */
+public enum DefaultHealthCheckUpdateHandler implements HealthCheckUpdateHandler {


Question) what do you think of just combining the two HealthCheckUpdateHandler and adding similar logic as HealthCheckService?

e.g.

if jsonNode.contains("healthy") { // previous logic } else if jsonNode.contains("status") { // update to the status } else { // 400 }

seonWKim · 2024-06-24T14:09:24Z

Question) what do you think of just combining the two HealthCheckUpdateHandler and adding similar logic as HealthCheckService?

Looks better. I don't think we need to split them into different classes. 👍

…dateHandler`

jrhee17

This is probably the last of my comments.
Please also address https://github.com/line/armeria/pull/5741/files#r1651918233 🙏

jrhee17 · 2024-06-26T08:53:24Z

core/src/main/java/com/linecorp/armeria/server/healthcheck/SettableHealthChecker.java

@@ -46,22 +44,45 @@ public SettableHealthChecker() {
     * changed using {@link #setHealthy(boolean)}.
     */
    public SettableHealthChecker(boolean isHealthy) {
-        this.isHealthy = new AtomicBoolean(isHealthy);
+        this.healthStatus = isHealthy ? HealthStatus.HEALTHY : HealthStatus.UNHEALTHY;


Suggested change

this.healthStatus = isHealthy ? HealthStatus.HEALTHY : HealthStatus.UNHEALTHY;

healthStatus = isHealthy ? HealthStatus.HEALTHY : HealthStatus.UNHEALTHY;

jrhee17 · 2024-06-26T08:56:11Z

core/src/main/java/com/linecorp/armeria/server/healthcheck/HealthChecker.java

+     * Returns the {@link HttpStatus} representing the health status of the {@link Server}.
+     * Override below method if you want more fine-grained health status.
+     */
+    default HealthStatus healthStatus() {


Can you modify that ScheduledHealthChecker also uses the healthStatus API?

jrhee17 · 2024-06-26T08:57:17Z

core/src/main/java/com/linecorp/armeria/server/healthcheck/HealthChecker.java

+        return new CpuHealthChecker(targetSystemCpuUsage, targetProcessCpuUsage,
+                                    degradedTargetSystemCpuUsage, degradedTargetProcessCpuLoad);
+    }
+
    /**
     * Returns {@code true} if and only if the {@link Server} is healthy.
     */
    boolean isHealthy();


Note: I imagine that some time in the future we may likely end up deprecating this API since we're only using HealthChecker#healthStatus internally in HealthCheckService.

I prefer this be handled separately though if necessary.

Meanwhile, can you go through the code in armeria and convert any usages using this API to use healthStatus instead?

seonWKim requested review from ikhoon, jrhee17, minwoox and trustin as code owners June 8, 2024 07:38

seonWKim mentioned this pull request Jun 8, 2024

Support for degraded health #5705

Open

seonWKim force-pushed the feature/ISSUE-5705-degradable-health-checker branch from f20ed57 to 665ec13 Compare June 8, 2024 08:20

trustin reviewed Jun 8, 2024

View reviewed changes

trustin added the new feature label Jun 8, 2024

seonWKim commented Jun 9, 2024

View reviewed changes

core/src/main/java/com/linecorp/armeria/server/healthcheck/HealthCheckServiceBuilder.java Outdated Show resolved Hide resolved

seonWKim commented Jun 9, 2024

View reviewed changes

core/src/main/java/com/linecorp/armeria/server/healthcheck/HealthCheckService.java Show resolved Hide resolved

seonWKim force-pushed the feature/ISSUE-5705-degradable-health-checker branch from 52fdd7b to aed0a93 Compare June 14, 2024 01:24

jrhee17 reviewed Jun 14, 2024

View reviewed changes

trustin reviewed Jun 19, 2024

View reviewed changes

core/src/main/java/com/linecorp/armeria/server/healthcheck/HealthStatus.java Outdated Show resolved Hide resolved

core/src/main/java/com/linecorp/armeria/server/healthcheck/HealthStatus.java Outdated Show resolved Hide resolved

seonWKim commented Jun 22, 2024

View reviewed changes

seonWKim and others added 15 commits June 22, 2024 11:32

Add support for degradable health status in HttpCheckService

d821502

Fix CpuHealthChecker and add test

866d72f

Remove SettableHealthStatusChecker and merge the features to `Setta…

f7db1d1

…bleHealthChecker`

Add healthStatusUpdated for more fine fine-grained health status up…

40fad94

…dates

Add UNDER_MAINTENANCE

a677109

Fix getResponse

4ac6a6f

Fix lint

eed2b0a

Fix test name

59184fc

Fix docs

8b5d8f9

Add (server) STOPPING status

7a4b880

Fix test

3af30d2

Apply suggestions from code review

4d780db

Co-authored-by: jrhee17 <[email protected]>

Apply comments

26a4979

Fix to code to consider backward compatibility and allow users to spe…

ebb4b6e

…cify HealthStatus

Apply comments

495ce58

- Add javadoc on `HealthStatus` - Checks for IF-NONE-MATCH header for specific `HealthStatus` - Add `HealthStatusUpdateHandler`

Set HealthStatusUpdateHAndler to public

487997f

seonWKim force-pushed the feature/ISSUE-5705-degradable-health-checker branch from 1548102 to 487997f Compare June 22, 2024 02:32

jrhee17 reviewed Jun 24, 2024

View reviewed changes

Merge HealthStatusUpdateHandler features into `DefaultHealthCheckUp…

a22beb8

…dateHandler`

jrhee17 added this to the 1.30.0 milestone Jun 26, 2024

jrhee17 reviewed Jun 26, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ISSUE-5705] Support for degraded health #5741

[ISSUE-5705] Support for degraded health #5741

seonWKim commented Jun 8, 2024 •

edited

Loading

github-actions bot commented Jun 8, 2024 •

edited

Loading

trustin left a comment

jrhee17 left a comment

jrhee17 Jun 14, 2024

seonWKim Jun 14, 2024 •

edited

Loading

jrhee17 Jun 17, 2024

seonWKim Jun 22, 2024 •

edited

Loading

jrhee17 Jun 14, 2024

seonWKim Jun 14, 2024

jrhee17 Jun 17, 2024

seonWKim Jun 22, 2024

jrhee17 Jun 14, 2024

seonWKim Jun 14, 2024 •

edited

Loading

jrhee17 Jun 17, 2024

trustin Jun 19, 2024

jrhee17 Jun 14, 2024

seonWKim Jun 14, 2024

jrhee17 Jun 17, 2024

trustin left a comment

seonWKim Jun 22, 2024

jrhee17 Jun 24, 2024

trustin Jun 25, 2024

seonWKim commented Jun 22, 2024

jrhee17 left a comment

jrhee17 Jun 24, 2024

seonWKim commented Jun 24, 2024

jrhee17 left a comment

jrhee17 Jun 26, 2024

jrhee17 Jun 26, 2024

jrhee17 Jun 26, 2024

jrhee17 Jun 26, 2024 •

edited

Loading

	headers = builder.add(HttpHeaderNames.IF_NONE_MATCH,
	wasHealthy ? "\"healthy\"" : "\"unhealthy\"")

	this.healthStatus = isHealthy ? HealthStatus.HEALTHY : HealthStatus.UNHEALTHY;
	healthStatus = isHealthy ? HealthStatus.HEALTHY : HealthStatus.UNHEALTHY;

[ISSUE-5705] Support for degraded health #5741

Are you sure you want to change the base?

[ISSUE-5705] Support for degraded health #5741

Conversation

seonWKim commented Jun 8, 2024 • edited Loading

github-actions bot commented Jun 8, 2024 • edited Loading

🔍 Build Scan® (commit: a22beb8)

trustin left a comment

Choose a reason for hiding this comment

jrhee17 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

seonWKim Jun 14, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

seonWKim Jun 22, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

seonWKim Jun 14, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

trustin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

seonWKim commented Jun 22, 2024

jrhee17 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

seonWKim commented Jun 24, 2024

jrhee17 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jrhee17 Jun 26, 2024 • edited Loading

Choose a reason for hiding this comment

seonWKim commented Jun 8, 2024 •

edited

Loading

github-actions bot commented Jun 8, 2024 •

edited

Loading

🔍 Build Scan® (commit: `a22beb8`)

seonWKim Jun 14, 2024 •

edited

Loading

seonWKim Jun 22, 2024 •

edited

Loading

seonWKim Jun 14, 2024 •

edited

Loading

jrhee17 Jun 26, 2024 •

edited

Loading