Add proposal for circuit breaker #397

Okabe-Rintarou-0 · 2024-05-29T13:19:13Z

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes:
Fixes #

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

Signed-off-by: [email protected] <[email protected]>

kmesh-bot · 2024-05-29T13:19:19Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign hzxuzhonghu for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

nlgwcy · 2024-05-30T01:05:45Z

docs/proposal/pics/circuit_breaker_config.png

Good job~ Pls upload an editable svg diagram for subsequent update.
could you share the proposal in the community meeting this week?

online web:
https://app.diagrams.net/

I already exported images using drawio. The .png images can be directly opened in drawio(It contains extra info about drawio).

nlgwcy · 2024-05-30T01:31:41Z

docs/proposal/circuit_breaker.md

+
+In the closed state, all requests pass through normally and are sent to the target service, even if occasional failures occur, without triggering the circuit breaker. However, when the number of failed requests exceeds a preset threshold, the circuit breaker quickly switches to the open state. In this state, all requests immediately fail, avoiding sending requests to a service known to be failing, thus reducing the system's burden.
+
+In the open state, the circuit breaker starts a timeout timer. After the timeout period, the circuit breaker transitions to the half-open state. In the half-open state, the system allows a small number of requests to go through to test whether the target service has recovered. If these requests succeed, the circuit breaker closes again, allowing the service to handle more requests. If the requests fail, the circuit breaker reopens, maintaining protection for the system against the potentially faulty service.


It would be better to add some implementation details, such as:

How is the timer logic implemented

Specifies the BPF hook point that triggers circuit breaker

Sure, it's a rough outline of the current proposal. I need more follow-up discussions and code review to obtain a more detailed design. I will keep working on it.

timer logic might be implemented by ebpf timer: https://ebpf-docs.dylanreimerink.nl/linux/concepts/timers/

here's an example:

// Initialize the timer. // First 4 bits of 'flags' specify clockid. // Only CLOCK_MONOTONIC, CLOCK_REALTIME, CLOCK_BOOTTIME are allowed. long bpf_timer_init(struct bpf_timer *timer, struct bpf_map *map, int flags); // Configure the timer to call 'callback_fn' static function. long bpf_timer_set_callback(struct bpf_timer *timer, void *callback_fn); // Arm the timer to expire 'nsec' nanoseconds from the current time. long bpf_timer_start(struct bpf_timer *timer, u64 nsec, u64 flags); // Cancel the timer and wait for callback_fn to finish if it was running. long bpf_timer_cancel(struct bpf_timer *timer); Here is how BPF program might look like: struct map_elem { int counter; struct bpf_timer timer; }; struct { __uint(type, BPF_MAP_TYPE_HASH); __uint(max_entries, 1000); __type(key, int); __type(value, struct map_elem); } hmap SEC(".maps"); static int timer_cb(void *map, int *key, struct map_elem *val); /* val points to particular map element that contains bpf_timer. */ SEC("fentry/bpf_fentry_test1") int BPF_PROG(test1, int a) { struct map_elem *val; int key = 0; val = bpf_map_lookup_elem(&hmap, &key); if (val) { bpf_timer_init(&val->timer, &hmap, CLOCK_REALTIME); bpf_timer_set_callback(&val->timer, timer_cb); bpf_timer_start(&val->timer, 1000 /* call timer_cb2 in 1 usec */, 0); } }

Great~ As described, bpf_timer does not have ctx context information, so the definition of the input parameter map of timer_callback is very important.
The key of map needs to be able to identify different circuit breakers, and the context information of the circuit breaker event needs to be recorded in the vaule of map.

LiZhenCheng9527 · 2024-05-30T01:37:35Z

docs/proposal/circuit_breaker.md

+
+#### Implement the state machine of a circuit breaker
+
+![](./pics/circuit_breaker_state_machine.png)


Centre the image will better

Ok, thanks.

LiZhenCheng9527 · 2024-05-30T01:40:31Z

docs/proposal/circuit_breaker.md

+
+We can implement a circuit breaker mechanism to ensure system stability and reliability while minimizing access to faulty services. First, we need to implement a circuit breaker with three states: closed, open, and half-open.
+
+In the closed state, all requests pass through normally and are sent to the target service, even if occasional failures occur, without triggering the circuit breaker. However, when the number of failed requests exceeds a preset threshold, the circuit breaker quickly switches to the open state. In this state, all requests immediately fail, avoiding sending requests to a service known to be failing, thus reducing the system's burden.


Suggested change

In the closed state, all requests pass through normally and are sent to the target service, even if occasional failures occur, without triggering the circuit breaker. However, when the number of failed requests exceeds a preset threshold, the circuit breaker quickly switches to the open state. In this state, all requests immediately fail, avoiding sending requests to a service known to be failing, thus reducing the system's burden.

In the closed state, all requests pass through normally and are sent to the target service, even if occasional failures occur, without triggering the circuit breaker. However, when the number of failed requests exceeds a preset threshold, the circuit breaker quickly switches to the open state. In this state, all requests fail immediately, reducing the burden on the system by avoiding sending requests to service that is known to fail.

Suggested change

In the closed state, all requests pass through normally and are sent to the target service, even if occasional failures occur, without triggering the circuit breaker. However, when the number of failed requests exceeds a preset threshold, the circuit breaker quickly switches to the open state. In this state, all requests immediately fail, avoiding sending requests to a service known to be failing, thus reducing the system's burden.

In the closed state, all requests pass through normally and are sent to the target service, even if occasional failures occur, without triggering the circuit breaker. However, when the number of failed requests exceeds a preset threshold, the circuit breaker quickly switches to the open state. In this state, all requests immediately fail, avoiding sending requests to a service known to be failing, thus reducing the system's burden.

Ok, thanks.

LiZhenCheng9527 · 2024-05-30T01:52:03Z

docs/proposal/circuit_breaker.md

+
+ Once an anomaly is detected, Outlier Detection temporarily removes the instance from the load balancing pool, effectively "ejecting" the instance to prevent it from receiving new requests. After a certain period, the system will reassess the health status of the instance and, if it has returned to normal, will reintegrate it into the load balancing pool.
+
+We can monitor HTTP return statuses in eBPF to determine if a service is experiencing 5xx errors. When the number of such errors reaches a certain threshold, we need to exclude the corresponding endpoints from the load balancing selection.


Can we get HTTP status codes in eBPF?
Now in Kmesh, we have L7 governance via waypoint.
Maybe we should discuss this at the regular community meeting

I have just started learning about eBPF. I did some researches last night and found this potential solution: https://github.com/weaveworks-plugins/scope-http-statistics/blob/master/ebpf-http-statistics.c

Signed-off-by: [email protected] <[email protected]>

codecov-commenter · 2024-05-30T05:10:44Z

Codecov Report

All modified and coverable lines are covered by tests ✅

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Flag	Coverage Δ
unittests	`31.10% <ø> (?)`

Flags with carried forward coverage won't be shown. Click here to find out more.

hzxuzhonghu · 2024-05-30T07:01:41Z

docs/proposal/circuit_breaker.md

+
+### Summary
+
+Roadmap:


This is more about target not roadmap

Ok，i got it.

Signed-off-by: [email protected] <[email protected]>

hzxuzhonghu · 2024-06-03T08:03:29Z

docs/proposal/circuit_breaker.md

+ High error rate in the service
+ High latency in the service
+ Exhaustion of service resources
+ Service unavailability


some other metrics like:

max requests

max connections

hzxuzhonghu · 2024-06-03T08:10:11Z

docs/proposal/circuit_breaker.md

+struct {
+    __uint(type, BPF_MAP_TYPE_HASH);
+    __uint(max_entries, 1000);
+    __type(key, int);


what is the key

a circuit is bound to a cluster, maybe we can use the cluster's identifier?

hzxuzhonghu · 2024-06-03T08:11:01Z

docs/proposal/circuit_breaker.md

+
+To implement the aforementioned functionality, we can maintain several counters in eBPF to calculate the number of requests reaching each cluster (including the number of successful requests, timed-out requests, failed requests, etc.). Once certain conditions are met, we can trigger the circuit breaker mechanism to reject traffic destined for a particular cluster.
+
+We can use [timers](https://ebpf-docs.dylanreimerink.nl/linux/concepts/timers/) in eBPF to implement the open state of the circuit breaker. When the circuit breaker breaker is open, it will start a timer by the following code:


From which kernel version does timer begin to be supported

since v5.15 torvalds/linux@b00628b

Why not implement timer in userspace of Go program.

Why not implement timer in userspace of Go program.

that's a good problem. Shall we implement circuit breaker in user space or kernel space.

hzxuzhonghu · 2024-06-04T02:49:15Z

I didnot see the response recording, how could we achieve that?

Okabe-Rintarou-0 · 2024-06-04T06:16:29Z

I didnot see the response recording, how could we achieve that?

can we use this https://github.com/weaveworks-plugins/scope-http-statistics/blob/master/ebpf-http-statistics.c?

I'm busy for taking examines these two weeks, so my progress may be slow recently.

Okabe-Rintarou-0 · 2024-06-04T08:01:31Z

Can we maintain a statistic structure map in ebpf (so that can be accessed in user space), recording http statistic information like: number of connections, number of requests. We hook some functions in ebpf to collect this info, and we can do circuit breaker logic using golang in user space.

hzxuzhonghu · 2024-06-04T08:27:53Z

Agree, there need response handling

Signed-off-by: [email protected] <[email protected]>

Okabe-Rintarou-0 · 2024-06-04T12:03:18Z

I tried https://eunomia.dev/zh/tutorials/23-http/, works.

127.0.0.1:59434(src) -> 127.0.0.1:8000(dst)
payload: GET / HTTP/1.1
Host: 0.0.0.0:8000
User-Agent: curl/7.81.0
Acc
127.0.0.1:8000(src) -> 127.0.0.1:59434(dst)
payload: HTTP/1.0 200 OK
Server: SimpleHTTP/0.6 Python/3.10.12
Date: Tu

add proposal for circuit breaker

ccd4331

Signed-off-by: [email protected] <[email protected]>

kmesh-bot requested review from nlgwcy and supercharge-xsy May 29, 2024 13:19

kmesh-bot added the size/M label May 29, 2024

nlgwcy reviewed May 30, 2024

View reviewed changes

LiZhenCheng9527 reviewed May 30, 2024

View reviewed changes

adjust images

f9e865d

Signed-off-by: [email protected] <[email protected]>

kmesh-bot added size/L and removed size/M labels May 30, 2024

hzxuzhonghu reviewed May 30, 2024

View reviewed changes

optimize proposal

675d454

Signed-off-by: [email protected] <[email protected]>

hzxuzhonghu reviewed Jun 3, 2024

View reviewed changes

optimize proposal

cf4590c

Signed-off-by: [email protected] <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add proposal for circuit breaker #397

Add proposal for circuit breaker #397

Okabe-Rintarou-0 commented May 29, 2024

kmesh-bot commented May 29, 2024

nlgwcy May 30, 2024 •

edited

Loading

Okabe-Rintarou-0 May 30, 2024

nlgwcy May 30, 2024 •

edited

Loading

Okabe-Rintarou-0 May 30, 2024

Okabe-Rintarou-0 May 31, 2024

nlgwcy May 31, 2024

LiZhenCheng9527 May 30, 2024

Okabe-Rintarou-0 May 30, 2024

LiZhenCheng9527 May 30, 2024

Okabe-Rintarou-0 May 30, 2024

LiZhenCheng9527 May 30, 2024

Okabe-Rintarou-0 May 30, 2024

codecov-commenter commented May 30, 2024 •

edited

Loading

hzxuzhonghu May 30, 2024

Okabe-Rintarou-0 May 30, 2024

hzxuzhonghu Jun 3, 2024

hzxuzhonghu Jun 3, 2024

Okabe-Rintarou-0 Jun 3, 2024

hzxuzhonghu Jun 3, 2024

Okabe-Rintarou-0 Jun 3, 2024

Okabe-Rintarou-0 Jun 3, 2024

bfforever Jun 3, 2024

Okabe-Rintarou-0 Jun 3, 2024

hzxuzhonghu commented Jun 4, 2024

Okabe-Rintarou-0 commented Jun 4, 2024

Okabe-Rintarou-0 commented Jun 4, 2024

hzxuzhonghu commented Jun 4, 2024

Okabe-Rintarou-0 commented Jun 4, 2024


		In the closed state, all requests pass through normally and are sent to the target service, even if occasional failures occur, without triggering the circuit breaker. However, when the number of failed requests exceeds a preset threshold, the circuit breaker quickly switches to the open state. In this state, all requests immediately fail, avoiding sending requests to a service known to be failing, thus reducing the system's burden.

		In the open state, the circuit breaker starts a timeout timer. After the timeout period, the circuit breaker transitions to the half-open state. In the half-open state, the system allows a small number of requests to go through to test whether the target service has recovered. If these requests succeed, the circuit breaker closes again, allowing the service to handle more requests. If the requests fail, the circuit breaker reopens, maintaining protection for the system against the potentially faulty service.


		#### Implement the state machine of a circuit breaker

		![](./pics/circuit_breaker_state_machine.png)


		We can implement a circuit breaker mechanism to ensure system stability and reliability while minimizing access to faulty services. First, we need to implement a circuit breaker with three states: closed, open, and half-open.

		In the closed state, all requests pass through normally and are sent to the target service, even if occasional failures occur, without triggering the circuit breaker. However, when the number of failed requests exceeds a preset threshold, the circuit breaker quickly switches to the open state. In this state, all requests immediately fail, avoiding sending requests to a service known to be failing, thus reducing the system's burden.


		+ Once an anomaly is detected, Outlier Detection temporarily removes the instance from the load balancing pool, effectively "ejecting" the instance to prevent it from receiving new requests. After a certain period, the system will reassess the health status of the instance and, if it has returned to normal, will reintegrate it into the load balancing pool.

		We can monitor HTTP return statuses in eBPF to determine if a service is experiencing 5xx errors. When the number of such errors reaches a certain threshold, we need to exclude the corresponding endpoints from the load balancing selection.


		To implement the aforementioned functionality, we can maintain several counters in eBPF to calculate the number of requests reaching each cluster (including the number of successful requests, timed-out requests, failed requests, etc.). Once certain conditions are met, we can trigger the circuit breaker mechanism to reject traffic destined for a particular cluster.

		We can use [timers](https://ebpf-docs.dylanreimerink.nl/linux/concepts/timers/) in eBPF to implement the open state of the circuit breaker. When the circuit breaker breaker is open, it will start a timer by the following code:


		### Summary

		Roadmap:

Add proposal for circuit breaker #397

Are you sure you want to change the base?

Add proposal for circuit breaker #397

Conversation

Okabe-Rintarou-0 commented May 29, 2024

kmesh-bot commented May 29, 2024

nlgwcy May 30, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nlgwcy May 30, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov-commenter commented May 30, 2024 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hzxuzhonghu commented Jun 4, 2024

Okabe-Rintarou-0 commented Jun 4, 2024

Okabe-Rintarou-0 commented Jun 4, 2024

hzxuzhonghu commented Jun 4, 2024

Okabe-Rintarou-0 commented Jun 4, 2024

nlgwcy May 30, 2024 •

edited

Loading

nlgwcy May 30, 2024 •

edited

Loading

codecov-commenter commented May 30, 2024 •

edited

Loading