Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Epic: Scaling latency metrics #594

Open
2 of 5 tasks
sharnoff opened this issue Oct 30, 2023 · 2 comments · May be fixed by #983
Open
2 of 5 tasks

Epic: Scaling latency metrics #594

sharnoff opened this issue Oct 30, 2023 · 2 comments · May be fixed by #983
Assignees
@Omrigan
Labels
c/autoscaling/autoscaler-agent Component: autoscaling: autoscaler-agent t/Epic Issue type: Epic

Comments

@sharnoff
Copy link
Member

sharnoff commented Oct 30, 2023 •
edited by Omrigan
Loading

Motivation

Two reasons:

  1. We're currently flying blind w.r.t. how long scaling takes
  2. Scaling latency should be part of any autoscaling SLOs

DoD

We should have histogram metrics recording:

  • end-to-end latency of scaling (down and up; cpu and memory)
  • latency of all the components:
    • requests to scheduler plugin (including retries)
    • requests to vm-monitor (including retries)
    • delay between initial NeonVM patch request and when status was updated

Implementation ideas

AFAICT the basic idea is that we store some extra info in agent/core.State and add some extra callbacks in agent/core.Config to increment some metrics when we determine that various parts of scaling (and the entire thing) have occurred.

More design work is required, because the edge cases are quite subtle.

Tasks

Blockers
Beta Give feedback
  1. Bug: VM memory scale-up leaves "Scaling" phase before updating status resources #453
    c/autoscaling/neonvm t/bug
    shayanh
  2. Bug: Fractional CPU scale-down temporarily exceeds spec maximum #462
    c/autoscaling/neonvm t/bug
    shayanh
  3. Bug: pkg/agent/state should use VM status, not spec to represent "current" resources #592
    1 of 2
    c/autoscaling/autoscaler-agent t/bug

Implementation
Beta Give feedback

Other related tasks, Epics, and links
@sharnoff sharnoff added t/Epic Issue type: Epic c/autoscaling/autoscaler-agent Component: autoscaling: autoscaler-agent labels Oct 30, 2023
@sharnoff
Copy link
Member Author

sharnoff commented Nov 7, 2023

Latency metrics measured by pkg/agent/core.State would have allowed us to notice the effects of #614 (i.e. core.State believed there was a super long-running request).

@sharnoff sharnoff mentioned this issue Dec 15, 2023
Open
This was referenced Jan 24, 2024
Closed
Closed
@Omrigan Omrigan self-assigned this Jun 4, 2024
@sharnoff
Copy link
Member Author

Status: waiting on @sharnoff and @stradig to review the internal RFC.

@Omrigan Omrigan linked a pull request Jun 20, 2024 that will close this issue
Draft
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@Omrigan Omrigan

Labels
c/autoscaling/autoscaler-agent Component: autoscaling: autoscaler-agent t/Epic Issue type: Epic
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Implement scaling latency metrics through logical clock neondatabase/autoscaling
2 participants
@Omrigan @sharnoff