Vector high memory utilization

[Karjan1]

A note for the community

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Problem

Hi, we have one of the environments behaving strange. After each deployment on that env, memory usage of Vector spikes to dangerous values. It's a preprod environment with the same configuration as prod environments, but much lower traffic per Vector container. Some numbers:

Common for all envs:

• We're using pretty large containers with 13vCPUs and 26GB of memory
• We have autoscaling set 20% memory and 65% CPU utilization
• We have a lot of transformations, this impacts Vector startup time that is ~9minutes.
• For buffer_full on all sinks, we're dropping newest.

Prod:

• CPU utilization is cycling at around 50-90% (occasional spikes to 100%)
• Memory is usually ~3-4GB (occasionally spikes to 6GB)
• With those numbers Vector processes data reliably, without drops.

Preprod:

• Single preprod container processes around 15% of traffic going through prod one
• CPU utilization is ~20-25%. During deploys spikes to 40-50%
• Memory is usually ~1GB. During deploys spikes to 11-13GB (50-60%) (rarely reaches 13GB, it can cause container to crash)

Below is an image of memory utilization of one container after deploy

My guess about what could be possibly happening is that Vector struggles with quick scaling of a single transformation. In our case vector has a lot of transformations that are specific to a single service. And preprod takes traffic from a small set of services. When looking at utilization of transformations after deploy, we see spikes but it looks relatively healthy and nothing else is different on that env, so I'm probably wrong.

Utilization of each transformation

Happy to take any suggestions what might be the issue hee

Configuration

No response

Version

vector 0.38.0 (aarch64-unknown-linux-musl ea0ec6f 2024-05-07 14:34:39.794027186)

Debug Output

No response

Example Data

No response

Additional Context

No response

References

No response

[jszwedko]

Can you share the configuration?

[Karjan1]

Hi @jszwedko. I can't share the whole repo, but if there particular parts you're interested in, we can share those.

[jszwedko]

Memory use is very workload and configuration dependent so this is likely to be difficult to track down without the configuration and at least a description of the workload (with as many specifics as possible).

If you want, you could try to engage in some memory profiling using valgrind or a similar tool.

[Karjan1]

That's fair. It'll probably be tricky to provide description of details of the pipeline, we have 100s VRLs. But on high level, it processes all of our telemetry before delivering it to Datadog.

You might recall our pipeline. We've met on a video call last year. I work for a gaming company from U.S. We wrote the Vector metric_metadata transformer that was added to Vector as part of log_to_metric (allowin datatype conversion from log to metric when underlying data is compatible)

We'll look into memory profiling and try to get some more details. Thanks Jesse