Improve the memory usage of our own HLL implementation

[MMeent]

We don't use sliding windows, so we can just drop the historical snapshot requirement from the implementation, thus removing the 50%+ tracking overhead.

A nice benefit is that this allows working set cardinality estimation across the full lifetime of the instance, rather than just a recent window. A demerit is that you can't take an arbitrary window, the end of any estimation window must be current or in the future.

Problem

Summary of changes

Checklist before requesting a review

• I have performed a self-review of my code.
• If it is a core feature, I have added thorough tests.
• Do we need to implement analytics? if so did you add the relevant metrics to the dashboard?
• If this PR requires public announcement, mark it with /release-notes label and add several sentences in this section.

Checklist before merging

• Do not forget to reformat commit message to not include the above checklist

[github-actions]

3222 tests run: 3079 passed, 1 failed, 142 skipped (full report)

---

Failures on Postgres 14

• test_sliding_working_set_approximation: debug

# Run all failed tests locally:
scripts/pytest -vv -n $(nproc) -k "test_sliding_working_set_approximation[debug-pg14]"

Flaky tests (1)

Postgres 14

• test_metric_collection: debug

Test coverage report is not available

_{The comment gets automatically updated with the latest test results
2faa0df at 2024-06-18T17:46:27.550Z :recycle:}

[knizhnik]

I wonder why do we need tp assign current timestamp to all elements with index less or equal then count?
Why it is not enough just to assign:

cState->regs[index][count] = now;

[MMeent]

This way we don't have to start at the end of the array of timestamps when we try to get the summary.

Your suggestion would work as well, but it'd require a scan from high counts, rather than low counts. As higher counts are less often set when the summary would result in a low distinct count, generating a summary could be more expensive on average.

I don't mind it going either way though.

[knizhnik]

Sorry, I do not understand it.
We need to find maximal count, right?
So your loop:

for (size_t i = 0; i < HLL_C_BITS + 1; i++)
	{
		if (reg[i] >= since)
		{
			max = i;
		}
	}

is actually locating the largest R with timestamp within specified period.

[MMeent]

correct, but by also updating all registers < count, you can do a binary search or linear search with early break, as all values in the array are sorted (with potential duplicates, but still, sorted).

	for (size_t i = 0; i < HLL_C_BITS + 1; i++)
	{
		if (reg[i] >= since)
			max = i;
		else
			break;
	}

Thus saving a few compares for this bucket.

[knizhnik]

1. Binary search in array of size 22 seems to have no sense.
2. Time of calculating estimation is not critical: it is needed rarely when autoscaler agent needs to update its working set estimation: once per second or even per minute.
3. Time of updating hashes is much more critical because it is called on each SMGR read operation

So we should optimise addHLL rather than getMaximum

[MMeent]

That makes sense.