functional tests: fix automatic ref updating

[sshane]

Type of Changes

	Type
✓	🐛 Bug fix

Description

Allows automatic test ref updating to function properly.

Closes #9688

[Pierre-Sassoulas]

Seeing the result of the pipeline, it's likely you found an actual issue. The testutil are really poorly tested (in fact not at all directly, only indirectly from running other tests). Might be the time to actually add direct tests for them. Do you feel like doing this ? You can also upgrade the now failing tests otherwise. I'll need some time to understand what's happening to be able to review in any case.

[sshane]

Looks like it was asserting the previously-first assert's message?

[sshane]

I reverted the change to the output of LintModuleTest and made it update the output file after a failed test. I think the real solution is refactoring this test such that it isn't testing for two things/split it up. That way testutil can test for both assertion cases, but I probably don't have time for that anytime soon.

[github-actions]

🤖 According to the primer, this change has no effect on the checked open source code. 🤖🎉

This comment was generated for commit 63cbd3e

[Pierre-Sassoulas]

I think the first commit / intention revealed an issue with the functional test : upgrading the expected output before checking the message raised or not should not make the tests fail. Maybe the expected output generation is wrong.

I think the real solution is refactoring this test such that it isn't testing for two things/split it up. That way testutil can test for both assertion cases, but I probably don't have time for that anytime soon.

Yeah, I agree, those monster classes can use composition and two sub classes anytime (and tested with more ease as a result). Not sure this is strictly necessary to fix the issue you unearthed though.

I probably don't have time for that anytime soon.

No worries, thank you for bringing the issue to our attention 😄