make e2e-csv-full-report-all-backends target

[jayunit100]

Lets make the tests run from a golang program instead of bash. then use the go to customize how we output the results (as a csv) and what ginkgo --focus and --skips we use.

• make it so different backends can run different tests (for example, skip topology tests on ipvs..., skip serviceType=LB on all tests, ... and so on) via making a map of backend->test skips.
• add a target that just tests everything, and outputs a CSV, that we can systematically use to determine what functionality is supported overall, so we can detect any holes in our coverage (i.e. if serviceAffinity breaks on ipvs, thats ok as long as it passes on iptables... but if it breaks on all backends, there might be a bug in the brain, so we want to know what tests pass on some backends, vs fail on all backends.)

Details

right now we have no way of knowing the full summary of all tests on ipv4 and ipv6 unless we aggregate the results manually... and run

make e2e-ipv4-iptables
make e2e-ipv4-ipvs
...

1. Integrate w/ prow results https://k8s-testgrid.appspot.com/sig-network-gce#presubmit-network-ingress,%20google-gce&show-stale-tests=

2. Just make our own make target make e2e-kpng-sig-network-csv target, where we:

• Run each e2e- target
• aggreate the results, per test, into a single table. This would require writing a parser to parse the e2e logs (i.e. from make e2e-ipv4-tables) ... Or else to just aggregate the XML and have e2e tests output xml data. Theres a way to do that w/ ginkgo.

3. One other thing that would be maybe the ideal is just to have a golang program like this:

"ipvs":[
 "sig-network] Services should preserve source pod IP for traffic thru service cluster IP  [LinuxOnly]",
 "[sig-storage] ConfigMap should be immutable if `immutable` field is set [Conformance]"
], 
"iptables:[ ... ],

and so on. IF we were to do this, we would effectively also be solving the problem of specifying different tests, for different backends.

]

[jayunit100]

/good-first-issue

[k8s-ci-robot]

@jayunit100:
This request has been marked as suitable for new contributors.

Guidelines

Please ensure that the issue body includes answers to the following questions:

• Why are we solving this issue?
• To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points?
• Does this issue have zero to low barrier of entry?
• How can the assignee reach out to you for help?

For more details on the requirements of such an issue, please see here and ensure that they are met.

If this request no longer meets these requirements, the label can be removed
by commenting with the /remove-good-first-issue command.

In response to this:

/good-first-issue

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[emabota]

/assign

[jayunit100]

cc @jaehnri

[jaehnri]

@jayunit100 This is the issue we talked about last meeting, right?

I have no access to my Linux system right now so it's hard to do much. If nobody touches it until Saturday, I'll pick it up.

/assign