OpenCI TF-A False positive Jenkins Test result (Bug)

Description

We have come across a situation where a TF-A Jenkins job is reporting success when an underlying test is failing and reported as failing in LAVA.

We discovered this when we were testing a patch that we expected to generate a CI failure but it did not. It does in Arm’s internal CI and in the OpenCI it is recognised as a failure by LAVA but just not cascaded up to the Jenkins job.

This job we expected to fail

It can be seen that if you inspect the lava logs for either of these jobs that tftf actually failed all tests:

and lava does pick up that the tests failed:

but seems to be returning success for the job overall anyway:

Contact for details.

Environment

None

Engineering Progress Update

None

Activity

Joanna Farley 
June 21, 2022 at 1:13 PM

who this issue was raised for is away until 4th July. We can ping him when he is back to confirm.

Arthur She 
June 8, 2022 at 2:39 PM

The patch has been merged.

Arthur She 
June 8, 2022 at 2:39 PM

The patch has been merged.

Arthur She 
June 7, 2022 at 5:34 PM

, ,

As Stevan mentioned, the completed job doesn’t mean the test pass. In the tftf-reboot_reset2-juno case, the job was completed, since there were no any infrastructure errors. We need to check the test results to determine the final result of the LAVA test.

I’ve submitted a patch to fix it, please have a look.

 

Thanks

Madhukar Pappireddy 
June 6, 2022 at 3:44 PM

Hi
As mentioned in the previous comments, I would expect the top level jenkins job (that launches a lava job on a Juno board) to fail if the test being run by Lava dispatcher on the target device fails. From our side, we teach the LAVA dispatcher how to interpret the results posted by our tests as seen here [1]. I would expect some mechanism to be built-in to the LAVA dispatcher to then return the fail status to the parent(Jenkins) job.

[1]

Delivered

Details

Assignee

Reporter

Upstream

Priority

Checklist

Sentry

Created May 25, 2022 at 1:04 PM
Updated October 23, 2023 at 1:44 PM
Resolved June 8, 2022 at 2:39 PM