please update test sdrvrf3.f

[juntangc]

The test fails on both x86 and graviton processors:

git clone https://github.com/xianyi/OpenBLAS
cd OpenBLAS; make clean; make -j48; make lapack-test
/home/ubuntu/OpenBLAS/lapack-netlib/TESTING/LIN/xlintstrfs < /home/ubuntu/OpenBLAS/lapack-netlib/TESTING/stest_rfp.in

one test case fail on c5, c6g, c7g
c5: Failure in STFSM, CFORM='T', SIDE='L', UPLO='U', TRANS='T', DIAG='N', M= 50, N = 50, test= 36.444
c6g: Failure in STFSM, CFORM='T', SIDE='L', UPLO='U', TRANS='T', DIAG='N', M= 50, N = 50, test= 33.319
c7g: Failure in STFSM, CFORM='T', SIDE='L', UPLO='U', TRANS='T', DIAG='N', M= 50, N = 50, test= 33.319

The test compare the resulting matrices from STRSM (reference) and STFSM to determine the correctness of STFSM function. The test assume that STRSM generate correct results, but in reality the function has numeric errors (when comparing to double precision result). Attached table shows the comparison of STFSM, STRSM, DTFSM, and DTRSM and there is no evidence showing STFSM result being wrong.

Attached is also the patch used to generated the table.

Please update the test.

sdrvrf3.zip

sim_555_openBlas_correctness_v0.xlsx

[martin-frbg]

Thanks for providing this analysis - the testsuite (like almost all of LAPACK) is taken directly from the "netlib" reference implementation where it compares their unoptimized STRSM and STFSM. So if I understand you correctly, it appears to be the OpenBLAS STRSM kernels on the affected platforms (or the common interface code they share) that needs correcting, rather than the test.

[martin-frbg]

Also, failures in the LAPACK testsuite are not necessarily catastrophic, and this one would be in the benign "minor" category according to http://www.netlib.org/lapack/faq.html#_how_do_i_interpret_lapack_testing_failures

[juntangc]

Thanks for the information about the testsuite. Based on the comparison against double precision results, I don't think that improving STRSM or STFSM is necessary (probably not easy to do if possible). My suggestion is

1. (still use STRSM as reference) to adjust the input matrices or test configurations or the way to normalize the total error to make the testcase to pass.
   or 2. update the testcase to use either double precision function output or ground truth matrix as the reference (you can use ground truth matrix and matrix A to create matrix B; and use A & B as input to STFSM).

I understand that the fail is probably not a catastrophic problem, but people looking at these test results aren't all aware of this fact. I am curious to see how this test do on netlib version of implementations.

Regards,

Thanks for providing this analysis - the testsuite (like almost all of LAPACK) is taken directly from the "netlib" reference implementation where it compares their unoptimized STRSM and STFSM. So if I understand you correctly, it appears to be the OpenBLAS STRSM kernels on the affected platforms (or the common interface code they share) that needs correcting, rather than the test.

[AlexanderZotkevich]

I agree strsm and stfsm are OK. And the input matrices for the test are well-conditioned by design. I believe minor threshold exceedance is caused by different operations order from that was expected xx years ago. ) I'd suggest to increase threshold for the test.

[juntangc]

Increasing the threshold is also an option, and it will affect several other testcases.

[brada4]

row 50 in excel looks corrupt.
whats ypur CPU?

[juntangc]

Row 50 looks large. The relatively larger difference is because the way the error attribute is calculated. I have tried this test on intel and graviton and the testcase is bad for both.

test error = max(sum(abs(B2(row, j) - B1(row,j)))) * c
test error is the max row sum between the result matrix and the reference matrix (then normalized by EPS and number of elements in each row).

Comparing to the next row (row 48) with relative larger error. The matrix element in row 50 is way larger and so is the absolute difference. The relative difference for row 50 and row 48 is about the same level and reasonable for single precision round up error. Attached excel shows the comparisons for row 48 and 50.

The test can probably be modified to use relative error instead of absolute- so that the outcome won't depend so much on the input matrix.

STFSM_comparisons.xlsx

[martin-frbg]

I am curious to see how this test do on netlib version of implementations.

I had not expressly mentioned this, but the test passes when using pure netlib BLAS+LAPACK on the platforms I tried. I have opened Reference-LAPACK/lapack#679 to get opinions of core LAPACK developers rather than resorting to unilateral changes in the OpenBLAS copy of the tests.

[juntangc]

I agree that is the appropriate approach. Thanks for raising this issue with lapack team,

I am curious to see how this test do on netlib version of implementations.

I had not expressly mentioned this, but the test passes when using pure netlib BLAS+LAPACK on the platforms I tried. I have opened Reference-LAPACK/lapack#679 to get opinions of core LAPACK developers rather than resorting to unilateral changes in the OpenBLAS copy of the tests.

[juntangc]

Thanks for the update,