HDF5 datatype ID leak in `h5str_detect_vlen_str()`

[K-ANOY]

I found a possible HDF5 datatype ID leak in h5str_detect_vlen_str(). When the function inspects an array or variable-length datatype, it obtains the base type with H5Tget_super(), but does not close that ID when the recursive check returns 0.

File: java/src-jni/jni/h5util.c

Function: h5str_detect_vlen_str

Relevant code:

static htri_t
h5str_detect_vlen_str(hid_t tid)
{
    H5T_class_t tclass = H5T_NO_CLASS;
    htri_t      ret    = 0;

    ret = H5Tis_variable_str(tid);
    if ((ret == 1) || (ret < 0))
        goto done;

    tclass = H5Tget_class(tid);
    if (tclass == H5T_ARRAY || tclass == H5T_VLEN) {
        hid_t btid = H5Tget_super(tid);

        if (btid < 0) {
            ret = (htri_t)btid;
            goto done;
        } /* end if */
        ret = h5str_detect_vlen_str(btid);
        if ((ret == 1) || (ret < 0)) {
            H5Tclose(btid);
            goto done;
        } /* end if */
    }     /* end if */

    ...

done:
    return ret;
}

H5Tget_super() returns a new datatype ID that must be closed with
H5Tclose(). The current code closes btid only when the recursive call finds
a variable-length string (ret == 1) or reports an error (ret < 0). If the
recursive call returns 0, which is the normal "not a variable-length string"
case, control falls through to done and btid is leaked.

The compound datatype branch in the same function closes member datatype IDs on
both the positive/error path and the normal ret == 0 path, which suggests the
array/vlen branch should follow the same ownership pattern.

This helper is called by h5str_detect_vlen(), which is used by JNI read/write
paths such as H5Dread, H5Dwrite, H5Aread, and H5Awrite wrappers before
choosing the variable-length string handling path. Therefore the leak can occur
when Java code reads or writes datasets/attributes whose datatype is an array or
vlen type that does not contain a variable-length string.

Suggested fix: close btid after the recursive check regardless of the returned
value, for example:

ret = h5str_detect_vlen_str(btid);
H5Tclose(btid);
if ((ret == 1) || (ret < 0))
    goto done;

or use a cleanup path that closes btid whenever it has been successfully
created.