Not able to set stripe and stripe sie using mpi_info_set

[monsieuralok]

We are using PnetCDF in CESM and usually, it sets automatic stripes on files using following call:
call mpi_info_set(info,"striping_factor",stripestr,ierr)
call mpi_info_set(info,"striping_unit",stripestr2,ierr)

while using cray mpich and intel MPI we are not having any problem. We are having a lustre file system and it is OpenMPI 4.0.3 compiled using intel/2020a compiler.
Please could you look over it.
Thanks a lot Alok

[ggouaillardet]

It is possible striping_factor and striping_unit keys are ignored by our io/ompio component (but used by the io/romio... component.

Are striping parameters used if you

mpirun --mca io ^ompio ...

[monsieuralok]

Hi,
Thanks a lot for reply and I have tried now mpirun -np 4 --mca io ^ompio ./a.out still stripe counts are not set. I guess on our system they have only installed ompio not romio as when I tried:
mpirun -np 4 --mca io romio ./a.out
I get message "A requested component was not found, or was unable to be opened. This
means that this component is either not installed or is unable to be
used on your system (e.g., sometimes this means that shared libraries
that the component requires are unable to be found/loaded). Note that
Open MPI stopped checking at the first component that it did not find."
Thanks a lot,
I will ask my sysadmin to install it.

[ggouaillardet]

With Open MPI 4.0.3, the ROMIO component is io/romio321, so you should try

mpirun -np 4 --mca io romio321 ./a.out

[monsieuralok]

mpirun -np 4 --mca io romio321 ./a.out
node 0 : Hello world
node 2 : Hello world
node 3 : Hello world
node 1 : Hello world
[agu002@login-2.BETZY /cluster/work/users/agu002/pnetcdf]$ lfs getstripe test.nc
test.nc
lmm_stripe_count: 1
lmm_stripe_size: 1048576
lmm_pattern: raid0
lmm_layout_gen: 0
lmm_stripe_offset: 18
lmm_pool: projects
obdidx objid objid group
18 80733022 0x4cfe35e 0x6c0000402

still, it does not set the stripe size and stripe count which should be 8. Below are the results from IntelMPI:
lmm_stripe_count: 8
lmm_stripe_size: 2097152
lmm_pattern: raid0
lmm_layout_gen: 0
lmm_stripe_offset: 18
lmm_pool: projects
obdidx objid objid group
18 80733271 0x4cfe457 0x6c0000402
16 82127405 0x4e52a2d 0x640000402
15 82447408 0x4ea0c30 0x840000402
17 81287868 0x4d85abc 0x800000402
13 86116640 0x5220920 0x7c0000402
14 82058460 0x4e41cdc 0x680000402
12 80181979 0x4c77adb 0x880000402
22 83375768 0x4f83698 0x600000402

[monsieuralok]

Here is my test program in fortran:

      program stripe

include <pnetcdf.inc>
include <mpif.h>
integer4 ,save :: info=MPI_INFO_NULL
character(len=3) :: stripestr
character(len=9) :: stripestr2
integer ierr,status,ncid,size,rank
character(8) :: fnm
fnm="test.nc"
write(stripestr,('(i3)')) 8
write(stripestr2,('(i9)')) 20481024
call MPI_INIT(ierr)
call MPI_COMM_SIZE(MPI_COMM_WORLD, size, ierr)
call MPI_COMM_RANK(MPI_COMM_WORLD, rank, ierr)
print, 'node', rank, ': Hello world'
call mpi_info_create(info,ierr)
call mpi_info_set(info,'romio_ds_read','disable',ierr)
call mpi_info_set(info,'romio_ds_write','disable',ierr)
call mpi_info_set(info,'striping_factor',stripestr,ierr)
call mpi_info_set(info,'striping_unit',stripestr2,ierr)
ierr = nfmpi_create(MPI_COMM_WORLD,fnm,
& IOR(nf_clobber,nf_64bit_offset),INFO,ncid)
CALL MPI_INFO_FREE(INFO, IERR)
ierr=nfmpi_close(ncid)
call MPI_FINALIZE(ierr)
stop
end

[ggouaillardet]

I briefly checked the io/romio321 component, and it should handle striping_factor correctly.

A possible explanation is lustre support was not detected at configure time.

What if you replace test.nc with lustre:test.nc?
it should force ROMIO to use the ad_lustre component and fail if it is not builtin.

[monsieuralok]

I replaced
fnm="test.nc" in program with fnm="lustre:test.nc"
and I get following error:
MPI error (MPI_File_open) : MPI_ERR_IO: input/output error

so you want to say that OpenMPI is not built/configured with lustre support. I will write to our support. Is there any other way to check it? Thanks a lot Alok

[ggouaillardet]

Right, the message means that the ROMIO component has no lustre support.

I am not sure how to verify Lustre support was built from the install tree, so better check the output of configure (or ompi/mca/io/romio321/romio/config.log if the admins kept it)

[edgargabriel]

just to clarify briefly an item, ompio does recognize on luster file systems the info objects "stripe_size", and "stripe_width"(We might have to adjust the names of the info objects to match the MPI spec, I actually missed this aspect).

However, in the 4.0.x series romio is the default io component on Lustre file systems, that changed starting from the 4.1.x release series. And second, if I recall my tests correctly, setting stripe size and stripe width only works if the file does not exist at the time file opening, lustre will not allow to change the settings of an existing, non-empty file.

[ericch1]

Hi,
(I post here, since it seems I can't post to discussion list right now???)

In 2012 we wrote and tested our functions to use MPI I/O to have good performances while doing I/O on a Lustre filesystem. Everything was fine about "striping_factor" we passed to file creation.

Now I am trying to verify some performance degradation we observed and I am surprised because it looks like I am unable to create a new file with a given "striping_factor" with any mpi flavor.

I attached a simple example for file creation with hints, and tried it the following way with OpenMPI:

OpenMPI-4.0.3:

which mpicc
/cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/Compiler/gcc9/openmpi/4.0.3/bin/mpicc

mpicc -o s simple_file_create_with_hint.c

rm -f foo && mpiexec -n 1 --mca io romio321 ./s foo && lfs getstripe foo

Creating the file by MPI_file_open : foo
Informations on file:
Key is 'striping_factor' and worth: '2'
 ...closing the file foo
Informations on file:
foo
lmm_stripe_count:  1
lmm_stripe_size:   1048576
lmm_pattern:       raid0
lmm_layout_gen:    0
lmm_stripe_offset: 49
       obdidx           objid           objid           group
           49       485863591     0x1cf5b0a7                0

Not forcing romio321:

rm -f foo && mpiexec -n 1 ./s foo && lfs getstripe foo  
               
Creating the file by MPI_file_open : foo
Informations on file:
Key is 'striping_factor' and worth: '2'
 ...closing the file foo
Informations on file:
foo
lmm_stripe_count:  1
lmm_stripe_size:   1048576
lmm_pattern:       raid0
lmm_layout_gen:    0
lmm_stripe_offset: 19
       obdidx           objid           objid           group
           19       482813449     0x1cc72609                0

First, as you can see, even if I ask for a striping_factor of 2, I only get one! I tried to write some data too, but it changed nothing...

ompi_info seems to tell lustre component is there (fs-lustre):

ompi_info |grep lustre
  Configure command line: '--prefix=/cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/Compiler/gcc9/openmpi/4.0.3' '--build=x86_64-pc-linux-gnu' '--host=x86_64-pc-linux-gnu' '--enable-shared' '--with-verbs' '--with-hwloc=external' '--without-usnic' '--disable-wrapper-runpath' '--disable-wrapper-rpath' '--with-munge' '--with-slurm' '--with-pmi=/opt/software/slurm' '--enable-mpi-cxx' '--with-hcoll' '--disable-show-load-errors-by-default' '--enable-mpi1-compatibility' '--enable-mca-dso=common-ucx,common-verbs,event-external,atomic-ucx,btl-openib,btl-uct,coll-hcoll,ess-tm,fs-lustre,mtl-ofi,mtl-psm,mtl-psm2,osc-ucx,plm-tm,pmix-ext3x,pmix-s1,pmix-s2,pml-ucx,pnet-opa,psec-munge,ras-tm,spml-ucx,sshmem-ucx,hwloc-external' '--enable-mpirun-prefix-by-default' '--with-ofi=/cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/Core/libfabric/1.10.1' '--with-ucx=/cvmfs/soft.computecanada.ca/easybuild/software/2020/avx2/Core/ucx/1.8.0'
                  MCA fs: lustre (MCA v2.1.0, API v2.0.0, Component v4.0.3)

Where am I wrong?

Second, I was expecting that when I re-open the file for read-only, I would have some information in "MPI_Info" but it is empty... is that normal?

For example, using mpich-3.2.1 I have the following output:

which mpicc
/cvmfs/soft.computecanada.ca/easybuild/software/2017/avx2/Compiler/gcc7.3/mpich/3.2.1/bin/mpicc

mpicc -o s simple_file_create_with_hint.c             

                 
rm -f foo && mpiexec -n 1 ./s foo && lfs getstripe foo 

Creating the file by MPI_file_open : foo
Informations on file:
Key is 'cb_buffer_size' and worth: '16777216'
Key is 'romio_cb_read' and worth: 'automatic'
Key is 'romio_cb_write' and worth: 'automatic'
Key is 'cb_nodes' and worth: '1'
Key is 'romio_no_indep_rw' and worth: 'false'
Key is 'romio_cb_pfr' and worth: 'disable'
Key is 'romio_cb_fr_types' and worth: 'aar'
Key is 'romio_cb_fr_alignment' and worth: '1'
Key is 'romio_cb_ds_threshold' and worth: '0'
Key is 'romio_cb_alltoall' and worth: 'automatic'
Key is 'ind_rd_buffer_size' and worth: '4194304'
Key is 'ind_wr_buffer_size' and worth: '524288'
Key is 'romio_ds_read' and worth: 'automatic'
Key is 'romio_ds_write' and worth: 'automatic'
Key is 'cb_config_list' and worth: '*:1'
Key is 'romio_filesystem_type' and worth: 'UFS: Generic ROMIO driver for all UNIX-like file systems'
Key is 'romio_aggregator_list' and worth: '0 '
 ...closing the file foo
Informations on file:
Key is 'cb_buffer_size' and worth: '16777216'
Key is 'romio_cb_read' and worth: 'automatic'
Key is 'romio_cb_write' and worth: 'automatic'
Key is 'cb_nodes' and worth: '1'
Key is 'romio_no_indep_rw' and worth: 'false'
Key is 'romio_cb_pfr' and worth: 'disable'
Key is 'romio_cb_fr_types' and worth: 'aar'
Key is 'romio_cb_fr_alignment' and worth: '1'
Key is 'romio_cb_ds_threshold' and worth: '0'
Key is 'romio_cb_alltoall' and worth: 'automatic'
Key is 'ind_rd_buffer_size' and worth: '4194304'
Key is 'ind_wr_buffer_size' and worth: '524288'
Key is 'romio_ds_read' and worth: 'automatic'
Key is 'romio_ds_write' and worth: 'automatic'
Key is 'cb_config_list' and worth: '*:1'
Key is 'romio_filesystem_type' and worth: 'UFS: Generic ROMIO driver for all UNIX-like file systems'
Key is 'romio_aggregator_list' and worth: '0 '
foo
lmm_stripe_count:  1
lmm_stripe_size:   1048576
lmm_pattern:       raid0
lmm_layout_gen:    0
lmm_stripe_offset: 67
       obdidx           objid           objid           group
           67       357367195     0x154cfd9b                0

but still have only a striping_factor of 1 on the created file... while lfs dh -h gives me 85 OSTs...

Thanks,

Eric

[ericch1]

Oups, I forgot... here is the example I used:

#include "mpi.h"
#include <stdio.h>
#include <stdlib.h>

/*
 * Simple function to abort execution and print an error based on MPI return values:
*/
void abortOnError(int ierr) {
  if (ierr != MPI_SUCCESS) {
    printf("ERROR Returned by MPI: %d\n",ierr);
    char* lCharPtr = (char*) malloc(sizeof(char)*MPI_MAX_ERROR_STRING);
    int lLongueur = 0;
    MPI_Error_string(ierr,lCharPtr, &lLongueur);
    printf("ERROR_string Returned by MPI: %s\n",lCharPtr);
    free(lCharPtr);
    MPI_Abort( MPI_COMM_WORLD, ierr );
  }
}

/*
 * Here I use only the first hint for now, but you can try a few to see the difference:
*/
const int lNbHints = 1;
char *lHints      [] = {"striping_factor",
                        "striping_unit",
                        "access_style"};

char *lHintsValues[] = {"2",
                        "1048576",
                        "write_once,random"};

/*
 * Simple function to extract all information in MPI_Info:
*/
void displayFileInfo(MPI_File pFile)
{
  printf( "Informations on file:\n");fflush(stdout);

  char lKey[MPI_MAX_INFO_KEY];
  char lValue[MPI_MAX_INFO_VAL];
  int lFlag = 0;
  MPI_Info lInfo;
  int lNKeys = 0;
  abortOnError(MPI_File_get_info( pFile, &lInfo ));
  abortOnError(MPI_Info_get_nkeys( lInfo, &lNKeys ));

  /* Note that an implementation is allowed to ignore the set_info, so we'll accept either the original or the updated version */
  for (int i = 0 ; i < lNKeys; ++i ) {
    abortOnError(MPI_Info_get_nthkey( lInfo, i, lKey ));
    abortOnError(MPI_Info_get       ( lInfo, lKey, MPI_MAX_INFO_VAL, lValue, &lFlag ));
    if (lFlag) {
      printf( "Key is '%s' and worth: '%s'\n", lKey, lValue);fflush(stdout);
    }
    else {
      printf( "Error getting key is '%s' \n", lKey);fflush(stdout);
    }
  }

  MPI_Info_free( &lInfo );

}

/*
 * Function to create a file with *hints*:
*/
void createFileCollectivelyWithHint(char* pFileName)
{
  int size, rank;

  MPI_Comm_rank( MPI_COMM_WORLD, &rank );
  MPI_Comm_size( MPI_COMM_WORLD, &size );

  MPI_File lFile = 0;

  MPI_Info lInfoIn;
  abortOnError(MPI_Info_create( &lInfoIn ));

  for (int i = 0 ; i < lNbHints; ++i ) {
    abortOnError(MPI_Info_set( lInfoIn, lHints[i], lHintsValues[i] ));
  }

  printf("Creating the file by MPI_file_open : %s\n", pFileName);
  abortOnError(MPI_File_open( MPI_COMM_WORLD, pFileName, ( MPI_MODE_CREATE | MPI_MODE_WRONLY ), lInfoIn, &lFile ));

  displayFileInfo(lFile);

  if (lFile) {
    printf("  ...closing the file %s\n", pFileName);
    abortOnError(MPI_File_close(&lFile ));
  }
}

int main(int argc, char *argv[])
{
    MPI_Init(&argc, &argv);

    if (2 != argc) {
      printf("ERROR: you must specify a filename to create.\n");
      MPI_Finalize();
      return 1;
    }
    char* lFileName = argv[1];


    createFileCollectivelyWithHint(lFileName);

    /* Re-open the file in read-only mode just to see MPI_Info values */
    MPI_Info lInfo = MPI_INFO_NULL;
    MPI_File lFile = 0;
    abortOnError(MPI_Info_create( &lInfo ));
    abortOnError(MPI_File_open( MPI_COMM_WORLD, lFileName, ( MPI_MODE_RDONLY ), lInfo, &lFile ));
    displayFileInfo(lFile);
    abortOnError(MPI_File_close(&lFile ));

    MPI_Finalize();

    return 0;
}

[ericch1]

Hi,

@edgargabriel @ggouaillardet I am really confused about hint names. Please help me understand.

Lustre document "stripe_count" as: "Indicates the number of OSTs that this file will be striped across."
(https://github.com/DDNStorage/lustre_manual_markdown/blob/master/06.05-Setting%20Lustre%20Properties%20in%20a%20C%20Program%20(llapi).md#llapi_file_create)

OpenMPI calls:

llapi_file_create(filename,
                              fs_lustre_stripe_size,     /*** Eric's comment: this is stripe_size in lustre api doc. ***/
                              -1, /* MSC need to change that */
                              fs_lustre_stripe_width, /*** Eric's comment: this is stripe_count in lustre api doc. ***/
                              0); /* MSC need to change that */

with fs_lustre_stripe_width taken from hint stripe_width if given into a MPI_Info.

so far so good.

But, in commit 1631f38, support for "striping_factor" and "striping_unit" has been introduced from MPI standard, which defines

"striping_factor" (integer) [SAME]: This hint specifies the number of I/O devices that
the file should be striped across, and is relevant only when the file is created.

(see https://www.mpi-forum.org/docs/mpi-4.0/mpi40-report.pdf, page 656)

So I understand that "striping_factor" in MPI is the "stripe_count" in Lustre.

However, the commit 1631f38 links striping_factor to fs_lustre_stripe_size as shown here:

  opal_info_get (info, "striping_factor", &stripe_str, &flag);
    if ( flag ) {
        sscanf ( stripe_str->string, "%d", &fs_lustre_stripe_size );
        OBJ_RELEASE(stripe_str);
    }

then striping_factor is given as the 4th parameter to lustre api which is strip_size instead of the second, which is strip_count...

Am I confused or this a bug?

Thanks,

Eric

[edgargabriel]

@ericch1 sorry for the late reply, I was out of office last week. I will have a look in a bit, there is a good chance that you are right and this is a bug.

[ericch1]

No problem @edgargabriel , there is no hurry for me... Thanks for looking into this! :)

[edgargabriel]

@ericch1 would you have a chance to test https://github.com/edgargabriel/ompi/tree/topic/lustre-info-swap to see whether this fixes the info object assignment on lustre? I do not have currently access to a lustre file system to test this out, but I think you are right that I accidentally swapped the meaning of the two MPI info objects on lustre.