Add rucio upload

[panta-123]

Any upload fo files should be registred to rucio.

1. We can directly use rucio upload
2. Or contnue with xrdcp or cp and register files to rucio

1 is better and atomic.

Can someone describe me which files are uploaded and is there check on local vs remote copy ?

[s6pepaul]

Hi,
currently our workflow is:

1. Produce files on OSG
2. Write them to JLab via pelican
   Once all files of a project are accumulated:
3. Merge files appropriately to reduce number of small files
4. Write the merged files to tape

It is not clear to me at the moment at which point you would like rucio to come in.

[panta-123]

@s6pepaul , Merge files are the ones used by users and they don't need the files before merged , right?

[T-Britton]

correct. There are rare exceptions during certain specific outages but other methods can be employed in dire situations

[panta-123]

And the merging is manual processes or automated ?

[T-Britton]

mainly automated with some manual interventions from time to time. I should say when it is working we don't touch any of it.

[panta-123]

Thanks. So we should register to rucio after the merging end.
We will add register to rucio functionality at the end of the merge in that autoated scripts.

[T-Britton]

sounds reasonable. We do have records of "all" data already processed in a DB