Custome shard templates with YYYY/MM/dd/HH/mm replacements

[eridal]

Hi all,

We are trying to setup a custom shard template like gs://bucket/YYYY/MM/dd/HH/W/P-SS-of-NN, so that the bucket can still be easily browse manually.

We are having issues with that custom shard template as it seems like those replacements are not supported. The short description listed when creating the job does not says anything about the likes of year/month/date/hour replacements. It just says:

The shard template defines the unique/dynamic portion of each windowed file. Recommended to use the default (W-P-SS-of-NN). At runtime, 'W' is replaced with the window date range and 'P' is replaced with the pane info.

Searching on this repo we found these references but we are not sure if these are available as part of to the custom shard template.

Questions:

• Are YYYY, MM, dd, HH available to the custom shard template?

• What are the supported replacements?

Any pointers will be greatly appreciated.

[sabhyankar]

@eridal Great questions. You can indeed create an output path with year(YY), month(MM), day(DD) and hour(HH). However, they need to be part of the outputDirectory parameter as follows:
outputDirectory=gs://bucket/YYYY/MM/DD/HH/

The filename can then use the shard template with window and pane info:
outputShardTemplate=SS-of-NN

You can ofcourse combine these with outputFilenamePrefix and outputFilenameSuffix. The supported replacements for the actual filename would be currentShardNumber(S), NumOfShards(N), Pane(P) and Window(W).

Hope this helps! Feel free to re-open if needed.