[BUG] Unable to create a new RollUp Job in OpenSearch 2.12

[karthikeyan21]

What is the bug?
RollUp Job creation fails with 500 error code in Opensearch 2.12

Error Message :
{"error":{"root_cause":[{"type":"null_pointer_exception","reason":"Cannot invoke \"java.time.Instant.plusMillis(long)\" because \"startTime\" is null"}],"type":"null_pointer_exception","reason":"Cannot invoke \"java.time.Instant.plusMillis(long)\" because \"startTime\" is null"},"status":500}

How can one reproduce the bug?
Steps to reproduce the behavior:

1. Use the below API to create a new RollUp Job - Create RollUp Job
   Sample -
   curl -X PUT localhost:9200/_plugins/_rollup/jobs/test -H 'Content-Type:application/json' -d '{"rollup":{"target_index":"rollup_hourly_fmstats_test","description":"Hourly Stats Rollup","source_index":"test_*","enabled":true,"schedule":{"interval":{"period":60,"unit":"Minutes"}},"delay":0,"continuous":"true","metrics":[{"source_field":"abc.accepted","metrics":[{"max":{}}]},{"source_field":"abc.rejected","metrics":[{"max":{}}]},{"source_field":"abc.matched","metrics":[{"max":{}}]}],"page_size":5000,"dimensions":[{"date_histogram":{"fixed_interval":"60m","source_field":"timestamp"}},{"terms":{"source_field":"name"}}]}}'

2. RollUp Job creation is fails with error (500)

What is the expected behavior?
RollUp Job to be created and data to be rolled up

What is your host/environment?

• OS: Ubuntu
• Version 22
• Plugins - index-state-management (ISM)

Do you have any screenshots?
NA

Do you have any additional context?
I was debugging the code and noticed that we have not initialised Schedule
Modifying the code to Instant.now() instead of schedule.startTime fixed the issue

Update - This doesn't affect the existing RolUp Jobs. Any job created using earlier version (2.10) seems to be working as the time is initialised

[mgodwan]

Related to #1040

@bowenlan-amzn @ikibo Could you please check?

[bowenlan-amzn]

Yes, I think this is a miss and causes a breaking change.
Regarding this #1040 (comment) , if user doesn't pass in start_time, the schedule.startTime will be null, and will cause the exception when instantialize the IntervalSchedule.

The solution is to add schedule.startTime ?: Instant.now() back

[sarthakaggarwal97]

@bowenlan-amzn so looks like schedule.startTime is not a required field. What do you think, should it be a required field?

[ikibo]

@mgodwan, thank U for this finding.

Good point, @bowenlan-amzn : the case when start_time is not defined in the request must have been handled.
But the question is how?

according to the official rollup-api-doc schedule.interval.start_time is a required field (@sarthakaggarwal97 FYI).

@bowenlan-amzn plz help me understand what would be the best way to handle this issue

• bad-request : start_time is not defined (according to the doc)
• handling null check as U suggest (in this case, I would suggest changing the doc to determine that start-time is set to current time if not set explicitly in the request, making it 'kind-of' not mandatory)

@bowenlan-amzn the same issue exists for the Transform job( the fix should be pretty much the same as for the roll-up). I think we can handle both under this ticket. Plz assign this issue to me.

[bowenlan-amzn]

@ikibo Thanks!
The goal here is to not introduce breaking change. I think the documentation is wrong, obviously start_time is not a required field, as the example provided in this issue, if provided schedule like this

"schedule": {
            "interval": {
                "period": 60,
                "unit": "Minutes"
            }
        },

rollup can be created before, and start time default to current time.
so please go with the second path

handling null check as U suggest (in this case, I would suggest changing the doc to determine that start-time is set to current time if not set explicitly in the request, making it 'kind-of' not mandatory)

also link the transform change #1040

[louzadod]

The workaround I used was replacing schedule.interval by schedule.cron. But I miss schedule.interval a lot.

[bowenlan-amzn]

@louzadod this has been fixed 2.14