Deprecate TripUpdate.schedule_relationship = ADDED, add TripUpdate.schedule_relationship = NEW / REPLACEMENT to specify new / replaced trips which do not run on a schedule from the GTFS static.

[miklcct]

The use of TripUpdate.schedule_relationship = ADDED was unspecified and different producers / consumers used it in different ways. For example, it is sometimes used to specify additional departures on an existing route, but it is also used to specify departures which can't be matched to any existing trips.

This PR attempts to fully deprecate ADDED, and introduce NEW and REPLACEMENT, based on the implementation of OpenTripPlanner which specifies the whole journey to be added or replaced. Additional fields, such as headsigns, and pickup / drop-off types, are introduced as required to support the full specification of completely new trips.

NEW

In this proposal, TripUpdate.schedule_relationship = NEW should be used to add trips which do not duplicate an existing trip. Such trips are considered to be unrelated to any existing trips in the GTFS Static and can serve an arbitrary pattern, including completely new patterns not found in the GTFS Static.

A typical use case is for relief trips for extra demand, typically after big events.

NEW trips are intended as a migration target for existing feeds which use ADDED to specify new trips unrelated to the static GTFS, including OpenTripPlanner. trip_id in the TripDescriptor for new trips must be completely new (not found in GTFS static) and unique, and a start_date should also be specified as well (I am not using the word "must" here because it is permitted not to specify start_date to match scheduled trips, in this case the trip is assumed to run today).

The whole journey of the added trip must be specified, in stop order, as StopTimeUpdates inside the TripUpdate without any omission. Fields are added to TripProperties and StopTimeProperties for esssential information such as names, headsigns, pickup / drop off types.

REPLACEMENT

I propose to un-deprecate TripUpdate.schedule_relationship = REPLACEMENT as well. It works in the same way as NEW, apart from that the TripDescriptor must match one instance of a scheduled trip (like other values of TripUpdate.schedule_relationship), and that instance is replaced with the complete replacement trip specified in form of StopTimeUpdates like an added trip. The original stop times in the GTFS static are not considered by the replacement trip in any form to avoid confusion. The replacement trip can serve an arbitrary pattern with an arbitrary schedule, the only expectation is that the passenger should associate the replacement trip to actually be a replacement of the original trip.

A typical use case is for short-term timetable change, or short-term (near real-time) diversion, where the fact that the trip_id remains the same can be used by journey planners to notify the user that the booked service has been changed. (In particular, I have successfully used this feature to handle real-time train diversions in GB in OpenTripPlanner and route users to alight at diverted stops, which is something neither Google Maps nor Citymapper can do now)

This is the behaviour implemented in OpenTripPlanner, which is equivalent to deleting the matched trip, and processing the replacement TripUpdate as an ADDED trip mentioned above.

Relationship to TripModification

TripModification provides a way to modify trips en-masse by specifying a list of trip IDs where the same detour can be applied. However, it is not suited to change the schedule on a per-trip basis, replacing the trip with a completely different schedule after any diversions with different running times (common due to pathing constraints on railways).

It should be forbidden to modify the same trip via a REPLACEMENT trip update and also via a TripModification.

[gcamp]

Nice to see some movement in that direction!

I think you used a markdown editor that changed formatting on a lot of tables which makes it hard to see the actual diff from your proposal. Would it be possible to fix that?

You put a lot in the PR description that's not actually in the proposed changes. Is that just to start the discussion? Some of it is quite consequential, like the whole journey of the added trip must be specified.

I'm a bit puzzled on how a consumer is supposed to ingest ADDED changes like this with arbitrary trips with no more information than an headsign. Which route is that on? Is those added trips supported only on existing routes in the GTFS? If the answer is no, we're getting quite close to the service change proposal : https://bit.ly/gtfs-service-changes-v3_1

[leonardehrenfried]

Thanks for opening this PR!

OTP has had an implementation of ADDED for a long time but its behaviour is severely underspecified. I'd love to formalise it.

Yes, OTP allows you to create completely new free form trips that have no relation to an existing pattern or trip. It tries match the given route id to an existing one but if none is in the message a dummy one is created. For once, OTP is really following the "just give us what you have, and we will try to work it out" strategy.

The only requirement we have is that the stop ids must match the static GTFS. The question is what should happen when they don't. Should the entire update be dropped or individual stops? Does that even need to be specified?

I agree with what @gcamp said about the markdown tables and the issue description.

Lastly, you might find it easier to get this through review if you split it into two PRs: one for ADDED and one for REPLACEMENT. That's just a guess though.

[miklcct]

I think that the requirement for the whole trip to be specified is written in the code. Let me know if it is not clear enough.

I'll fix the formatting later today.

[miklcct]

Nice to see some movement in that direction!

I think you used a markdown editor that changed formatting on a lot of tables which makes it hard to see the actual diff from your proposal. Would it be possible to fix that?

You put a lot in the PR description that's not actually in the proposed changes. Is that just to start the discussion? Some of it is quite consequential, like the whole journey of the added trip must be specified.

I'm a bit puzzled on how a consumer is supposed to ingest ADDED changes like this with arbitrary trips with no more information than an headsign. Which route is that on? Is those added trips supported only on existing routes in the GTFS? If the answer is no, we're getting quite close to the service change proposal : https://bit.ly/gtfs-service-changes-v3_1

"The whole journey of the added trip must be specified" is a fact in the core of this PR, noted in the updated definition of StopTimeUpdate:

Updates to StopTimes for the trip (both future, i.e., predictions, and in some cases, past ones, i.e., those that already happened). The updates must be sorted by stop_sequence, and apply for all the following stops of the trip up to the next specified stop_time_update.
If trip.schedule_relationship is SCHEDULED, at least one stop_time_update must be provided for the trip.
If trip.schedule_relationship is ADDED or REPLACEMENT, stop_time_updates must be provided for all stops in the added or replacement trip, and the stop times in the static GTFS are not used.
If the trip is canceled or deleted, no stop_time_updates need to be provided. If stop_time_updates are provided for a canceled or deleted trip then the trip.schedule_relationship takes precedence over any stop_time_updates and their associated schedule_relationship. If the trip is duplicated, stop_time_updates may be provided to indicate real-time information for the new trip.

The route and direction of the ADDED trip is specified in TripDescriptor.route_id. Sorry I didn't make it clear and I'll going to refine the PR. It should not be possible to replace a trip to work on a different route in a REPLACEMENT trip as it may confuse consumers.

[miklcct]

I do not want to specify the behaviour of missing stops at this moment because it may depend on the client's capability for dynamically adding stops via Stop messages. Theoretically the stop_id must refer to a stop in GTFS static, or a stop added via Stop messages.

[leonardehrenfried]

So this is pretty much a codifcation of what OTP has been supporting for several years. This would of course be very convenient for us but I would like to hear more voices from the community, in particular producers.

I know that HSL (Helsinki) is using this as both a producer and consumer (OTP) for many years.

MBTA has also indicated that they use ADDED as a producer.

@optionsome @jfabi @sam-hickey-ibigroup

[skinkie]

So this is pretty much a codifcation of what OTP has been supporting for several years. This would of course be very convenient for us but I would like to hear more voices from the community, in particular producers.

Producing it for over 12 years too.

[optionsome]

I know that HSL (Helsinki) is using this as both a producer and consumer (OTP) for many years.

HSL doesn't produce or consume ADDED or REPLACEMENT updates currently, if that was what you were referring to.

[leonardehrenfried]

What happens when you ADD a trip and then CANCEL it again? Should it be become invisible in the system (DELETED?) or show up as a CANCELLED?

[miklcct]

That's a good question. I still need to think about how things will work.

My producer implementation cancels an added trip using TripUpdate.schedule_relationship = ADDED with all StopTimeUpdate having a SKIPPED relationship.

The questions are that:

1. What if a later version of the full dataset real time feed doesn't contain the ADDED feed? (My intention is that it no longer exists and should be considered as DELETED. A GTFS-RT full dataset should only be applied into the original static data.)
2. How do I cancel an ADDED trip? I think that a TripUpdate with a trip id not found in static GTFS, schedule_relationship = CANCELED and the original planned stops marked with StopTimeUpdate.schedule_relationship = 'SKIPPED' makes it clear that the trip was added then cancelled. (Use DELETED instead of CANCELED to hide it from the board)

[skinkie]

As we are considering FULL_DATASET shouldn't that just replace?

[leonardehrenfried]

Actually, @skinkie is right. If you use FULL_DATASET then the moment you fetch the new version of the RT feed the old ADDED trip will completely vanish and it neither exists as DELETED nor CANCELLED. It's like it never existed.

However, once there is movement towards specifying INCREMENTAL we will have to revisit this.

[leonardehrenfried]

Does anyone know how Google and Apple handle ADDED?

@bdferris-v2

@eliasmbd I don't know who the relevant person from Apple would be. Could you tag them?

[skinkie]

@leonardehrenfried at Google the thing was limited to stop sequences previously observed. Hence if the ADDED trip was an instance of a stop sequence that is part of the database, it could be processed. I don't know if it is already capable of processing a partial instance of a stop sequence.

https://support.google.com/transitpartners/answer/10106497?hl=en#zippy=%2Cadd-with-tripupdates

[miklcct]

@sam-hickey-arcadis The CLA check now fails because I have applied some of your review suggestions.

[sam-hickey-arcadis]

3. The atomic DELETE-then-NEW behaviour is the implementation in OpenTripPlanner, but I am open to other consumers using a different implementation as long as it achieves the same effect.

That's a fair and good point.

There needs to be more guidance on how to publish a trip cancellation for a NEW or DUPLICATED trip. The updated best practices says this:

If no stops in a trip will be visited, the trip should be CANCELED instead of having all stop_time_updates being marked as SKIPPED, unless the trip was a NEW or DUPLICATED trip and was subsequently canceled.

But this doesn't provide enough guidance. Should a NEW or DUPLICATED trip that is canceled after it is first published maintain schedule_relationship of NEW or DUPLICATED with all stops having schedule_relationship = SKIPPED and scheduled arrival and departure times must be defined?

In this case, yes, because otherwise it is impossible to know when and where the cancelled trip runs.

What about using schedule_relationship = CANCELED but still publishing StopTimeUpdates with scheduled arrival and departure times and schedule_relationship = SKIPPED? This would allow consumers to more easily conclude that the entire trip is canceled without having to check that all stops on the trip are SKIPPED, and it would still maintain the scheduled times.

I think this needs to be formalized in the specification, but that could happen in the future (at a minimum, before removing the experimental tags). I don't think that needs to happen now and doesn't need to hold up this vote.

[sam-hickey-arcadis]

@sam-hickey-arcadis The CLA check now fails because I have applied some of your review suggestions.

I'm working on this.

[sam-hickey-arcadis]

+1 Arcadis

[bdferris-v2]

+1 Google

I will say that I share some of the same concerns raised by @SteveGladstone and @sam-hickey-arcadis about REPLACEMENT but since the field is still marked as experimental, I guess that's a discussion for another day. In particular, reconciling TripModifications and REPLACEMENT is something to keep an eye on sooner rather than later as these two experimental fields start to gain adoption.

[SteveGladstone]

+1 Maryland Transit Administration. Looking forward to leveraging NEW trip functionality 😄

[jfabi]

+1 from the @mbta—as heavy users of ADDED trips, especially on our subway system, we're very excited for this new standardization of added and modified service, as well as for potential applications such as defining bus bridges.

[miklcct]

The voting is now closed. There are 9 votes supporting this proposal and 1 abstention. Therefore the proposal is passed.

There are a few grammatic changes needed as mentioned by @bdferris-v2 before merging. I'll handle them tomorrow.

[miklcct]

The voting is now closed. There are 9 votes supporting this proposal and 1 abstention. Therefore the proposal is passed.

There are a few grammatic changes needed as mentioned by @bdferris-v2 before merging. I'll handle them tomorrow.

The editorial changes have been done, waiting for @sam-hickey-arcadis to complete his CLA such that this PR can be merged.

[miklcct]

The voting is now closed. There are 9 votes supporting this proposal and 1 abstention. Therefore the proposal is passed.
There are a few grammatic changes needed as mentioned by @bdferris-v2 before merging. I'll handle them tomorrow.

The editorial changes have been done, waiting for @sam-hickey-arcadis to complete his CLA such that this PR can be merged.

@sam-hickey-arcadis the CLA is still hasn't done yet

[sam-hickey-arcadis]

The voting is now closed. There are 9 votes supporting this proposal and 1 abstention. Therefore the proposal is passed.
There are a few grammatic changes needed as mentioned by @bdferris-v2 before merging. I'll handle them tomorrow.

The editorial changes have been done, waiting for @sam-hickey-arcadis to complete his CLA such that this PR can be merged.

@sam-hickey-arcadis the CLA is still hasn't done yet

I am working on this. Will have it complete ASAP.

[sam-hickey-arcadis]

We submitted a corporate CLA to Google two days ago but have not received the DocuSign that they say is the next step. As soon as we receive that, we will sign it and the CLA check here should then succeed.

[dan-engler-arcadis]

We submitted a corporate CLA to Google two days ago but have not received the DocuSign that they say is the next step. As soon as we receive that, we will sign it and the CLA check here should then succeed.

Arcadis finally received the DocuSign this morning. It has been signed and the CLA rescan completed.

[miklcct]

We submitted a corporate CLA to Google two days ago but have not received the DocuSign that they say is the next step. As soon as we receive that, we will sign it and the CLA check here should then succeed.

Arcadis finally received the DocuSign this morning. It has been signed and the CLA rescan completed.

OK, this should be ready to be merged.