Repository navigation
Firestore: Support for real-time updates #1618
Description
Activity
- changed the title
[-]Firestore: Support real-time updates [/-][+]Firestore: Support for real-time updates [/+]on Nov 4, 2017 I don't know - I believe the expectation is that that's more of a mobile concern, whereas the current API is expected to be used server-side.
Thank you very much for your reply. I get your point, but there are some use cases where real-time support would be extremely useful, even on the server-side. For example, when:
- Integrating with custom server applications (in scenarios where Cloud Functions can't handle the business logic)
- Integrating with web services such as Asp.Net and Asp.Net Core
- Implementing client libraries for platforms which are not currently officially supported by Google, such as Xamarin, WPF, UWP and Windows Forms
Without real-time support, the API seems incomplete, especially because Firestore's RPC documentation does mention support for real-time updates through the Listen function of the service interface.
I'll pass your comment back to the team - I'm only doing the C# API for it, following the direction given by the main Firestore team.
Thank you very much. I appreciate your help.
Hi, Sebastian here from the Firestore client team. We are currently offering realtime support in the mobile clients for the Web, iOS, Android. We also launched the Node Server SDK with realtime support.
Since we are currently offering Firestore Server SDKs for Node, Go, Python & Java (with C# and others in the works), we first wanted to gauge what the demand for realtime support for these server languages was. Our users have since indicated that they do want to see realtime support in all of these languages, and hence we are treating this as a high priority item on our immediate roadmap. While I can't provide a timeline, expect to see realtime support across all of our SDKs in the not so distant future.
Reacted by Paul Roque, Joe Booth, Luke, Andrey Lastochkin, David Dai, Peter Stojanowski, Nik Ammerlaan and Gabriel HandfordGreat news! Thank you very much for your feedback.
Will leave this issue open as a placeholder to indicate that we're intending to do it. I'll mark it as P2+, but that just means it's not imminent (i.e. working to get it in the next release). It doesn't mean it's not important :)
Reacted by Paul Roque, Joe Booth and Min ZhaoNearly end of the year and still no support for C# while all the admin SDK got update last week
And not even a roadmap was solid
No support for this specific API call yet, no. We'll get there, but we're doing lots of other work too.
From a server side perspective this would be very useful for integrating data collected by mobile devices into line of business systems.
I take it the heavy lifting for this is in the protocol, and there does not exist yet anything in Google.Cloud.Firestore.V1Beta1 to support this? If support is in V1Beta1 it should be fairly trivial to add an
onSnapshotmethod toQuerySnapshot.It's far from trivial to provide the kind of support we want to provide. We would rather take time and do it properly than rush a shoddy option. You can use the client property to get the raw gRPC if you want in the meantime. We will attend this feature request over time, but will resist calls to rush.
21 remaining items
@Thaina: I've just pushed my prototype branch if you want to have a look: https://github.com/jskeet/google-cloud-dotnet/tree/firestore-watch
I believe it mostly works, but the public API is likely to change, particularly in terms of how a watch is stopped. I'm gradually merging bits of it into master - hence the recent PRs that don't seem to do anything useful, because they're building up the support required for the main event. So yes, work has been slow (as I've had other features to support in other APIs) but it's coming along.
Reacted by Thaina, Will Hart and Andreas Gullberg LarsenNow merged :)
I'll hold off doing a release until the Firestore team have had a look, but then I'll happily do another nuget package.
This is now released in Google.Cloud.Firestore version 1.0.0-beta04.
Reacted by Thaina, Daniel Zuidinga, Johan Magnusson, Will Hart, Matthew Swaidan and Dave BrooksI've just found a bug for document (not query) watch - it was an easy fix, and I've released 1.0.0-beta05. Apologies for the inconvenience. Separately a) there's no documentation for this feature yet; b) our API doc release process is temporarily broken. I'm hoping to fix both of those soon, but I'm at conferences for a week or so. I'll see how much I can do between talks...
Reacted by ThainaHi jskeet
I know that this has been closed and you mentioned there will be no docs, but I am trying to get this working in .net core and not sure why it is not firing when I change docs on collection
`
CollectionReference collection = db.Collection("users");
Query query = collection.WhereEqualTo("dealerId", 34);query.Listen(querySnapshot => { string firstName = ""; string lastName = ""; foreach (DocumentSnapshot queryResult in querySnapshot.Documents) { firstName = queryResult.GetValue<string>("displayName"); lastName = queryResult.GetValue<string>("phoneNumber"); } });`
Is my code Ok?
Thanks
NickThere are docs that are merged, just not published yet - see https://github.com/GoogleCloudPlatform/google-cloud-dotnet/blob/master/apis/Google.Cloud.Firestore/docs/userguide.md and https://github.com/GoogleCloudPlatform/google-cloud-dotnet/blob/master/apis/Google.Cloud.Firestore/Google.Cloud.Firestore.Snippets/UserGuideSnippets.cs
I'd expect your listener to fire if there's a dealer with ID 34, yes. I can't test it right now, but I should be able to tomorrow. Currently your listener doesn't do anything with the results - are you just checking whether or not it fires using breakpoints? If you can provide a small but complete example (e.g. as a console app) that would be really helpful. Please file it as a new issue as this is around the feature not working rather than not being provided.
Ok looks like I get this working. Another question, now when we create a listener the first time, it loads the whole collection and after fires when there are some changes. Is there any way to skip the whole loading he first time and just start to listen for changes?
The recommended way to do this is to manually track the update time in a field in your Firestore documents. You can then use this field in a query filter when you initialize your snapshot listener.
@schmidt-sebastian And that would break all client update and transaction in every place related to that collection. We then need to rebuild our client just to have put another field that already exist and managed by database itself
I was file the request to firestore to query with updateTime for half year and it still have no progress at all. This is also another use case that should put this feature to more priority
@Thaina Thanks for the feedback. Unfortunately querying by updateTime is not something we can't automatically support because it would require us to maintain a database index sorted by updateTime and that would impose a bottleneck of 500 writes/sec for the collection (See "Maximum write rate to a collection in which documents contain sequential values in an indexed field" from https://firebase.google.com/docs/firestore/quotas). While this might be okay for some users for some collections, we definitely do not want to impose this limit for all users for all collections. So for now our recommended solution is to maintain your own indexed field if this is a restriction that works for your use case.
Thanks again for the feedback. Perhaps in the future we could somehow let you opt-into indexing on the internal updateTime on a per-collection basis (with some warnings about the resulting bottleneck). But for now the recommended approach is to maintain your own field.
@mikelehen I don't understand why you maintain index of updateTime field would be slower than us developer need to put another field in every update
And if we really did that, I mean if we have to put our own
CustomUpdateTime = ServerValue.TimeStampeverywhere we update document, will it impose a bottleneck of 500 writes/sec for the collection ?If it not then maybe this problem could be solved by your server could just inject hidden
__updateTime__and__createTime__field in every document that get update. It could even have an option for each collection to enable this injection, maybe disabled by defaultThis feature actually have so many benefit. Especially to monitoring the format and validate document changed on the live application. I put many reason when I file this request and it really is crucial
And the last but not least
updateTime might cause bottleneck. But I don't see why it would have a problem to expose at least createTime for query? In fact I think createTime should be base order of collection instead of__name__ps. Sorry so have ask these thing on this repo that not directly relate to this problem. But firestore and firebase service don't have its own repo to use for public discussion like this
Hi @Thaina, you're welcome to ask questions like this on our google group here: https://groups.google.com/forum/#!forum/google-cloud-firestore-discuss
And unfortunately, yes, if you put your own
CustomUpdateTime = ServerValue.TimeStampeverywhere you update documents, that will impose a bottleneck of 500 writes/sec for the collection. And this is the reason we don't automatically do this. Same for createTime. If we included it and indexed it, then that would impose a bottleneck of 500 creates/sec.For high-usage collections you generally don't want to index any sequential field (like a timestamp) because it will cause a "hotspot" (a section of the index that every write needs to access). Often times you can solve this with a composite index. Rather than just index updateTime, you could index user + updateTime or similar. Then you'd have a 500 writes/sec/user bottleneck instead of 500 writes/sec for the entire collection.
Reacted by Thaina and Juan LReacted by Thaina
Is there any roadmap to add support for real-time updates?