(Firestore) Grouped HTTP requests while in offline mode

[tormjens]

[REQUIRED] Describe your environment

• Operating System version: macOS 11.6
• Browser version: Firefox Developer Edition 97.0b9 (64-bit)
• Firebase SDK version: ^8.3.0
• Firebase Product: firestore

[REQUIRED] Describe the problem

We have an app where we are sending data to FIrestore every 15 seconds (on an interval). This works fine, when online. When offline the requests are queued, which is expected. If there are <=10 requests they are grouped together in one HTTP POST request. However, if there are more than 10 requests queued only the first 10 requests are grouped in one HTTP requests. The subsequent requests are fired one by one. This means if you've been offline for a while, and then go back online, a whole lot of HTTP requests are fired. I would expect that these would be grouped like the first 10, which would drastically reduce the number of HTTP requests to be sent. Is this something that can be configured? If not, why are not subsequent requests grouped?

Steps to reproduce:

• Put browser in offline mode
• Set interval for 15 seconds
• Send an update to Firestore in that interval
• Put browser in online mode

Relevant Code:

import firebase from 'firebase/app';
import 'firebase/firestore';
import firebaseConfig from './firebase-config.json';

const firebaseApp = firebase
    .initializeApp(firebaseConfig);

const db = firebaseApp.firestore();

db.enablePersistence();

const document = db.doc('some/path');

setInterval(() => {
        document.update({
            content: JSON.stringify(subsession),
        });
}, 15000);

[dconeybe]

Thanks for reporting this. I've been able to reproduce. It looks like the first 10 mutations get grouped into a single batch, and all mutations thereafter happen in their own individual batches. You're right that those subsequent mutations should be batched into groups of 10 as well. I'll investigate and report back.

Here is the code that I used to reproduce:

import { initializeApp } from 'firebase/app'
import { disableNetwork, doc, enableIndexedDbPersistence, enableNetwork, getFirestore, setLogLevel, updateDoc } from 'firebase/firestore';

const firebaseConfig = { /*REDACTED*/ };
const firebaseApp = initializeApp(firebaseConfig);
const db = getFirestore(firebaseApp);
setLogLevel('debug');

const document = doc(db, 'JsIssue5971/TestDoc');

console.log('disableNetwork()');
disableNetwork(db);

for (let i=0; i<50; i++) {
  console.log('Updating value to ' + i);
  updateDoc(document, { content: { 'value': i } });
}

console.log('enableNetwork()');
enableNetwork(db);

The call to enableNetwork() starts all of the network communication. Here are the logs.

[dconeybe]

After investigating, my suspicion is that what you are seeing as apparent batching of the first 10 writes in a single HTTP POST request, and the rest of the writes in individual HTTP POST requests, is actually just a side effect of the hardcoded limit of 10 outstanding writes:

firebase-js-sdk/packages/firestore/src/remote/remote_store.ts

Line 62 in bf93a38

const MAX_PENDING_WRITES = 10;

Here is what is happening:

1. Network connectivity is restored.
2. The first 10 enqueued writes are sent to the server.
3. The server acknowledges the 1st write that was sent.
4. Now there are only 9 outstanding writes.
5. To fill the write pipeline back to 10, the next write is sent to the server.
6. The server acknowledges the 2nd write that was sent.
7. To fill the write pipeline back to 10, the next write is sent to the server.
8. and so on.

In the logs that I attached, you can see that the first 10 writes appear to be batched, followed by an individual write, followed by an individual write acknowledgement, followed by an individual write, ... , followed by 10 write acknowledgements. So there are still 50 individual writes sent to the server and 50 individual write acknowledgements.

Now all of this should be done over one continuous HTTP connection to the server. There should not be HTTP requests starting and stopping. So my question is, how did you observe these HTTP POST requests? Could you provide a log or a screenshot or a video or something that lead you to the conclusion that HTTP POST requests are being repeatedly created?

[tormjens]

Thank you for investigating further.

My project is running in the browser, and I'm monitoring network requests via the Network tab in Firefox.

My issue above was running v8 of the Web SDK, so I thought I might give v9 a try.

I modified your example by adding a timeout before network is enabled again.

Still there are numerous HTTP requests being dispatched.

Am i missing a core concept here? Is it supposed to be this way?

Here is the log from the console.

Here is a video showing requests being fired.

The code used for this:

import { initializeApp } from '@firebase/app'
import { disableNetwork, doc, enableIndexedDbPersistence, enableNetwork, getFirestore, setLogLevel, updateDoc } from '@firebase/firestore';

const firebaseConfig = {
// REDACTED
};

const firebaseApp = initializeApp(firebaseConfig);
const db = getFirestore(firebaseApp);
setLogLevel('debug');

const document = doc(db, 'JsIssue5971/TestDoc');

console.log('disableNetwork()');
disableNetwork(db);

for (let i=0; i<50; i++) {
  console.log('Updating value to ' + i);
  updateDoc(document, { content: { 'value': i } });
}

console.log('enableNetwork()');
setTimeout(() => {
    enableNetwork(db);
}, 1000);

[dconeybe]

Okay I see what you mean. I'm digging in more. There may be a bug here.

[schmidt-sebastian]

Filed an internal bug: b/218708019

This needs to be investigated by our WebChannel team. You should not see as many requests as you are seeing.

[piperone]

@schmidt-sebastian Is there any way we can follow the progress on this?

[schmidt-sebastian]

We are handling this internally. We do not need additional data since we are able to reproduce this.