Add Firestore indexes to local emulator

[nicholaschiang]

[REQUIRED] Environment info

firebase-tools:
7.15.0

Platform:
Ubuntu 18.04.4

[REQUIRED] Test case

I'm trying to test our Firestore indexes, but they seem to always be working (regardless of the indexing settings included in firestore.indexes.json). Below is the test case in question:

const FILTERS = {
    'grade': ['==', TUTOR.grade],
    'gender': ['==', TUTOR.gender],
    'location': ['==', TUTOR.location],
    'type': ['==', TUTOR.type],
    'payments.type': ['==', TUTOR.payments.type],
    'config.showProfile': ['==', true],
};
const SORTERS = [
    'avgRating',
    'numRatings',
];

describe('Tutorbook\'s Database Indexing', () => {
    const db = authedApp({
        uid: PUPIL.uid,
        email: PUPIL.email,
        access: PUPIL.access,
    });
    const filterCombos = combinations(Object.keys(FILTERS));
    return Promise.all(filterCombos.map(filters => it('lets users filter ' +
        'profiles by ' + filters.join(', '), () => {
            var query = db.collection('users')
                .where('access', 'array-contains-any', PUPIL.access);
            filters.map(filter => query =
                query.where(filter, FILTERS[filter][0], FILTERS[filter][1]));
            return Promise.all(SORTERS.map(sorter =>
                firebase.assertSucceeds(query.orderBy(sorter).get())));
        })));
});

[REQUIRED] Steps to reproduce

Just try testing any type of composite or collection group query that would (normally) require an index. It will work on the emulator suite regardless of the existence of the required index (but it won't work in production without the index).

[REQUIRED] Expected behavior

These tests should fail unless there are Firestore indexes (specified in firestore.indexes.json) that support the composite indexes tested.

[REQUIRED] Actual behavior

All the tests succeed (even though they shouldn't).

[nicholaschiang]

Just FYI the ultimate goal here is to test each possible combination of the different filters (as defined in FILTERS and SORTERS) available in our app's search screen.

[samtstern]

@nicholaschiang thanks for the feedback! Right now you're correct that the Firestore emulator will execute any query even if the index has not been "created".

@yuchenshi do you know if we will be able to support stricter indexing in the Firestore emulator?

[yuchenshi]

Strict indexing in the Firestore Emulator is not a planned feature (yet) but I'll definite discuss this with the team.

On the other hand, we do plan to support dumping all indexes automatically created (say during the test), which you can compare against firestore.indexes.json. I believe this will address OP's use case as well.

[nicholaschiang]

@yuchenshi That would be a useful (although not exactly what I'm asking for) feature; those "tests" would ensure that my firestore.indexes.json supported the "tested" queries. Just make sure that feature accounts for Firestore's ability to merge indexes and doesn't just create a composite index for each query type.

[nicholaschiang]

Oh, and @samtstern you should have this specification (that the Firestore emulator will execute any query even if there isn't an index to support it) added to the differences between emulator and production webpage.

[manwithsteelnerves]

Its unfortunate that firestore emulator doesn't support index rules.
@yuchenshi I see it useful feature considering the merge will be handled. Thats like a master piece!

What are the chances to consider this feature?

[yuchenshi]

We're considering both features of dumping and tracking index rules and we'd appreciate input on the use cases.

As a developer, how do you plan on using emulators to prototype, test, and gain more confidence about your index rules? Shoot us some workflows just like what @nicholaschiang did and we'll figure out the best way to make that happen.

[manwithsteelnerves]

Great!
Our use-case is pretty similar to the OP mentioned.
We have a REST Api's which has optional query parameters. To check if each request has required indexes already in place, we use jest framework to fire the requests with all possible combinations. If it throws errors that indexes are missing and we can create the required indexes.

So for 4 optional query params with multiple values allowed in each param, we need to test around more than 200 possibilities. And creating indexes for multiple combinations manually is error prone.

If there is a provision for firestore emulator to create the required indexes list, we can just run the all possible cases generated through code and use the indexes created from firestore emulator. This in-fact saves lot of time.

[manwithsteelnerves]

@yuchenshi Can you please let us know if anything planned related to this feature?

[sk-]

@yuchenshi we just got bitten by this.

We deployed a quick fix, which actually broke later in production due to the missing index. Note that that part of the code was tested, so it'd be great if the emulator had the same behavior as production.

[yuchenshi]

@manwithsteelnerves While we don't share timelines and cannot commit to any plans, but I do want to let you know that we hear you and real pain points (including the latest one shared by @sk- ) helps a lot driving our internal discussion. It's a technically challenging and involved feature for sure to get the index semantics match production, but that doesn't mean we're not considering / doing it.

[manwithsteelnerves]

As I see it may take more time, I ended up creating 139 manual indexes for my project for all possible test cases. Hopefully, there will be a better solution through emulator workflow!

[jpangburn]

Out of the two options discussed "match production" and "dumping all indexes automatically created [during some set of tests]" , if we could only have one then I prefer the second one. For me, the point of writing a bunch of automated tests is that so later when I add some feature or fix some bug, I don't break a bunch of other things. With that in mind, if the emulator could dump the list of indexes that I need that aren't already in my firestore.indexes.json- that seems pretty helpful.

If I ran a test set and nothing came out then I'd know that some tests I added (to verify some bug) weren't increasing the indexes I need. Or if I prototype some feature, it would be nice if the emulator made it clear "hey buddy, you're going to need another 100 indexes to support this" by dumping a list of those hundred indexes and I can realize I had a stupid idea before getting too far into it.

Matching production by giving me an error is kinda useful, but just giving the list of additional indexes needed is going to help me make progress faster I think. And @nicholaschiang 's comment is critical "Just make sure that feature accounts for Firestore's ability to merge indexes and doesn't just create a composite index for each query type." Might even be nice if it also said "hey, you've got this set of indexes [X,Y,Z] in your firestore.indexes.json that were not necessary during this test run".