Using a filtered query is generating an empty filter array

[nicholasjon]

I'm trying to build a filtered query (that I have working when just issuing curls against my elasticsearch cluster) using the elastic library, but I can't seem to get my "and" filter to be respected.

I searched the docs and issues and while I found a question about range / terms filters (#105) that sort of discussed using a filtered query I wasn't able to find specific examples for doing so.

Here's my elasticsearch query:

"query":{
  "filtered":{
    "query":{
      "match_all":{}
    },
    "filter":{
      "and":[
        {
          "terms":{
            "experience_id":[
              1,
              2,
              3,
              4,
              5,
              6,
              7,
              8,
              9,
              10
            ]
          }
        },
        {
          "terms":{
            "utc_unix_timestamp":[
              "1438270392.12345",
              "1438112113.12345",
              "1438111586.12345",
              "1438111528.12345",
              "1438111436.12345",
              "1438111203.12345",
              "1438028827.12345",
              "1437583656.12345",
              "1437583610.12345",
              "1437583526.12345"
            ]
          }
        }
      ]
    }
  }
}

... and I'm attempting to recreate it thusly (based on code from @olivere found in the above linked issue)

  matchAllQuery := elastic.NewMatchAllQuery()
  termFilter_Timestamps := elastic.NewTermsFilter("utc_unix_timestamp", utc_unix_timestamps)
  termFilter_ExperienceIds := elastic.NewTermsFilter("experience_id", experience_ids)
  andFilter := elastic.NewAndFilter()
  andFilter.Add(termFilter_Timestamps)
  andFilter.Add(termFilter_ExperienceIds)
  filteredQuery := elastic.NewFilteredQuery(matchAllQuery).Filter(andFilter)

(The variables utc_unix_timestamps and experience_ids contain slices populated just as in my sample query above.)

... but I've been getting a result set that has all records from my dataset in it. So I investigated with:

  ss := elastic.NewSearchSource().Query(&filteredQuery)
  src := ss.Source()
  out, err := json.MarshalIndent(src, "", "  ")
  if err != nil {
      log.Fatal(err)
  }
  log.Printf("%s", string(out))

... and discovered that the source looks like:

{
  "query": {
    "filtered": {
      "filter": {
        "and": {
          "filters": []
        }
      },
      "query": {
        "match_all": {}
      }
    }
  }
}

I am a newb in both Go and using this library, so I apologize in advance if I missed some obvious documentation. I'm curious if anyone can help me understand how to get my filter respected by the query.

[olivere]

Hi there! You tried your best, so no need to apologize. :-)

I think there is a simple solution. Can you replace

andFilter.Add(termFilter_Timestamps)

with

andFilter = andFilter.Add(termFilter_Timestamps)

The reason why you need to use it as a chain is that AndFilter is a value type, not a pointer type. And you know what: That's because I didn't know Go too well either at the time I've written most filters. :-) In Elastic 3.0, which targets Elasticsearch 2.0, your code would work. I cannot change this in Elastic 2.0 because it would be a breaking change, I'm afraid. Sorry for that.

Cheers.

[nicholasjon]

Of course! I should have seen that, it was even in the example you posted for issue #105. Thanks so much for the response, I really appreciate it.

If I could dig in, now that everything is being added to its respective filter object properly, I happened to notice that the generated output now looks like:

{
  "query": {
    "filtered": {
      "filter": {
        "and": {
          "filters": [
            {
              "terms": {
                "utc_unix_timestamp": [
                  [
                    "1438879373.12345",
                    "1438879170.12345",
                    "1438876996.12345",
                    "1438788098.12345",
                    "1438709041.12345",
                    "1438031741.12345",
                    "1438030950.12345",
                    "1438030887.12345",
                    "1438030744.12345",
                    "1435958415.12345"
                  ]
                ]
              }
            },
            {
              "terms": {
                "experience_id": [
                  [
                    1,
                    2,
                    3,
                    4,
                    5,
                    6,
                    7,
                    8,
                    9,
                    10
                  ]
                ]
              }
            }
          ]
        }
      },
      "query": {
        "match_all": {}
      }
    }
  }
}

... which of course fails because there's not supposed to be a filters element after the word and. An and filter feels like the right choice to use here, but is there another construct I should be using to build this filter?

(I realize this bends the original issue, and I'd be happy to take my question to a different spot if you'd prefer.)

Thanks.

[olivere]

Sorry, I'm on the road... you could use a BoolFilter with "must" clauses instead. Please try to find the related unit test for BoolFilter so see a usage example.

[nicholasjon]

Hey, thanks again for the feedback. I ended up figuring out the issue (mine, of course) and got everything working.

Just in case anyone digging around for something similar happens to find this, here's where I got tripped up. As expected, it has to do with me fumbling through some of the interesting parts of Go.

In the code that I originally posted, I had these two lines:

termFilter_Timestamps := elastic.NewTermsFilter("utc_unix_timestamp", utc_unix_timestamps)
termFilter_ExperienceIds := elastic.NewTermsFilter("experience_id", experience_ids)

... however the definition of the NewTermsFilter looks like:

func NewTermsFilter(name string, values ...interface{}) TermsFilter {
    f := TermsFilter{
        name:   name,
        values: make([]interface{}, 0),
    }
    f.values = append(f.values, values...)
    return f
}

... which you'll note takes a variadic values arguments and not a slice values argument -- I learned a bit more about that here.

When I changed my code to:

  experience_ids_as_interfaces := make([]interface{}, len(experience_ids))
  for i, v := range experience_ids {
    experience_ids_as_interfaces[i] = v
  }
  utc_unix_timestamps_as_interfaces := make([]interface{}, len(utc_unix_timestamps))
  for i, v := range utc_unix_timestamps {
    utc_unix_timestamps_as_interfaces[i] = v
  }
  termFilter_Timestamps := elastic.NewTermsFilter("utc_unix_timestamp", utc_unix_timestamps_as_interfaces...)
  termFilter_ExperienceIds := elastic.NewTermsFilter("experience_id", experience_ids_as_interfaces...)

... all became right with the world (it compiled), and my query began returning the expected results.

The final code that I ended up using looks like:

  experience_ids_as_interfaces := make([]interface{}, len(experience_ids))
  for i, v := range experience_ids {
    experience_ids_as_interfaces[i] = v
  }
  utc_unix_timestamps_as_interfaces := make([]interface{}, len(utc_unix_timestamps))
  for i, v := range utc_unix_timestamps {
    utc_unix_timestamps_as_interfaces[i] = v
  }
  matchAllQuery := elastic.NewMatchAllQuery()
  termFilter_Timestamps := elastic.NewTermsFilter("utc_unix_timestamp", utc_unix_timestamps_as_interfaces...)
  termFilter_ExperienceIds := elastic.NewTermsFilter("experience_id", experience_ids_as_interfaces...)
  andFilter := elastic.NewAndFilter()
  andFilter = andFilter.Add(termFilter_Timestamps)
  andFilter = andFilter.Add(termFilter_ExperienceIds)
  andFilter = andFilter.Cache(true)
  filteredQuery := elastic.NewFilteredQuery(matchAllQuery).Filter(andFilter)

I still need to figure out if that's actually a performant way for me to have solved the problem, but that's a totally different issue for a totally different forum. :)

Thanks again for the assist, it put me on the right path.