added advanced search query for elasticsearch #2617

AmasiaNalbandian · 2021-12-15T23:17:23Z

Issue This PR Addresses

This PR closes #2110

Type of Change

Bugfix: Change which fixes an issue
New Feature: Change which adds functionality
Documentation Update: Change which improves documentation
UI: Change which improves UI

Description

This PR Is aimed to help make search queries more flexible. What I have done here is I looked into Elasticsearch so that we can further query different fields we have indexed in Posts. I looked at the data presented back by Elasticsearch and realized we have more fields we can use to search:

post
title
author
date published

This PR allows for us to query all of this on the backend, with search and Elasticsearch.

Additionally, this query seems to be more promising than the original query for the post/author search. We should eventually use the query provided here, as the zero_terms_query: 'all', means that if that field is not provided, to search without it. This means we can use this query to search for one, two, three... or all fields, acting the same way as the previous implementation but doing more. The query will also work if we want to search for all posts published on X date, or from x to y dates.

Steps to test:

delete redis-data file if it exists
cp config/env.development .env
docker-compose --env-file ./config/env.development up --build search elasticsearch redis posts
to rebuild search: docker-compose --env-file ./config/env.development up --build -d search posts (NOT Necessary - Skip! just FYI)
in another terminal run pnpm start to start the feed parser. Wait a bit for some posts to populate.
As the changes are only reflected in the API, you can test queries using such:
http://localhost/v1/search/advanced/?post=elasticsearch&author=amasia
feel free to add more filters such as author, title, from, to and post

After you are done ensure your containers are down by running docker-compose --env-file ./config/env.development down

Checklist

Quality: This PR builds and passes our npm test and works locally
Tests: This PR includes thorough tests or an explanation of why it does not
Screenshots: This PR includes screenshots or GIFs of the changes made or an explanation of why it does not (if applicable)
Documentation: This PR includes updated/added documentation to user exposed functionality or configuration variables are added/changed or an explanation of why it does not(if applicable)

Example of how the search works:

The screenshot is taken from elasticsearch on localhost:9200 for reference of the post:

gitpod-io · 2021-12-15T23:17:25Z

humphd · 2021-12-15T23:18:22Z

This needs a proper description of what it is, what it changes, how to test it, etc.

AmasiaNalbandian · 2021-12-15T23:20:05Z

This needs a proper description of what it is, what it changes, how to test it, etc.

This was not meant to be compared to Seneca Master... I am still cherry picking commits - sorry about this.

humphd

This needs tests.

src/api/search/src/bin/validation.js

src/api/search/src/routes/query.js

src/api/search/src/search.js

src/api/search/src/bin/validation.js

AmasiaNalbandian · 2021-12-16T02:28:31Z

This needs tests.

Where can I find the tests to write for this? I know you've mentioned it a bunch of times... do we want to just test the routes?

humphd · 2021-12-16T14:26:00Z

They should be in https://github.com/Seneca-CDOT/telescope/tree/master/src/api/search, but apparently we have no tests for this service? Please file an issue for us to add them, since they don't exist, and we'll do it all at once.

AmasiaNalbandian · 2021-12-16T14:54:45Z

They should be in https://github.com/Seneca-CDOT/telescope/tree/master/src/api/search, but apparently we have no tests for this service? Please file an issue for us to add them, since they don't exist, and we'll do it all at once.

I've filed this: #2621

humphd · 2021-12-16T14:57:36Z

OK, so let's get my other comments addressed in updates and try to land this ASAP.

src/api/search/src/search.js

AmasiaNalbandian · 2021-12-16T17:18:43Z

Queries after requested changes

humphd

Getting close, a few more things

humphd · 2021-12-16T17:25:10Z

src/api/search/src/search.js

+    },
+  };
+
+  const must = query.query.bool.must;


const { must } = query.query.bool;

humphd · 2021-12-16T17:25:26Z

src/api/search/src/search.js

+ * Match field queries: https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-ppmatch-query.html#query-dsl-match-query-zero
+ */
+const advancedSearch = async (options) => {
+  const query = {


Lets' name this result so we don't have query.query

src/api/search/src/search.js

humphd · 2022-01-20T01:14:19Z

Let us know when this is ready for review.

added advanced route with params separated query validation based on route pathname cleaned up the files added validation cleaned up validator added oneof in declarations fixed validation changed query fields and added if statements for options passed changed description for advanced search fixed date format to correct format

AmasiaNalbandian · 2022-01-20T02:49:26Z

it is ready now! Thank you @JerryHue!

AmasiaNalbandian · 2022-01-20T02:50:04Z

I updated all the instructions if you want to build and try it out. :) But also for my own memory...

JerryHue

Where can I find the build instructions?

AmasiaNalbandian · 2022-01-20T14:05:43Z

Where can I find the build instructions?

It's in the PR Description. @JerryHue

HyperTHD

Tested, works good. My only concern is the lack of tests but the search service doesn't have any tests at all. Might want to file a follow-up to get those added in unless that's been filed already

AmasiaNalbandian · 2022-01-20T14:34:57Z

Tested, works good. My only concern is the lack of tests but the search service doesn't have any tests at all. Might want to file a follow-up to get those added in unless that's been filed already

It's been filed and assigned! :)

HyperTHD · 2022-01-20T14:36:36Z

Just noticed #2621 so once you get another approval, you can rebase and merge this in @AmasiaNalbandian

vercel bot deployed to Preview December 15, 2021 23:17 View deployment

vercel bot deployed to Preview December 15, 2021 23:20 View deployment

vercel bot deployed to Preview December 15, 2021 23:41 View deployment

vercel bot deployed to Preview December 15, 2021 23:50 View deployment

vercel bot deployed to Preview December 15, 2021 23:52 View deployment

humphd requested changes Dec 16, 2021

View reviewed changes

src/api/search/src/bin/validation.js Show resolved Hide resolved

src/api/search/src/bin/validation.js Outdated Show resolved Hide resolved

src/api/search/src/routes/query.js Show resolved Hide resolved

src/api/search/src/search.js Outdated Show resolved Hide resolved

vercel bot deployed to Preview December 16, 2021 01:21 View deployment

AmasiaNalbandian force-pushed the issue-2110-backend branch from f24b390 to 3925230 Compare December 16, 2021 01:21

vercel bot deployed to Preview December 16, 2021 01:22 View deployment

vercel bot deployed to Preview December 16, 2021 01:32 View deployment

AmasiaNalbandian mentioned this pull request Dec 16, 2021

Refactor the search to use the advanced search route #2618

Closed

humphd requested changes Dec 16, 2021

View reviewed changes

src/api/search/src/bin/validation.js Outdated Show resolved Hide resolved

AmasiaNalbandian self-assigned this Dec 16, 2021

vercel bot deployed to Preview December 16, 2021 17:15 View deployment

AmasiaNalbandian force-pushed the issue-2110-backend branch from e4e4f25 to ad0209e Compare December 16, 2021 17:16

vercel bot deployed to Preview December 16, 2021 17:16 View deployment

AmasiaNalbandian commented Dec 16, 2021

View reviewed changes

src/api/search/src/search.js Show resolved Hide resolved

humphd requested changes Dec 16, 2021

View reviewed changes

vercel bot deployed to Preview December 16, 2021 17:41 View deployment

AmasiaNalbandian force-pushed the issue-2110-backend branch from 5d6fd1f to e772fb6 Compare December 16, 2021 17:42

vercel bot deployed to Preview December 16, 2021 17:43 View deployment

AmasiaNalbandian force-pushed the issue-2110-backend branch from e772fb6 to 6c9c50b Compare December 16, 2021 17:56

AmasiaNalbandian force-pushed the issue-2110-backend branch from 6561ae8 to 41cf077 Compare January 14, 2022 20:20

vercel bot deployed to Preview January 14, 2022 20:21 View deployment

AmasiaNalbandian added the area: elasticsearch Elasticsearch Related issues and pull requests label Jan 14, 2022

humphd requested a review from JerryHue January 15, 2022 00:36

AmasiaNalbandian force-pushed the issue-2110-backend branch from 41cf077 to 30a0d26 Compare January 20, 2022 01:12

vercel bot deployed to Preview January 20, 2022 01:12 View deployment

vercel bot deployed to Preview January 20, 2022 02:43 View deployment

AmasiaNalbandian force-pushed the issue-2110-backend branch from e51118f to ff02418 Compare January 20, 2022 02:48

vercel bot deployed to Preview January 20, 2022 02:48 View deployment

TDDR requested review from menghif and HyperTHD January 20, 2022 13:15

JerryHue reviewed Jan 20, 2022

View reviewed changes

HyperTHD approved these changes Jan 20, 2022

View reviewed changes

humphd approved these changes Jan 20, 2022

View reviewed changes

JerryHue self-requested a review January 20, 2022 14:49

JerryHue approved these changes Jan 20, 2022

View reviewed changes

AmasiaNalbandian merged commit 690f589 into Seneca-CDOT:master Jan 20, 2022

AmasiaNalbandian deleted the issue-2110-backend branch January 20, 2022 15:50

AmasiaNalbandian mentioned this pull request Jan 20, 2022

ElasticSearch Advanced Route is breaking Search in Production #2698

Closed

This was referenced Feb 1, 2022

SearchBar (new PR) #2183

Closed

Advanced search #2199

Closed

This was referenced Mar 10, 2022

Fixes 2618 - Refactored search to use advanced route #3163

Closed

Fixes-2618: Refactor search routes #3164

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

added advanced search query for elasticsearch #2617

added advanced search query for elasticsearch #2617

AmasiaNalbandian commented Dec 15, 2021 •

edited by HyperTHD

Loading

gitpod-io bot commented Dec 15, 2021

humphd commented Dec 15, 2021

AmasiaNalbandian commented Dec 15, 2021

humphd left a comment

AmasiaNalbandian commented Dec 16, 2021

humphd commented Dec 16, 2021

AmasiaNalbandian commented Dec 16, 2021

humphd commented Dec 16, 2021

AmasiaNalbandian commented Dec 16, 2021

humphd left a comment

humphd Dec 16, 2021

humphd Dec 16, 2021

humphd commented Jan 20, 2022

AmasiaNalbandian commented Jan 20, 2022

AmasiaNalbandian commented Jan 20, 2022

JerryHue left a comment

AmasiaNalbandian commented Jan 20, 2022

HyperTHD left a comment

AmasiaNalbandian commented Jan 20, 2022

HyperTHD commented Jan 20, 2022

added advanced search query for elasticsearch #2617

added advanced search query for elasticsearch #2617

Conversation

AmasiaNalbandian commented Dec 15, 2021 • edited by HyperTHD Loading

Issue This PR Addresses

Type of Change

Description

Steps to test:

Checklist

gitpod-io bot commented Dec 15, 2021

humphd commented Dec 15, 2021

AmasiaNalbandian commented Dec 15, 2021

humphd left a comment

Choose a reason for hiding this comment

AmasiaNalbandian commented Dec 16, 2021

humphd commented Dec 16, 2021

AmasiaNalbandian commented Dec 16, 2021

humphd commented Dec 16, 2021

AmasiaNalbandian commented Dec 16, 2021

humphd left a comment

Choose a reason for hiding this comment

humphd Dec 16, 2021

Choose a reason for hiding this comment

humphd Dec 16, 2021

Choose a reason for hiding this comment

humphd commented Jan 20, 2022

AmasiaNalbandian commented Jan 20, 2022

AmasiaNalbandian commented Jan 20, 2022

JerryHue left a comment

Choose a reason for hiding this comment

AmasiaNalbandian commented Jan 20, 2022

HyperTHD left a comment

Choose a reason for hiding this comment

AmasiaNalbandian commented Jan 20, 2022

HyperTHD commented Jan 20, 2022

AmasiaNalbandian commented Dec 15, 2021 •

edited by HyperTHD

Loading