Implement Author Autocomplete (legacy backend and Search) #3301

RC-Lee · 2022-03-24T01:05:47Z

Add author autocomplete settings for indexer.js in backend
refactor Search service to work with new indexing
create new search query for author autocomplete
add supertest as a dev-dependency to pass ESLint

Issue This PR Addresses

Fixes #1417
also a part of issue #3225 (no new tests are added just yet)

Type of Change

New Feature: Change which adds functionality

Description

In the legacy backend, I've added settings and mapping for the posts index to use edge n-gram for author autocomplete
Because we couldn't specify type on index creation, I've refactored the Search service and tests to work with the new indexing. The current search query is not affected by this new change. pnpm test search would pass all tests.
To get author autocomplete suggestions, I've created a new query out of Search, and ways to test it and see it in action is as follows

Steps to test the PR

get on my branch, pnpm install
**Important** remove redis cache rm -rf redis-data, this is so the posts can be re-indexed
build Docker Images

 docker-compose --env-file ./config/env.development up --build redis elasticsearch posts traefik search nginx planet

Wait for a bit for ElasticSearch to connect, then pnpm start
Wait a bit for the posts to be indexed
Can now test Search queries at http://localhost/v1/search
The original Search queries shouldn't be affected. The new query can be tested at http://localhost/v1/search/authors/autocomplete/
When author is queried, it will return the number of suggested authors, and an array of suggested authors. As the query gets longer, results will be more precise

Problems

There is still a lot of tweaking that can be done

A strict validation for the http://localhost/v1/search/authors/autocomplete/ isn't yet in place. Currently I am only hijacking the old search validation. If we don't have an author query and it passes validation, we'll get a Error 400
A query of only one letter won't pass validation, it has to be 2 or more letters. This is part of our rules in validation.js

Checklist

Quality: This PR builds and passes our npm test and works locally
Tests: This PR includes thorough tests or an explanation of why it does not
Screenshots: This PR includes screenshots or GIFs of the changes made or an explanation of why it does not (if applicable)variables are added/changed or an explanation of why it does not(if applicable)

gitpod-io · 2022-03-24T01:05:56Z

humphd

This is amazing! I've done one read through, and will spend more time on it tomorrow, but it looks fantastic.

We should figure out if we can migrate this to the parser service, since it's landing soon (can happen in follow-up).

src/api/search/src/routes/query.js

RC-Lee · 2022-03-24T03:45:57Z

I've changed the route name, and I've also added highlights
ElasticSearch can help us wrap the searched terms with <em>...</em> tags. Which might be useful for the front end.
So we can decide if we want to keep one of authors and highlights or both

(I tested without waiting for all feeds to put in)

src/api/search/src/routes/query.js

src/api/search/src/search.js

src/backend/utils/indexer.js

DukeManh

Google should consider hiring us.

src/api/search/src/search.js

TueeNguyen

I had to spend some time learning on Edge n gram which was very interesting.
I tried to test this PR, the server ran but I couldn't to get any result back.
* http://localhost/v1/search/complete/?author=Da returns 404
* http://localhost/v1/search/authors/autocomplete/?author=Da returns 404

Edit: my backend took a while to index for some reason

Also the query looks strange, should it be http://localhost/v1/search/autocomplete?author=Da

DukeManh · 2022-03-24T18:48:31Z

tried to test this PR, the server ran but I couldn't to get any result back.

The URL changed to http://localhost/v1/search/authors/autocomplete/?author=dav.

Kevan-Y · 2022-03-24T18:51:25Z

tried to test this PR, the server ran but I couldn't to get any result back.

The URL changed to http://localhost/v1/search/authors/autocomplete/?author=dav.

I think it would be better to have http://localhost/v1/search/authors/autocomplete?author=dav

TueeNguyen · 2022-03-24T18:52:23Z

Yeah agree, the endpoint should look like autrocomplete?author=dav instead of /?author=dav

src/backend/utils/indexer.js

Kevan-Y · 2022-03-25T00:52:33Z

http://localhost/v1/search/authors/autocomplete/?author=dav

@TueeNguyen , That interesting how the routing works, http://localhost/v1/search/authors/autocomplete/?author=dav and http://localhost/v1/search/authors/autocomplete?author=dav are the same

TueeNguyen · 2022-03-25T00:54:45Z

I think whatever is after the ? counts as query parameter.

RC-Lee · 2022-03-25T02:03:42Z

Added comments
Added new author validation rule
Merged author and highlight

humphd

Let's do it.

humphd · 2022-03-25T14:39:42Z

src/api/search/src/bin/validation.js

 */
 const validateQuery = () => {
  return async (req, res, next) => {
-    await Promise.all(ValidationRules.map((rule) => rule.run(req)));
+    const rules = req.route.path === '/' ? ValidationRules : authorRules;


This makes me nervous, but let's go with it for now and fix later. I'd rather split these routes up vs. having this logic determine what we do.

src/api/search/src/search.js

Kevan-Y

LGTM, tested locally it works. Great feature @rclee91

resolved

- Add author autocomplete settings for indexer.js in backend - refactor Search service to work wwith new indexing - create /authors/autocomplete search query for author autocomplete - add search term highlights - add comments - create new validation rules for author autocomplete

vercel bot deployed to Preview March 24, 2022 01:05 View deployment

humphd requested review from TueeNguyen and humphd March 24, 2022 01:10

RC-Lee force-pushed the author-autocomplete branch from 1401467 to dac33df Compare March 24, 2022 01:12

vercel bot deployed to Preview March 24, 2022 01:12 View deployment

TueeNguyen requested a review from Kevan-Y March 24, 2022 01:12

humphd reviewed Mar 24, 2022

View reviewed changes

src/api/search/src/routes/query.js Outdated Show resolved Hide resolved

manekenpix assigned RC-Lee Mar 24, 2022

RC-Lee force-pushed the author-autocomplete branch from dac33df to d8867e9 Compare March 24, 2022 03:43

vercel bot deployed to Preview March 24, 2022 03:43 View deployment

AmasiaNalbandian requested review from AmasiaNalbandian, JerryHue, JiaHua-Zou and tcvan0707 March 24, 2022 12:11

humphd requested changes Mar 24, 2022

View reviewed changes

Kevan-Y reviewed Mar 24, 2022

View reviewed changes

src/backend/utils/indexer.js Show resolved Hide resolved

DukeManh previously requested changes Mar 24, 2022

View reviewed changes

src/api/search/src/search.js Outdated Show resolved Hide resolved

src/api/search/src/search.js Outdated Show resolved Hide resolved

TueeNguyen reviewed Mar 24, 2022

View reviewed changes

src/backend/utils/indexer.js Show resolved Hide resolved

TueeNguyen reviewed Mar 24, 2022

View reviewed changes

src/backend/utils/indexer.js Show resolved Hide resolved

RC-Lee force-pushed the author-autocomplete branch from d8867e9 to 00138d0 Compare March 25, 2022 02:02

vercel bot deployed to Preview March 25, 2022 02:02 View deployment

RC-Lee requested review from humphd, DukeManh, Kevan-Y and TueeNguyen March 25, 2022 02:03

RC-Lee force-pushed the author-autocomplete branch from 00138d0 to 5ebc4bf Compare March 25, 2022 02:14

vercel bot deployed to Preview March 25, 2022 02:14 View deployment

humphd approved these changes Mar 25, 2022

View reviewed changes

Kevan-Y approved these changes Mar 25, 2022

View reviewed changes

RC-Lee mentioned this pull request Mar 25, 2022

Clear ElasticSearch index on Staging/Production to re-index posts for author autocomplete #3332

Closed

JerryHue force-pushed the author-autocomplete branch from 5ebc4bf to 3c7bd21 Compare March 25, 2022 16:09

JerryHue merged commit 9875dcb into Seneca-CDOT:master Mar 25, 2022

JerryHue added this to the 2.9 Release milestone Mar 25, 2022

RC-Lee mentioned this pull request Apr 6, 2022

Autocomplete indexing in Parser #3416

Merged

RC-Lee deleted the author-autocomplete branch April 7, 2022 20:48

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement Author Autocomplete (legacy backend and Search) #3301

Implement Author Autocomplete (legacy backend and Search) #3301

RC-Lee commented Mar 24, 2022 •

edited

Loading

gitpod-io bot commented Mar 24, 2022

humphd left a comment

RC-Lee commented Mar 24, 2022

DukeManh left a comment

TueeNguyen left a comment •

edited

Loading

DukeManh commented Mar 24, 2022

Kevan-Y commented Mar 24, 2022 •

edited

Loading

TueeNguyen commented Mar 24, 2022

Kevan-Y commented Mar 25, 2022 •

edited

Loading

TueeNguyen commented Mar 25, 2022

RC-Lee commented Mar 25, 2022

humphd left a comment

humphd Mar 25, 2022

Kevan-Y left a comment •

edited

Loading

Implement Author Autocomplete (legacy backend and Search) #3301

Implement Author Autocomplete (legacy backend and Search) #3301

Conversation

RC-Lee commented Mar 24, 2022 • edited Loading

Issue This PR Addresses

Type of Change

Description

Steps to test the PR

Problems

Checklist

gitpod-io bot commented Mar 24, 2022

humphd left a comment

Choose a reason for hiding this comment

RC-Lee commented Mar 24, 2022

DukeManh left a comment

Choose a reason for hiding this comment

TueeNguyen left a comment • edited Loading

Choose a reason for hiding this comment

DukeManh commented Mar 24, 2022

Kevan-Y commented Mar 24, 2022 • edited Loading

TueeNguyen commented Mar 24, 2022

Kevan-Y commented Mar 25, 2022 • edited Loading

TueeNguyen commented Mar 25, 2022

RC-Lee commented Mar 25, 2022

humphd left a comment

Choose a reason for hiding this comment

humphd Mar 25, 2022

Choose a reason for hiding this comment

Kevan-Y left a comment • edited Loading

Choose a reason for hiding this comment

RC-Lee commented Mar 24, 2022 •

edited

Loading

TueeNguyen left a comment •

edited

Loading

Kevan-Y commented Mar 24, 2022 •

edited

Loading

Kevan-Y commented Mar 25, 2022 •

edited

Loading

Kevan-Y left a comment •

edited

Loading