Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

update #18

Merged
merged 347 commits into from
Dec 31, 2020
Merged
Changes from 1 commit
Commits
Show all changes
347 commits
Select commit Hold shift + click to select a range
4fa704c
Update index.md
mbaabuones Dec 12, 2020
24d88a4
Update index.md
mbaabuones Dec 12, 2020
f40bb7d
Update index.md
mbaabuones Dec 12, 2020
418d91d
- prelim revision done
LinusMuema Dec 12, 2020
379c5bf
- prelim revision done
LinusMuema Dec 12, 2020
3aff831
messed up the first PR, doing a second
tensoncai Dec 12, 2020
e3e7bb9
minor changes
lalith1403 Dec 13, 2020
3415743
minor changes
lalith1403 Dec 13, 2020
15de3ca
minor changes
lalith1403 Dec 13, 2020
4c7a61f
submitted article and images
EKahuha Dec 13, 2020
597061b
added image source link
EKahuha Dec 13, 2020
9322bc3
- fix the repo suggestion
LinusMuema Dec 13, 2020
b04ecd3
Merge remote-tracking branch 'upstream/master' into gitlab
adithbharadwaj Dec 13, 2020
7881d99
used headers h3 and beyond, reduced repetition of phrases
tensoncai Dec 13, 2020
b1208a6
fixed merge conflicts
tensoncai Dec 13, 2020
9d599a4
minor changes
JudyNduati Dec 14, 2020
18a06ab
final update
JudyNduati Dec 14, 2020
d1a88e8
final update
JudyNduati Dec 14, 2020
61fa570
Update index.md
adrianmurage Dec 14, 2020
ebc8f93
made requested change
lalith1403 Dec 14, 2020
64d8246
- rename repl.it
LinusMuema Dec 14, 2020
2b7e31a
minor changes
JudyNduati Dec 14, 2020
2fa3ef3
- format code snippets
LinusMuema Dec 14, 2020
af81c3f
Update index.md
paulodhiambo Dec 14, 2020
3bbcbe1
Update index.md
paulodhiambo Dec 14, 2020
aecfd4c
third commit
bluemutheu Dec 14, 2020
317d4e7
Made requested changes
Dec 14, 2020
5e0f4ee
Minor edits
cyberShaw Dec 14, 2020
18b4230
Add peer review line
cyberShaw Dec 14, 2020
9d6eb3b
added front matter
hectorkambow Dec 14, 2020
7298a8f
Merge pull request #794 from kimkimani/Google-Play-In-App-Review-usin…
hectorkambow Dec 14, 2020
f1697e4
collection of plagiarism
Jeff-mwangi Dec 14, 2020
fe8ab3e
added front matter
hectorkambow Dec 14, 2020
026ea81
Merge pull request #838 from EKahuha/iot-deployments-with-sd-wan
hectorkambow Dec 14, 2020
41b796b
change file name
hectorkambow Dec 14, 2020
1301022
added couple links
hectorkambow Dec 14, 2020
2ef1d46
Merge pull request #766 from mbaabuones/autoscaling-in-aws2
hectorkambow Dec 14, 2020
89f20ce
adding front matter review process walkthrough
hectorkambow Dec 15, 2020
e2aa002
made requested changes
lalith1403 Dec 15, 2020
b1a0a40
minor changes
lalith1403 Dec 15, 2020
44129f3
minor changes
lalith1403 Dec 15, 2020
4e66f11
reverting
bluemutheu Dec 15, 2020
59a4169
updating the changes
Jeff-mwangi Dec 15, 2020
e43b707
adding front matter
hectorkambow Dec 15, 2020
357ba0e
minor
hectorkambow Dec 15, 2020
8f9c3e7
minor
hectorkambow Dec 15, 2020
cc5667e
Merge pull request #816 from lalith1403/nlp-tl
hectorkambow Dec 15, 2020
9efd1be
preliminery changes
bluemutheu Dec 15, 2020
e0ab832
adding front matter
hectorkambow Dec 15, 2020
2221de6
minor time
hectorkambow Dec 15, 2020
7a1cc0b
Merge pull request #762 from lalith1403/divide-n-conquer
hectorkambow Dec 15, 2020
299d5d3
made the requested changes
adithbharadwaj Dec 15, 2020
2934c5a
Minor Updates to Flow & Spacing
zolomohan Dec 15, 2020
fb93a18
Finish grammer review
Dec 15, 2020
96f70a7
Merge pull request #961 from section-io/adding-front-matter-doc
hectorkambow Dec 15, 2020
27b6d64
minor - list format
hectorkambow Dec 15, 2020
0b975c9
adding note
hectorkambow Dec 15, 2020
d14c462
change front matter
hectorkambow Dec 15, 2020
c29a6c2
minor made into list
hectorkambow Dec 15, 2020
d3595cd
minor
hectorkambow Dec 15, 2020
f8ee11f
adding image of front matter
hectorkambow Dec 15, 2020
77f83f4
minor
hectorkambow Dec 15, 2020
9d0f796
date and update image
hectorkambow Dec 15, 2020
b82175a
Merge pull request #971 from section-io/adding-image-front-matter
hectorkambow Dec 15, 2020
a2fae75
added front matter
hectorkambow Dec 15, 2020
ee15d40
minor
hectorkambow Dec 15, 2020
a3fc6f0
Merge pull request #854 from LinusMuema/android-workmanager
hectorkambow Dec 15, 2020
73625a7
second revision
bluemutheu Dec 16, 2020
0fd899d
spacing
zolomohan Dec 16, 2020
629e7b6
remove redundant headings
zolomohan Dec 16, 2020
e57e5e4
Fix headings
zolomohan Dec 16, 2020
32676a9
third revision
bluemutheu Dec 16, 2020
4a35970
Merge branch 'master' of https://github.com/bluemutheu/engineering-ed…
bluemutheu Dec 16, 2020
0278b22
fourth revision
bluemutheu Dec 16, 2020
7bc6e75
minor changes
lalith1403 Dec 16, 2020
ffe1ebd
minor
lalith1403 Dec 16, 2020
724b8e5
current edits
hectorkambow Dec 16, 2020
0e55454
minor changes
lalith1403 Dec 16, 2020
c02b870
updating changes
Jeff-mwangi Dec 16, 2020
38c023b
Finish tech review
Dec 16, 2020
e285f1b
Added peer reviewer line
Dec 16, 2020
13f6761
additional minor edits for flow
hectorkambow Dec 16, 2020
bb48334
edits to flow and grammar
zolomohan Dec 16, 2020
eac8fa0
updating minor changes
Jeff-mwangi Dec 16, 2020
b113d02
final preview
hectorkambow Dec 16, 2020
7a90791
time
hectorkambow Dec 16, 2020
d5dccf9
Merge pull request #851 from paulodhiambo/securing-django-api
hectorkambow Dec 16, 2020
79e426d
- added peer reviewer line
LinusMuema Dec 16, 2020
ebcd9ef
- added peer reviewer line
LinusMuema Dec 16, 2020
e638753
Minor fixes
srishilesh Dec 16, 2020
8d290d5
fix merge conflict issues
zolomohan Dec 16, 2020
f01bf90
minor updates
zolomohan Dec 16, 2020
ea8d5e8
minor updates
zolomohan Dec 16, 2020
8d1974e
add front matter
zolomohan Dec 16, 2020
1df9525
changed hero image
hectorkambow Dec 16, 2020
42d5f3f
minor fix
zolomohan Dec 16, 2020
0185d0f
tech revision
bluemutheu Dec 16, 2020
efb12c7
added author picture and bio
tensoncai Dec 16, 2020
9d6eeb0
Merge branch 'master' of https://github.com/bluemutheu/engineering-ed…
bluemutheu Dec 16, 2020
d5aa6b6
changed hero image again-not rendering
hectorkambow Dec 16, 2020
dae6ac0
first tech rev
bluemutheu Dec 16, 2020
6db4ef1
had to place older image-hero did not render
hectorkambow Dec 16, 2020
234035f
Merge pull request #902 from EKahuha/breaking-down-public-cloud-vs-hy…
hectorkambow Dec 16, 2020
49fa86d
edits to flow
zolomohan Dec 16, 2020
00be8ef
minor update
zolomohan Dec 16, 2020
b75e98f
punctuation and minor updates
zolomohan Dec 16, 2020
4f681a4
minor updates
zolomohan Dec 16, 2020
cee7186
add front matter
zolomohan Dec 16, 2020
e24825d
adding hero image
hectorkambow Dec 16, 2020
c0463d2
Merge pull request #883 from mbaabuones/understanding-hill-climbing-i…
hectorkambow Dec 16, 2020
f0809fc
minor
hectorkambow Dec 16, 2020
ea0104e
adding front matter
hectorkambow Dec 16, 2020
2ed46b6
preim review
hectorkambow Dec 17, 2020
acdd6ad
minor edits for flow
hectorkambow Dec 17, 2020
b48ab79
alt tag
hectorkambow Dec 17, 2020
5b774f2
Merge pull request #872 from JudyNduati/exception-handling-in-javascript
hectorkambow Dec 17, 2020
cff95ba
Minor changes on image
Pius1991 Dec 17, 2020
b19e261
Merge branch 'digital-assistants-vs-digital-companions' of https://gi…
Pius1991 Dec 17, 2020
f0031b5
re-submitted PR
EKahuha Dec 17, 2020
f7aaed1
minor edits for flow
hectorkambow Dec 17, 2020
32e980a
minor
hectorkambow Dec 17, 2020
61a5881
Merge pull request #790 from lalith1403/activation-func
hectorkambow Dec 17, 2020
0da9b63
added front matter
hectorkambow Dec 17, 2020
023bb79
Merge pull request #937 from tensoncai/master
hectorkambow Dec 17, 2020
d8b7168
Minor updates
collins-a Dec 17, 2020
c6130a6
adding front matter
hectorkambow Dec 17, 2020
00ddf40
Merge pull request #874 from kayere/introduction-to-room-db
hectorkambow Dec 17, 2020
3141ac8
add a missing explanation
cwilvx Dec 17, 2020
6ee3213
adding front matter
hectorkambow Dec 17, 2020
c1a42eb
minor & time
hectorkambow Dec 17, 2020
6adb072
Merge pull request #906 from collins-a/active-learning
hectorkambow Dec 17, 2020
1de7698
made some edits for flow
hectorkambow Dec 18, 2020
ff28631
minor
hectorkambow Dec 18, 2020
92c12a7
Merge pull request #915 from Pius1991/digital-assistants-vs-digital-c…
hectorkambow Dec 18, 2020
1cfead0
Update index.md
Manley12 Dec 18, 2020
06057d9
resized hero
adithbharadwaj Dec 18, 2020
ff636e2
Merge branch 'gitlab' of https://github.com/adithbharadwaj/engineerin…
adithbharadwaj Dec 18, 2020
094364d
Fixed changes
bloodprashure Dec 18, 2020
69a2a1b
add peer review attribution
adrianmurage Dec 18, 2020
7c68cec
add peer review contribution
adrianmurage Dec 18, 2020
74e1297
Update index.md
WanjaMIKE Dec 18, 2020
0190aeb
very minor changes for flow
hectorkambow Dec 18, 2020
a32eaed
minor
hectorkambow Dec 18, 2020
af17147
Merge pull request #884 from mwangiKibui/diving-deeper-into-js-objects
hectorkambow Dec 18, 2020
f4ab185
adding front matter
hectorkambow Dec 18, 2020
6553e7d
image paths
hectorkambow Dec 18, 2020
94d7adf
Merge pull request #890 from geoffrey45/django-google-oauth
hectorkambow Dec 18, 2020
dcca221
alt image tag
hectorkambow Dec 18, 2020
c3afca0
adding front matter process link
hectorkambow Dec 18, 2020
c1a1c14
added front matter
hectorkambow Dec 19, 2020
1f12096
Merge pull request #998 from EKahuha/how-to-optimize-cloud-computing-…
hectorkambow Dec 19, 2020
f486eaa
Update index.md
WanjaMIKE Dec 19, 2020
82eb03f
minor edits for flow
hectorkambow Dec 19, 2020
4a7f044
Merge pull request #833 from bloodprashure/master
hectorkambow Dec 19, 2020
54e7308
minor edits for flow
hectorkambow Dec 21, 2020
5c40b6d
minor
hectorkambow Dec 21, 2020
92560e9
Merge pull request #868 from srishilesh/building-xml-webpages-using-xslt
hectorkambow Dec 21, 2020
f9d4a59
edit to hero image
hectorkambow Dec 21, 2020
ce88a6a
alt image tage
hectorkambow Dec 21, 2020
0aa3a40
Merge pull request #893 from adithbharadwaj/gitlab
hectorkambow Dec 21, 2020
260ae2e
Update index.md
jumamiller Dec 21, 2020
afde33e
change file name
hectorkambow Dec 21, 2020
eab1570
added front matter
hectorkambow Dec 21, 2020
3bea775
alt image tag
hectorkambow Dec 21, 2020
14fb05e
Merge pull request #923 from collins-a/underspecification
hectorkambow Dec 21, 2020
98d8833
edit spacing
zolomohan Dec 22, 2020
4788c45
Update index.md
WanjaMIKE Dec 22, 2020
2759636
Add files via upload
mbaabuones Dec 22, 2020
2f06427
Update index.md
mbaabuones Dec 22, 2020
d30b561
Delete equation.gif
mbaabuones Dec 22, 2020
5dc95b4
Add files via upload
mbaabuones Dec 22, 2020
af20a42
Update index.md
mbaabuones Dec 22, 2020
8b5184d
Update index.md
mohamedgh16 Dec 22, 2020
f1dface
Add files via upload
mohamedgh16 Dec 22, 2020
90112a5
changed file name
hectorkambow Dec 22, 2020
679861f
minor change
lalith1403 Dec 22, 2020
4fa1baf
adding peer reviewer line
hectorkambow Dec 22, 2020
a8dad58
change name of file
hectorkambow Dec 22, 2020
0bc0650
minor change
lalith1403 Dec 22, 2020
acef63a
minor
hectorkambow Dec 22, 2020
b0438d5
Merge pull request #925 from Jeff-mwangi/introduction-to-c#
hectorkambow Dec 22, 2020
b03f862
minor
hectorkambow Dec 22, 2020
db33223
minor
hectorkambow Dec 22, 2020
52ee915
Finish prelim
Dec 22, 2020
70d131d
minor edits
zolomohan Dec 22, 2020
285ea62
minor edit on further reading
zolomohan Dec 22, 2020
907e584
minor edits
zolomohan Dec 22, 2020
a483e87
added front matter
hectorkambow Dec 22, 2020
0ecde8d
Merge pull request #938 from EKahuha/comparing-intrusion-detection-sy…
hectorkambow Dec 22, 2020
ac2a8be
very minor edits for flow
hectorkambow Dec 22, 2020
8c96b28
minor added mathjax script
hectorkambow Dec 22, 2020
0172120
Merge pull request #917 from mbaabuones/an-overview-of-fuzzy-logic-sy…
hectorkambow Dec 22, 2020
259f8e3
minor edits for flow
hectorkambow Dec 23, 2020
89f8ec2
minor
hectorkambow Dec 23, 2020
66a5e47
minor
hectorkambow Dec 23, 2020
67709c9
date time
hectorkambow Dec 23, 2020
9d8623c
minor
hectorkambow Dec 23, 2020
aeaabde
Merge pull request #933 from bluemutheu/master
hectorkambow Dec 23, 2020
ec11932
minor changes
WanjaMIKE Dec 23, 2020
41f1dc7
Finish tech review
Dec 23, 2020
c8dc7bd
Major edits
kimkimani Dec 23, 2020
3abdc26
Update index.md
kimkimani Dec 23, 2020
a37fac2
Update index.md
mohamedgh16 Dec 23, 2020
8899afa
minor edits
zolomohan Dec 23, 2020
234aff1
minor
zolomohan Dec 23, 2020
f2ad73c
minor
zolomohan Dec 23, 2020
ff43a00
Minor edits
cyberShaw Dec 23, 2020
f8492e0
Add front matter
cyberShaw Dec 23, 2020
060f8ad
Add peer review line
cyberShaw Dec 23, 2020
1e0cc30
minor
zolomohan Dec 23, 2020
4ce142e
add front matter
zolomohan Dec 23, 2020
9515a58
Add Image URLs
zolomohan Dec 23, 2020
5d333b1
Delete properties window and toolbox.png
mohamedgh16 Dec 24, 2020
8b535b3
replacing image
mohamedgh16 Dec 24, 2020
45ae44d
renaming image
mohamedgh16 Dec 24, 2020
dae0ea0
Add files via upload
mbaabuones Dec 24, 2020
0425372
Delete hero-image.webp
mbaabuones Dec 24, 2020
4e58e87
Add files via upload
mbaabuones Dec 24, 2020
f0e3067
add peer review line
zolomohan Dec 28, 2020
a643ec2
Update Fork with Master
louisefindlay23 Dec 28, 2020
fe5016c
Fixed EJS partial syntax error
louisefindlay23 Dec 28, 2020
8101f7d
Merge pull request #1087 from louisefindlay23/article-fix
ninjaginja Dec 28, 2020
a4598e1
some minor changes to Format and style for flow
hectorkambow Dec 28, 2020
5b0c326
minor
hectorkambow Dec 28, 2020
520087e
Merge pull request #844 from jumamiller/laravel-8-new-features
hectorkambow Dec 28, 2020
d642a49
minor
hectorkambow Dec 28, 2020
b54d6ae
minor edits for flow
hectorkambow Dec 28, 2020
57327f0
minor
hectorkambow Dec 28, 2020
2daa800
description change
hectorkambow Dec 28, 2020
abfac3b
Merge pull request #904 from mbaabuones/automation-testing-in-selenium
hectorkambow Dec 28, 2020
22dc799
previewed
hectorkambow Dec 29, 2020
1ce0658
title
hectorkambow Dec 29, 2020
cfd1a02
minor
hectorkambow Dec 29, 2020
65e804b
Merge pull request #840 from lalith1403/hugging-face
hectorkambow Dec 29, 2020
ffdc1d7
very minor edits for flow and format
hectorkambow Dec 29, 2020
ec28928
minor
hectorkambow Dec 29, 2020
6ebf313
minor
hectorkambow Dec 29, 2020
6853d4b
Merge pull request #905 from LinusMuema/android-navigation-components
hectorkambow Dec 29, 2020
b03c618
very minor edits for flow
hectorkambow Dec 29, 2020
365e29a
Merge pull request #913 from mohamedgh16/master
hectorkambow Dec 30, 2020
efffc27
minor
WanjaMIKE Dec 30, 2020
5a78d58
rebrand of EngEd portion of site
ninjaginja Dec 30, 2020
afda56d
Merge branch 'master' of https://github.com/section-io/engineering-ed…
ninjaginja Dec 30, 2020
bc8b234
Finish review
Dec 30, 2020
55dd957
minor formatting updates
ninjaginja Dec 30, 2020
78cafa7
Merge pull request #796 from kimkimani/Android-Application-An-app-Upd…
ninjaginja Dec 30, 2020
bc79eb1
added front matter
ninjaginja Dec 30, 2020
7ad0ded
minor formatting updates
ninjaginja Dec 30, 2020
b940952
Remove personal note
lalith1403 Dec 30, 2020
7113676
Merge pull request #801 from lalith1403/qna
ninjaginja Dec 30, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Finish review
  • Loading branch information
Peter Kayere committed Dec 30, 2020
commit bc8b2341200a01167b9ce892ba5a0dfaa6a72dca
63 changes: 28 additions & 35 deletions articles/question-answering/index.md
Original file line number Diff line number Diff line change
@@ -1,10 +1,8 @@
### An Introduction to Question Answering Systems

Recently I came across this library that enables one to create large-scale and distributed question answering systems. It solves a significant problem that we usually face in a vast field as Natural Language Processing. The problem of creating models that work accurately with large amounts of data. Hence my interest in the area of question answering systems arose. I have tried to put together the general trends in the field of question answering systems. Once we have a good picture of the area, we will explore the library ** Haystack**, which arose my interest in the first place.
Recently, I came across this library that enables one to create large-scale and distributed question answering systems. It solves a significant problem that we usually face in the vast field of Natural Language Processing. The problem of creating models that work accurately with large amount of data. Hence, my interest in the area of question answering systems arose. I have tried to put together the general trends in the field of question answering systems. Once we have a good picture of the area, we will explore the library **Haystack**, which arose my interest in the first place.

### Goal of Question Answering Systems

What are the question answering systems trying to achieve? Given a question, should the system return the nearest answer? What if the answer is a mixture of multiple answers that it has been trained on? We are trying to build intelligent systems that don't just return documents related to the question but extracts relevant information within the documents and puts forth the detailed answer, one like a human would have come up with. This task comes under the field of [Information Retrieval ](https://nlp.stanford.edu/IR-book/pdf/01bool.pdf)(IR), [Information Extraction](https://www.ontotext.com/knowledgehub/fundamentals/information-extraction/) (IE), and Natural Language Processing (NLP).
What are the question answering systems trying to achieve? Given a question, should the system return the nearest answer? What if the answer is a mixture of multiple answers that it has been trained on? We are trying to build intelligent systems that don't just return documents related to the question, but extracts relevant information within the documents and puts forth the detailed answer. Like one that a human would have come up with. This task comes under the field of [Information Retrieval ](https://nlp.stanford.edu/IR-book/pdf/01bool.pdf)(IR), [Information Extraction](https://www.ontotext.com/knowledgehub/fundamentals/information-extraction/) (IE), and Natural Language Processing (NLP).

Google is a search engine. But, some questions are generated automatically while we search. This is due to the [knowledge graphs](/engineering-education/web-developers-guide-seo/#pagerank) that Google uses to represent vast amounts of data.

Expand All @@ -13,67 +11,62 @@ Google is a search engine. But, some questions are generated automatically while
[*Image Source*](https://www.google.com/search?q=what+is+python&oq=what+is+python&aqs=chrome..69i57.10903j0j1&sourceid=chrome&ie=UTF-8)

### Structure of Question Answering System

The design of a question answering system has specific vital components. There are three distinct modules used in a question-answering system:
- Query Processing Module: Classifies questions according to the context. This module identifies the context and focus, classifies the type of question, and sets the answer type's expectations.
- Document Processing Module: Information retrieval module that focuses on gathering relevant documents
The design of a question answering system has specific vital components. There are three distinct modules used in a question-answering system:
- Query Processing Module: Classifies questions according to the context. This module identifies the context and focus, classifies the type of question, and sets the answer type's expectations.
- Document Processing Module: Information retrieval module that focuses on gathering relevant documents.
- Answer Processing Module: Once the relevant documents are retrieved, they need to be parsed through to obtain an accurate and appropriate answer.

Each of these modules takes up a different task to give relevant answers. The image given below summarizes the entire architecture.
Each of these modules perform a different task to give relevant answers. The image below summarizes the entire architecture.

![architecture of question answering system](architecture-qna.png)

[*Image Source*](http://www.aliallam.net/upload/598575/documents/ECFF549932079694.pdf)

### Query Processing Module

As mentioned earlier, the query processing system deals with three main tasks:
1. *Analysis* of the question to obtain the preliminary information from the question.
2. *Classification* of the type of question to better understand the required context for the answer. For example, the phrase _Python error_ should lead to answers related to coding, and _Python bite_ should lead to snake bites.
3. *Reformulation* of the question to obtain relevant answers. Therefore the question is converted into a pre-trained vector with several such pairs of examples of questions and answers. This aspect is responsible for information retrieval.
1. *Analysis* of the question to obtain preliminary information from the it.
2. *Classification* of the type of question to better understand the required context for the answer. For example, the phrase _Python error_ should lead to answers related to coding, and _Python bite_ should lead to snake bites.
3. *Reformulation* of the question to obtain relevant answers. Therefore, the question is converted into a pre-trained vector with several examples of question and answer pairs. This aspect is responsible for information retrieval.

### Document Processing Module
The document processing module accepts the reformulated question as its input. The document processing module uses an internal information retrieval (IR) system to map the closest documents to the question presented. The documents are sorted according to their similarity and relevance to the question. The document processing module performs three main tasks.

The document processing module accepts the reformulated question as its input. The document processing module uses an information retrieval (IR) system internally to map the closest documents to the question presented. The most similar documents are sorted according to their similarity and relevance to the question. The document processing module performs mainly three significant tasks.

1. *Retrieve* the set of the documents from the IR system
2. *Filter* the set of documents obtained in the previous step and reduce the amount of text in each document to produce a concise answer.
1. *Retrieve* the set of documents from the IR system.
2. *Filter* the set of documents obtained from the previous step, and reduce the amount of text in each document to produce a concise answer.
3. *Order* the documents by similarity and relevance to the question.

### Answer Processing Module

The final module gets the list of concise documents filtered and ordered by the previous module. The answer processing module considers the set of documents and performs three major tasks:

1. *Identify* statements/answers within the concise set of documents.
2. *Extract* the relevant answer by choosing the appropriate phrases and words that answers the question. This used to be performed heuristically. *Heuristic algorithms* are a way of solving problems where solving in the best possible way is given importance. Parameters such as cost complexity in terms of space and time, accuracy, or speed are neglected, and instead, obtaining a solution close to the actual answer is emphasized.

Heuristic algorithms are usually used to solve [NP-complete problems](https://www.britannica.com/science/NP-complete-problem), where there is no known efficient way to find a solution. However, it can be verified if the answer is known by plugging the parameters into the algorithm. Therefore, developing heuristic algorithms involves going back and forth between the solution and produced and the algorithm itself.

The algorithm is said to perform well when the algorithm's solution is as efficient as possible, given the constraints.
2. *Extract* the relevant answer by choosing the appropriate phrases and words that answers the question. This used to be performed heuristically. *Heuristic algorithms* are a way of solving problems where solving in the best possible way is given importance. Parameters such as cost, complexity in terms of space and time, accuracy, or speed are neglected, and instead, obtaining a solution close to the actual answer is emphasized.

3. *Validation* of the answers obtained by the previous step. This is usually used during the design of such question-answering systems to evaluate the outputs obtained. During deployment, the evaluation metrics are reduced to achieve high data throughput from the system.
Heuristic algorithms are usually used to solve [NP-complete problems](https://www.britannica.com/science/NP-complete-problem), where there is no known efficient way to find a solution. However, it can be verified if the answer is known by plugging the parameters into the algorithm. Therefore, developing heuristic algorithms involves going back and forth between the solution produced and the algorithm itself.

The algorithm is said to perform well when the algorithm's solution is as efficient as possible, given the constraints.

3. *Validation* of the answers obtained by the previous step. This is usually used during the design of such question-answering systems to evaluate the outputs obtained. During deployment, the evaluation metrics are reduced to achieve high data throughput from the system.

### Evaluating the answers obtained

The criteria can vary from paper to paper. But more generally, we want the question answering system to produce relevant, correct, and complete answers to the point. Hence many evaluation metrics were developed to measure such ambiguous terminologies. A few of those metrics are F1-scores, precision, recall, etc. For a more detailed insight into evaluation metrics, refer to this [article](/engineering-education/evaluating-ml-model-performance/).
### Evaluating the answers obtained
The criteria can vary from paper to paper. But more generally, we want the question answering system to produce relevant, correct, and complete answers to the point. Hence many evaluation metrics were developed to measure such ambiguous terminologies. A few of those metrics are F1-scores, precision, recall, etc. For a detailed insight into evaluation metrics, refer to this [article](/engineering-education/evaluating-ml-model-performance/).

Transformers at scale for question answering & neural search. Using NLP via a modular Retriever-Reader-Pipeline. Supporting DPR, Elasticsearch, HuggingFace's Modelhub

### Applications
The applications of such systems are varied and exciting. Currently used in commercial applications, these have a broad scope in almost every aspect. Humans are question-driven organisms, and therefore these systems are the easiest way for humans to interact with machines.

The applications of such systems are varied and exciting. Currently used in commercial applications, these have a broad scope in almost every aspect. Humans are question-driven organisms, and therefore these systems are the easiest way for humans to interact with machines.

1. **Customer Support**: The most commonly used application is using such systems to reduce the loads on customer service teams. The idea behind introducing such systems was to remove the early jitters that these systems face uncertainty related to the queries. This results in enabling the customer service team to focus on things that truly matter. However, today the quest for complete automation is on, and question answering systems integrated into chatbots are revolutionizing the digital experience rapidly.
2. **Education**: Google is a great search engine. But [Chegg](https://www.chegg.com/) is a great search engine built for finding answers to questions. Imagine a world where such devices power up the entire thought process required for a kid to learn. The kid is prompted with the right questions to be asked to understand the concepts. This is an example of a question answering system integrated into personalized learning systems.
3. **Search Engines**: As mentioned above, Google uses such systems to come up with questions and answers. When clicked on either of the questions, the number of questions shown increases. The updated list is more similar to the question that was clicked on. If not observed, next time you click on a question on Google, be sure to check it out.
4. **Data Analytics**: [HyperVerge](https://hyperverge.co/) is an exciting company that works on generating analytical reports based on queries typed by the user in natural language. This application is liberating an entire generation of the workforce from developing reports and identifying patterns in data. Just asking for specific queries results in beautiful reports. This is an example of a question and answering system integrated with an analytics engine.
1. **Customer Support**: The most common application is using such systems to reduce the loads on customer service teams. The idea behind introducing such systems was to remove the early jitters that these systems face uncertainty related to the queries. This enables the customer service team to focus on things that truly matter. However, today, the quest for complete automation is on, and question answering systems integrated into chatbots are revolutionizing the digital experience rapidly.
2. **Education**: Google is a great search engine. But [Chegg](https://www.chegg.com/) is a great search engine built for finding answers to questions. Imagine a world where such devices power up the entire thought process required for a kid to learn. The kid is prompted with the right questions to understand the concepts. This is an example of a question answering system integrated into personalized learning systems.
3. **Search Engines**: As mentioned above, Google uses such systems to come up with questions and answers. When clicked on either of the questions, the number of questions shown increases. The updated list is more similar to the question that was clicked on. If not observed, next time you click on a question on Google, be sure to check it out.
4. **Data Analytics**: [HyperVerge](https://hyperverge.co/) is an exciting company that works on generating analytical reports based on queries typed by the user in natural language. This application is liberating an entire generation of the workforce from developing reports to identifying patterns in data. Just asking for specific queries results in beautiful reports. This is an example of a question and answering system integrated with an analytics engine.

### Research Direction

The building of such distributed systems is a very tough job. The real-time question and answering machines are challenging due to the scale at which they operate. Today databases are in Petabytes, and accessing information rapidly is a huge challenge. Many solutions for efficient storage and computation are present. But, ensuring algorithms work correctly with such solutions is a significant challenge. One such example is [Haystack](https://haystack.deepset.ai/). They have brought together neural question answering systems and made it compatible with distributed file systems such as [Hadoop](https://hadoop.apache.org/), [Elasticsearch](https://www.elastic.co/), etc. This way, we have the advantage of the most powerful [language models](https://huggingface.co/) generating answers and the power of speedy information retrieval systems taking in as much data as possible to generate the responses.
The building of such distributed systems is a very tough job. The real-time question and answering machines are challenging due to the scale at which they operate. Today's databases are in Petabytes, and accessing information rapidly is a huge challenge. Many solutions for efficient storage and computation are present. But, ensuring algorithms work correctly with such solutions is a significant challenge. One such example is [Haystack](https://haystack.deepset.ai/). They have brought together neural question answering systems and made it compatible with distributed file systems such as [Hadoop](https://hadoop.apache.org/), [Elasticsearch](https://www.elastic.co/), etc. This way, we have the advantage of the most powerful [language models](https://huggingface.co/) generating answers and the power of speedy information retrieval systems taking in as much data as possible to generate the responses.


### Conclusion
In this article, we considered the various aspects of a question answering system and its applications. The possibilities are endless. I hope I have succeeded in kindling the curiosity within you to explore the topic furthermore. I would recommend reading this [survey paper](https://www.sciencedirect.com/science/article/pii/S1877050915034663) published in 2015 at Elsevier.

In this article, we considered the various aspects of a question answering system and its applications. The possibilities are endless. I hope I have succeeded in kindling the curiosity within you to explore the topic furthermore. I would recommend reading this [survey paper](https://www.sciencedirect.com/science/article/pii/S1877050915034663) published in 2015 at Elsevier.
---
Peer Review Contributions by: [Peter Kayere](/engineering-education/authors/peter-kayere/)