Skip to content

Adding course name #122

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 31 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
31 commits
Select commit Hold shift + click to select a range
fe645ef
Project proposal submission
christianopperman Oct 27, 2023
6cc7539
Scrape tool created. Subtitles scraped.
a598504775 Nov 17, 2023
a4d325e
Chrome extension - initial commit
himangshu81 Nov 17, 2023
c91948c
Update the subtitle dataset and scrape code
a598504775 Nov 18, 2023
632ce68
Merge branch 'main' of https://github.com/christianopperman/CS410_Fal…
a598504775 Nov 18, 2023
ce06271
Add progress report to Github
christianopperman Nov 19, 2023
ef96643
Indexing data to Elasticsearch
himangshu81 Nov 24, 2023
0b18519
Adding basic search logic
himangshu81 Nov 29, 2023
05764b0
Working plugin
himangshu81 Dec 1, 2023
6b0c737
Update README documentation
christianopperman Dec 3, 2023
7619d23
Generalize scraping script
christianopperman Dec 3, 2023
1463476
Update gitignore, include documentation images
christianopperman Dec 3, 2023
ec86dc0
Increase sleep waittime
christianopperman Dec 3, 2023
c852c1c
Update .gitignore
christianopperman Dec 5, 2023
7e1881f
Refine scraper script to have a single point of entry
christianopperman Dec 5, 2023
1c00933
Add requirements.txt file for Python scraper
christianopperman Dec 5, 2023
fd17060
Update ElasticSearch writer
christianopperman Dec 5, 2023
2555ae6
Rename Python script parent folder
christianopperman Dec 5, 2023
67e773f
Move project proposal and progress report to Documentation folder
christianopperman Dec 5, 2023
7fc19bd
Remove log and credentials from GitHub
christianopperman Dec 5, 2023
ad413d5
Rename Extension folder
christianopperman Dec 5, 2023
af2406a
Update ES to index Week, lecture title and subtitles and including th…
himangshu81 Dec 7, 2023
470f9fc
Adding the manifest file back for chrome extension
himangshu81 Dec 7, 2023
3a295f3
Removing .idea
himangshu81 Dec 7, 2023
3b73924
Build UI
a598504775 Dec 8, 2023
0a1f5de
chat coursera first commit. Does not run yet.
aadi123 Dec 8, 2023
c6934f3
Fix course title scraping
christianopperman Dec 10, 2023
90b0244
Update README documentation. Add course title scraping to CourseraScr…
christianopperman Dec 10, 2023
f48293a
Updating ES URL and password.
himangshu81 Dec 11, 2023
b77e7be
Cleaning up to "subtitles" index.
himangshu81 Dec 11, 2023
97c667c
Adding course_name in search result and pushing to ES
himangshu81 Dec 12, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
__pycache__/*
.idea
.idea/*
node_modules
*.docx
*.DS_Store
*.iml
*.log
*.csv
*.pyc
*/subtitles.json
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
52 changes: 52 additions & 0 deletions ChromeExtension/index.html
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8" />
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1.0" />
<link rel="stylesheet" href="style.css" />
<title>Search Coursera Lectures</title>
</head>
<body>
<div class="extension__container">
<header class="header__course">
<select name="course_title" id="course-options">
<option value="text">Text Information System</option>
<option value="data">Data Mining</option>
</select>
</header>
<div class="result__container" id="result-container">

<!-- This is the sample format of widgets
<div class="result__item">
<a class="lecture__url" href="www.google.com"></a>
<h4>
Week 1
</h4>
<div class="result__second--row">
<h4 class="timestamp">
00:00:00
</h4>
<h4>
Lecture name
</h4>
</div>
<p>
Subtitles:sdfsdfwfwdfwdfwdfwdfwdfdfwdfwdfwe
</p>
</div> -->

</div>
<footer class="footer__input">
<input type="text" id="searchbox" placeholder="Search text...">
<button id="submit-button">Submit</button>
</footer>
</div>
<script src="js/search.js"></script>
<!-- <script type="module" src="../node_modules/elasticsearch"></script>
<script type="module" src="../node_modules/@elastic"></script> -->
</body>
</html>



181 changes: 181 additions & 0 deletions ChromeExtension/js/search.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,181 @@
const search_btn = document.getElementById("submit-button");
const result_container = document.querySelector('#result-container')

search_btn.addEventListener('click', function () {
if (result_container.childElementCount > 0) {
// console.log("Has child(ren)")
remove_all_children(result_container)
}

search_api()
});

async function search_wild() {
// console.log("Inside search_wild..")
//import {Client} from '@elastic'

const ES_URL = "https://search-cs410-project-hw5dhpc4jsg3m74vnbalajt754.aos.us-east-1.on.aws"
const ES_USER = "elastic"
const ES_PASSWORD = "replace me"

const client = new Client({
node: ES_URL,
auth: {
username: ES_USER,
password: ES_PASSWORD
}
})


const query_str = document.getElementById("searchbox").textContent
// console.log("query_str ", query_str)
const result = await client.search({
index: 'subtitles',
size: 1,
from: 0,
query: {
"query_string": {
"query": query_str,
"default_field": "search_for"
}
}
})
const timestam_obj = result.hits.hits[0]._source
return timestam_obj;
}


async function search_api() {

// console.log("Inside search_api..")

var headers = new Headers();
headers.append("Content-Type", "application/json");
headers.append("Authorization", "Basic ZWxhc3RpYzpwY2lXY2xwTE5kWHVpY1VoWFY4YmhnazI=");

const query_txt = document.getElementById("searchbox").value
// console.log("query_txt ", query_txt)
const query_payload = {
size: 5,
from: 0,
query: {
"query_string": {
"query": query_txt
}
}
}
// console.log("query_payload ", query_payload)
var requestOptions = {
method: 'POST',
headers: headers,
body: JSON.stringify(query_payload)
};

const response = await fetch("https://ac55987c83844faa90726d4e5efe92b9.us-central1.gcp.cloud.es.io/subtitles/_search", requestOptions)
const record = await response.json()
// console.log("record ", record)
if(record.hits.total.value > 0) {
const result_num = Math.min(record.hits.total.value, 5)
// console.log("Maximum number of result: ", result_num)
for (let i = 0; i < result_num; i++) {
const result = record.hits.hits[i]._source
// console.log(result)
const result_dict = {}
const response_str = '<strong>'+ result.week + ' </br> </strong>'
+ '<strong> Title :: </strong>' + result.lecture_title + '</br>' +
'<a href="' + result.url + '"> timestamp </a>:: ' + result.time + '<br/>'
+ '<strong> Subtitles </strong> : '+result.text
+ '</br>'
console.log("Resoponse :: ", response_str)
result_dict["week"] = result.week
result_dict["lecture_title"] = result.lecture_title
result_dict["url"] = result.url
result_dict["time"] = result.time
result_dict["subtitles"] = result.text
result_dict["course_name"] = result.course_name
set_result_format(result_dict)
}
} else {
const result_div = document.createElement('div')
result_div.innerHTML = "We could not find a related topic"
result_container.appendChild(result_div)
}

}

function set_result_format(result_dict) {

// Initiate html components
const result_item = document.createElement('div')
const result_second_row = document.createElement('div')
const result_url = document.createElement('a')
const result_week = document.createElement('h4')
const result_time = document.createElement('h4')
const result_lecture_title = document.createElement('h4')
const result_subtitles = document.createElement('p')

// Set up class/ id for some components
result_item.classList.add("result__item")
result_second_row.classList.add("result__second--row")
result_time.classList.add("timestamp")
result_url.classList.add("lecture__url")

// Set the content of components
result_url.href = result_dict["url"]
result_week.innerHTML = result_dict["week"]
time_reformat = format_time(result_dict["time"])
result_time.innerHTML = time_reformat
result_lecture_title.innerHTML = result_dict["lecture_title"]
result_subtitles.innerHTML = result_dict["subtitles"]

// Organize html component structure
result_item.appendChild(result_url)
result_item.appendChild(result_week)
result_item.appendChild(result_second_row)
result_second_row.appendChild(result_time)
result_second_row.appendChild(result_lecture_title)
result_item.appendChild(result_subtitles)

result_container.appendChild(result_item)
}

function format_time(time) {
let parts = time.split(':').map(part => parseInt(part, 10));
let seconds = parts[0];
let minutes = parts[1];
let hours = parts.length > 2 ? parts[2] : 0;

// Make sure each part has two digits
hours = hours.toString().padStart(2, '0');
minutes = minutes.toString().padStart(2, '0');
seconds = seconds.toString().padStart(2, '0');

return `${hours}:${minutes}:${seconds}`;
}

function remove_all_children(element) {
while (element.firstChild) {
element.removeChild(element.firstChild);
}
}

document.addEventListener('DOMContentLoaded', function () {
const parent = document.querySelector('.result__container');

parent.addEventListener('click', function (event) {
// Check if the clicked element or its parent has the class 'container'
let container = event.target.classList.contains('result__item')
? event.target
: event.target.closest('.result__item');

if (container) {
// Extract the URL from the child anchor tag
let url = container.querySelector('.lecture__url').getAttribute('href');

// Open the URL
if (url) {
chrome.tabs.create({ url: url });
}
}
});
});
16 changes: 16 additions & 0 deletions ChromeExtension/manifest.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
{
"name": "CS410_Fall2023_CourseProject_TeamCAHJ",
"description": "Base Level Extension",
"version": "1.0",
"permissions": [
"storage",
"tabs"
],
"host_permissions": ["http://*/*", "https://*/*"],
"manifest_version": 3,
"action": {
"default_popup": "index.html",
"default_icon": "img/CS410_Fall2023_CourseProject_TeamCAHJ.png",
"default_title": "CS410_Fall2023_CourseProject_TeamCAHJ"
}
}
132 changes: 132 additions & 0 deletions ChromeExtension/style.css
Original file line number Diff line number Diff line change
@@ -0,0 +1,132 @@
@import url('https://fonts.googleapis.com/css2?family=Roboto:wght@400;700&display=swap');

* {
box-sizing: border-box;
background-color: transparent;
}

body {
font-family: 'Roboto', sans-serif;
align-items: center;
justify-content:center;
height: 100%;
overflow: hidden;
margin: 0px;
}

.extension__container{
display: flex;
flex-direction: column;
outline: 1px solid black;
height: 600px;
width: 450px;
margin: 0px;
}

.header__course {
display: flex;
align-items: center;
background: white;
border-bottom: 1px solid rgb(225, 225, 225);
box-shadow: 4px 4px 8px 0 rgba(0, 0, 0, 0.2), 6px 6px 10px 0 rgba(0, 0, 0, 0.19);
height: 60px;
margin: 0;
padding: 10px;
}


#course-options {
border: none;
background-color: transparent;
font-size: 1.5rem;
font-weight: bold;
color: rgb(55, 55, 55);
flex-grow: 1;
word-wrap: break-word;
overflow: hidden;
max-height: 1.5em;
}

.result__container {
flex-grow: 1;
background: rgb(245,245,245);
overflow-y: auto;
margin: 0;
padding: 15px;
}

.result__container .result__item:hover {
cursor: pointer;
}

.result__item {
display: flex;
flex-direction: column;
background: white;
box-shadow: 0 2px 4px 0 rgba(0, 0, 0, 0.1), 0 3px 10px 0 rgba(0, 0, 0, 0.1);
border-radius: 8px;
margin-bottom: 15px;
padding: 10px;
}

.result__item h4 {
line-height: 1rem;
margin: 4px;
word-wrap: break-word;
overflow: hidden;
max-height: 1.5em;
}

.result__second--row {
display: flex;
flex-direction: row;
}

.timestamp {
color: rgb(47, 151, 242);
}

.result__item p {
margin: 4px;
word-wrap: break-word;
line-height: 1em;
max-height: 3em;
overflow: hidden;
position: relative;
}

/* .result__item p::after {
content: '...';
position: absolute;
bottom: 0;
right: 0;
} */

.footer__input {
display: flex;
align-items: center;
height: 60px;
background: white;
box-shadow: 0 4px 8px 0 rgba(0, 0, 0, 0.1), 0 6px 20px 0 rgba(0, 0, 0, 0.1);
border-top: 1px solid rgb(225, 225, 225);
margin: 0;
padding: 10px;
}

#searchbox{
flex-grow: 1;
margin-right: 10px;
background-color: white;
border: 2px solid grey;
border-radius: 5px;
height: 30px;
}

#submit-button {
color: white;
background-color: rgb(96, 176, 246);
border: none;
height: 30px;
border-radius: 3px;
}

Loading