Skip to content

Commit 56def4e

Browse files
aspitarlaspitarl
committed
add energy storage nlp plot
1 parent 98b2ecd commit 56def4e

File tree

5 files changed

+279
-0
lines changed

5 files changed

+279
-0
lines changed

‎energy_storage_nlp/generate_df_info.py

Lines changed: 34 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
#%%
2+
import pandas as pd
3+
import sqlite3
4+
import os
5+
import sys
6+
7+
data_folder = r'C:\Users\aspit\Git\MHDLab-Projects\Energy Storage\data'
8+
9+
con = sqlite3.connect(os.path.join(data_folder, 'nlp.db'))
10+
cursor = con.cursor()
11+
12+
df = pd.read_sql_query("SELECT * FROM texts", con, index_col='ID')
13+
14+
df = df.dropna(subset=['processed_text'])
15+
df = df[df['language'] == 'en']
16+
17+
#%%
18+
19+
# %%
20+
num_papers = str(len(df))
21+
22+
print('Number of papers: ' + num_papers)
23+
24+
# %%
25+
search_terms = ", ".join(set(df['searchterm']))
26+
27+
28+
print('Search Terms: ' + search_terms)
29+
30+
31+
32+
33+
34+
# %%

‎energy_storage_nlp/index.html

Lines changed: 105 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,105 @@
1+
<!DOCTYPE html>
2+
<meta content="text/html;charset=utf-8" http-equiv="Content-Type">
3+
<meta content="utf-8" http-equiv="encoding">
4+
<html>
5+
<head>
6+
<title>Energy Storage Abstract Clustering</title>
7+
<style>
8+
9+
.column {
10+
float: left;
11+
}
12+
13+
/* Clear floats after the columns */
14+
.row:after {
15+
content: "";
16+
display: table;
17+
clear: both;
18+
19+
}
20+
</style>
21+
22+
</head>
23+
24+
<body>
25+
26+
<h1>
27+
<center>Clustering of abstracts related to energy storage</center>
28+
</h1>
29+
30+
<p>
31+
Below are interactive plots visualizing topic modeling on a collection of article abstracts pulled from Microsoft Academic related to energy storage.
32+
</p>
33+
34+
<h2>
35+
Obtaining the abstracts
36+
</h2>
37+
<p>
38+
The abstracts were obtained with the search terms below, returning the top 1000 results. Duplicate papers were removed (identified by DOI) and only articles in english were retained, resulting in 7857 abstracts. :
39+
</p>
40+
<p>
41+
<u>Search Terms:</u> High Temperature Energy Storage, Energy Storage, Fossil Energy Storage, Superconducting Magnetic Energy Storage, Thermal Energy Storage, Flow Battery Energy Storage, Electrochemical Energy Storage, Advanced Adiabatic Compressed Air Energy Storage, Liquid Air Energy Storage, Thermochemical Energy Storage, Mechanical Energy Storage, Sensible Thermal Energy Storage, Methanol Energy Storage, Hydrogen Energy Storage, Li-ion Energy Storage, Lead Acid Energy Storage, Latent Thermal Energy Storage, Ammonia Energy Storage
42+
43+
44+
</p>
45+
46+
<h2>
47+
Topic Modeling
48+
</h2>
49+
50+
<p>
51+
Topic modeling was performed using Latent Diriclet Allocation (LDA) with gensim. LDA is an unsupervised machine learning technique to determine a set of topics that can represnt the modeled collection of texts (corpus).
52+
53+
Each document is given a probability of being in each topic, where topics are probability distributions over words. This is a 'soft' clustering technique, in contrast to Kmeans (used previously) which assigns each document to just one cluster. This removes the nuance of papers that lie at the intersection between fields.
54+
</p>
55+
56+
<h2> Topic Visualization with t-SNE </h2>
57+
58+
<p>
59+
Below is a visualization of the topic modeling of the corpus. First, the texts are represented as points on a 2D surface using t-Distributed Stochastic Neighbor Embedding (t-SNE).
60+
61+
The topic distribution for each paper is visualized by representing each paper as a pie chart. Each slice represents a topic, and the fractional size (angle) of each slice represents the probability of that topic. Only the top 3 topics for each paper are inclused (resulting in an incomplete pie chart) for the sake of graphics processing.
62+
63+
The top words for each topic are indicated in the legend (see next visualization to explore the topic words in more detail). The topics in the legend are sorted by the number of papers that have that topic as their most probable topic.
64+
65+
<br><br>
66+
To use the plot, mouse over each item to get information about the paper. Papers can be clicked to open up the articles web page. Use the tools on the right to move around, and note the 'refresh' button to reset the graph. Topics can be hidden by clicking on the topic color in the legend.
67+
</p>
68+
69+
70+
<div class="row">
71+
72+
<embed type="text/html" src="wedgeplot.html" style="width:100%" height=650>
73+
74+
75+
</div>
76+
77+
<h2> Topic Visualization with pyLDAvis </h2>
78+
79+
<p>
80+
81+
Below is the visualization of the LDA model using pyLDAvis. The graph on the left using Principal Component Analysis to visualize the topics in 2D, similar to TSNE. The dashboard on the right is useful for exploring the words associated with each topic. Slide the relevance metric to about 0.5 to get words more specific to each topic.
82+
83+
</p>
84+
85+
86+
<div class="row">
87+
<embed type="text/html" src="lda_model.html" style="width:100%" height=900>
88+
</div>
89+
90+
91+
<h1> Model Parameters</h1>
92+
93+
<p>
94+
Bigram modeling parameters: {'min_count': 5, 'threshold': 100}
95+
<br>
96+
Num Bigrams: 950, Total Words: 25076, Bigram Fraction: 0.038
97+
<br>
98+
LDA modeling paramters: {'alpha': 0.2, 'eta': 0.2, 'num_topics': 20, 'passes': 5}
99+
100+
</p>
101+
102+
103+
</body>
104+
</html>
105+

‎energy_storage_nlp/lda_model.html

Lines changed: 41 additions & 0 deletions
Large diffs are not rendered by default.

‎energy_storage_nlp/wedgeplot.html

Lines changed: 85 additions & 0 deletions
Large diffs are not rendered by default.

‎index.html

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
<!DOCTYPE html>
2+
<meta content="text/html;charset=utf-8" http-equiv="Content-Type">
3+
<meta content="utf-8" http-equiv="encoding">
4+
<html>
5+
<body>
6+
<center>
7+
8+
<h1>Welcome!</h1>
9+
10+
<h2><a href="energy_storage_nlp/index.html">Energy Storage Literature Clustering</a> </h2>
11+
12+
13+
</body>
14+
</html>

0 commit comments

Comments
 (0)