You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: README.md
+12-3Lines changed: 12 additions & 3 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -4,7 +4,7 @@ This project aims to be a parallel and distributed implementation of the Gibbs d
4
4
The Gibbs sampling algorithm details are showed in the following [paper](http://stanford.edu/class/ee367/Winter2018/yue_ee367_win18_report.pdf).
5
5
6
6
### Processing pipeline
7
-
The program get the image that you want to process and split it into smaller chunks. Each sub-image, and its corresponding position, is pushed into a RDD and then processed by a Spark worker.
7
+
The program get the image that you want to process and split it into smaller chunks. Each sub-image, and its corresponding position, is pushed into a RDD (Resilient Distributed Dataset) and then processed by a Spark worker.
8
8
9
9
Multiple operations can be performed on the same chunk, these can be implemented exenting the <code>Pipeline</code> class and setting all the tasks needed. Tasks can also be implemented by extending the <code>Algorithm</code> trait.
10
10
@@ -79,9 +79,11 @@ To run the program on the Google Cloud platform you have to create a new project
79
79
80
80
And if you haven't done it yet you have to enable billing for the project.
81
81
82
-
The first step is to do the setup of the notebook environment variables. You will be asked to enable access to your google drive.
82
+
The first step is to do the setup of the notebook environment variables. You will be asked to enable access to your Google Drive and Google credentials.
83
83

84
-
Then you need to compile the file that is created in the root of your google drive with your project id and the name you want to give the bucket.
84
+
Then you need to fill the file that is created in the root of your google drive with your project id and the name you want to give the bucket.
85
+
86
+
Finally run all the cells in *Setup environment*, this will create a new bucket and all the the *./data* files are copied into it. It also create a new directory (*./bucket*) on the Colab runtime that is directly binded to the cloud storage bucket.
85
87
86
88
#### Simple job
87
89
To run a simple cluster with 2 workers (8 core) execute the cell *Simple cluster*. You can change the commands parameters to meet your needs.
@@ -90,12 +92,15 @@ To run a simple cluster with 2 workers (8 core) execute the cell *Simple cluster
90
92
91
93
#### Delete resources
92
94
To delete all resources allocated by the cluster and also all the bucket content, you can run the cell *Delete cluster and data*.
95
+
93
96

94
97
### Performance tests
95
98
There are also tests in the notebook to evaluate the performance of the cluster.
96
99
#### Strong scalability
97
100
In this test we want to see how the execution time changes by keeping the workload fixed and increasing the number of computational resources.
98
101
102
+
**REMEMBER TO DO THIS** to avoid paying for what you are not using.
0 commit comments