Skip to content

Commit bf6a3fe

Browse files
mziccardmziccard
committed
Add BigQuery module README
1 parent 7eee528 commit bf6a3fe

File tree

1 file changed

+239
-7
lines changed

1 file changed

+239
-7
lines changed

‎gcloud-java-bigquery/README.md‎

Lines changed: 239 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -8,16 +8,21 @@ Java idiomatic client for [Google Cloud BigQuery] (https://cloud.google.com/bigq
88
<!-- TODO(mziccard): add in the maven shield once the artifact is pushed to maven -->
99

1010
- [Homepage] (https://googlecloudplatform.github.io/gcloud-java/)
11-
+<!-- TODO(mziccard): add link to API documentatin -->
11+
- [API Documentation] (http://googlecloudplatform.github.io/gcloud-java/apidocs/index.html?com/google/gcloud/bigquery/package-summary.html)
1212

1313
> Note: This client is a work-in-progress, and may occasionally
1414
> make backwards-incompatible changes.
1515
1616
Quickstart
1717
----------
18-
Add this to your pom.xml file
19-
<!-- TODO(mziccard): add dependency code -->
18+
If you are using Maven, add this to your pom.xml file
19+
<!-- TODO(mziccard): add mvn dependency code -->
2020

21+
If you are using Gradle, add this to your dependencies
22+
<!-- TODO(mziccard): add gradle dependency code -->
23+
24+
If you are using SBT, add this to your dependencies
25+
<!-- TODO(mziccard): add sbt dependency code -->
2126

2227
Example Application
2328
-------------------
@@ -36,14 +41,239 @@ About Google Cloud BigQuery
3641
Data can be streamed into BigQuery at millions of rows per second to enable real-time analysis.
3742
With BigQuery you can easily deploy Petabyte-scale Databases.
3843

39-
Be sure to activate the Google Cloud BigQuery API on the Developer's Console to use BigQuery from your project.
44+
Be sure to activate the Google Cloud BigQuery API on the Developer's Console to use BigQuery from
45+
your project.
4046

4147
See the ``gcloud-java`` API [bigquery documentation][bigquery-api] to learn how to interact
4248
with Google Cloud BigQuery using this Client Library.
4349

44-
Here is a code snippet showing a simple usage example from within Compute/App Engine. Note that you must [supply credentials](https://github.com/GoogleCloudPlatform/gcloud-java#authentication) and a project ID if running this snippet elsewhere.
50+
Getting Started
51+
---------------
52+
#### Prerequisites
53+
For this tutorial, you will need a
54+
[Google Developers Console](https://console.developers.google.com/) project with the BigQuery API
55+
enabled. You will need to [enable billing](https://support.google.com/cloud/answer/6158867?hl=en) to
56+
use Google Cloud BigQuery.
57+
[Follow these instructions](https://cloud.google.com/docs/authentication#preparation) to get your
58+
project set up. You will also need to set up the local development environment by [installing the
59+
Google Cloud SDK](https://cloud.google.com/sdk/) and running the following commands in command line:
60+
`gcloud auth login` and `gcloud config set project [YOUR PROJECT ID]`.
61+
62+
#### Installation and setup
63+
You'll need to obtain the `gcloud-java-bigquery` library. See the [Quickstart](#quickstart) section
64+
to add `gcloud-java-bigquery` as a dependency in your code.
65+
66+
#### Creating an authorized service object
67+
To make authenticated requests to Google Cloud BigQuery, you must create a service object with
68+
credentials. You can then make API calls by calling methods on the BigQuery service object. The
69+
simplest way to authenticate is to use
70+
[Application Default Credentials](https://developers.google.com/identity/protocols/application-default-credentials).
71+
These credentials are automatically inferred from your environment, so you only need the following
72+
code to create your service object:
73+
74+
```java
75+
import com.google.gcloud.bigquery.BigQuery;
76+
import com.google.gcloud.bigquery.BigQueryOptions;
77+
78+
BigQuery bigquery = BigQueryOptions.defaultInstance().service();
79+
```
80+
81+
For other authentication options, see the
82+
[Authentication](https://github.com/GoogleCloudPlatform/gcloud-java#authentication) page.
83+
84+
#### Creating a dataset
85+
With BigQuery you can create datasets. A dataset is a grouping mechanism that holds zero or more
86+
tables. Add the following import at the top of your file:
87+
88+
```java
89+
import com.google.gcloud.bigquery.DatasetInfo;
90+
```
91+
Then, to create the dataset, use the following code:
92+
93+
```java
94+
// Create a dataset
95+
String datasetId = "my_dataset_id";
96+
bigquery.create(DatasetInfo.builder(datasetId).build());
97+
```
98+
99+
#### Creating a table
100+
With BigQuery you can create different types of tables: normal tables with an associated schema,
101+
external tables backed by data stored on Google Cloud Storage and view tables that are created from
102+
a BigQuery SQL query. In this code snippet we show how to create a normal table with only one string
103+
field. Add the following imports at the top of your file:
104+
105+
```java
106+
import com.google.gcloud.bigquery.Field;
107+
import com.google.gcloud.bigquery.Schema;
108+
import com.google.gcloud.bigquery.TableId;
109+
import com.google.gcloud.bigquery.BaseTableInfo;
110+
import com.google.gcloud.bigquery.TableInfo;
111+
```
112+
Then add the following code to create the table:
113+
114+
```java
115+
TableId tableId = TableId.of(datasetId, "my_table_id");
116+
// Table field definition
117+
Field stringField = Field.builder("StringField", Field.Type.string())
118+
.mode(Field.Mode.NULLABLE)
119+
.build();
120+
// Table schema definition
121+
Schema schema = Schema.of(stringField);
122+
// Create a table
123+
BaseTableInfo createdTableInfo = bigquery.create(TableInfo.of(tableId, schema));
124+
```
125+
126+
#### Loading data into a table
127+
BigQuery provides several ways to load data into a table: streaming rows or loading data from a
128+
Google Cloud Storage file. In this code snippet we show how to stream rows into a table.
129+
Add the following imports at the top of your file:
130+
131+
```java
132+
import com.google.gcloud.bigquery.InsertAllRequest;
133+
import com.google.gcloud.bigquery.InsertAllResponse;
134+
135+
import java.util.HashMap;
136+
import java.util.Map;
137+
```
138+
Then add the following code to insert data:
45139

46-
<!-- TODO(mziccard): add code snippet -->
140+
```java
141+
Map<String, Object> firstRow = new HashMap<>();
142+
Map<String, Object> secondRow = new HashMap<>();
143+
firstRow.put("StringField", "value1");
144+
secondRow.put("StringField", "value2");
145+
// Create an insert request
146+
InsertAllRequest insertRequest = InsertAllRequest.builder(tableId)
147+
.addRow(firstRow)
148+
.addRow(secondRow)
149+
.build();
150+
// Insert rows
151+
InsertAllResponse insertResponse = bigquery.insertAll(insertRequest);
152+
// Check if errors occurred
153+
if (insertResponse.hasErrors()) {
154+
System.out.println("Errors occurred while inserting rows");
155+
}
156+
```
157+
158+
#### Querying data
159+
BigQuery enables querying data by running queries and waiting for the result. Queries can be run
160+
directly or through a Query Job. In this code snippet we show how to run a query directly and wait
161+
for the result. Add the following imports at the top of your file:
162+
163+
```java
164+
import com.google.gcloud.bigquery.FieldValue;
165+
import com.google.gcloud.bigquery.QueryRequest;
166+
import com.google.gcloud.bigquery.QueryResponse;
167+
168+
import java.util.Iterator;
169+
import java.util.List;
170+
```
171+
Then add the following code to run the query and wait for the result:
172+
173+
```java
174+
// Create a query request
175+
QueryRequest queryRequest = QueryRequest.builder("SELECT * FROM my_dataset_id.my_table_id")
176+
.maxWaitTime(60000L)
177+
.maxResults(1000L)
178+
.build();
179+
// Request query to be executed and wait for results
180+
QueryResponse queryResponse = bigquery.query(queryRequest);
181+
while (!queryResponse.jobComplete()) {
182+
Thread.sleep(1000);
183+
queryResponse = bigquery.getQueryResults(queryResponse.jobId());
184+
}
185+
// Read rows
186+
Iterator<List<FieldValue>> rowIterator = queryResponse.result().iterateAll();
187+
System.out.println("Table rows:");
188+
while (rowIterator.hasNext()) {
189+
System.out.println(rowIterator.next());
190+
}
191+
```
192+
#### Complete source code
193+
194+
Here we put together all the code shown above into one program. This program assumes that you are
195+
running on Compute Engine or from your own desktop. To run this example on App Engine, simply move
196+
the code from the main method to your application's servlet class and change the print statements to
197+
display on your webpage.
198+
199+
```java
200+
import com.google.gcloud.bigquery.BaseTableInfo;
201+
import com.google.gcloud.bigquery.BigQuery;
202+
import com.google.gcloud.bigquery.BigQueryOptions;
203+
import com.google.gcloud.bigquery.DatasetInfo;
204+
import com.google.gcloud.bigquery.Field;
205+
import com.google.gcloud.bigquery.FieldValue;
206+
import com.google.gcloud.bigquery.InsertAllRequest;
207+
import com.google.gcloud.bigquery.InsertAllResponse;
208+
import com.google.gcloud.bigquery.QueryRequest;
209+
import com.google.gcloud.bigquery.QueryResponse;
210+
import com.google.gcloud.bigquery.Schema;
211+
import com.google.gcloud.bigquery.TableId;
212+
import com.google.gcloud.bigquery.TableInfo;
213+
214+
import java.util.HashMap;
215+
import java.util.Iterator;
216+
import java.util.List;
217+
import java.util.Map;
218+
219+
public class GcloudBigQueryExample {
220+
221+
public static void main(String[] args) throws InterruptedException {
222+
223+
// Create a service instance
224+
BigQuery bigquery = BigQueryOptions.defaultInstance().service();
225+
226+
// Create a dataset
227+
String datasetId = "my_dataset_id";
228+
bigquery.create(DatasetInfo.builder(datasetId).build());
229+
230+
TableId tableId = TableId.of(datasetId, "my_table_id");
231+
// Table field definition
232+
Field stringField = Field.builder("StringField", Field.Type.string())
233+
.mode(Field.Mode.NULLABLE)
234+
.build();
235+
// Table schema definition
236+
Schema schema = Schema.of(stringField);
237+
// Create a table
238+
BaseTableInfo createdTableInfo = bigquery.create(TableInfo.of(tableId, schema));
239+
240+
// Define rows to insert
241+
Map<String, Object> firstRow = new HashMap<>();
242+
Map<String, Object> secondRow = new HashMap<>();
243+
firstRow.put("StringField", "value1");
244+
secondRow.put("StringField", "value2");
245+
// Create an insert request
246+
InsertAllRequest insertRequest = InsertAllRequest.builder(tableId)
247+
.addRow(firstRow)
248+
.addRow(secondRow)
249+
.build();
250+
// Insert rows
251+
InsertAllResponse insertResponse = bigquery.insertAll(insertRequest);
252+
// Check if errors occurred
253+
if (insertResponse.hasErrors()) {
254+
System.out.println("Errors occurred while inserting rows");
255+
}
256+
257+
// Create a query request
258+
QueryRequest queryRequest = QueryRequest.builder("SELECT * FROM my_dataset_id.my_table_id")
259+
.maxWaitTime(60000L)
260+
.maxResults(1000L)
261+
.build();
262+
// Request query to be executed and wait for results
263+
QueryResponse queryResponse = bigquery.query(queryRequest);
264+
while (!queryResponse.jobComplete()) {
265+
Thread.sleep(1000);
266+
queryResponse = bigquery.getQueryResults(queryResponse.jobId());
267+
}
268+
// Read rows
269+
Iterator<List<FieldValue>> rowIterator = queryResponse.result().iterateAll();
270+
System.out.println("Table rows:");
271+
while (rowIterator.hasNext()) {
272+
System.out.println(rowIterator.next());
273+
}
274+
}
275+
}
276+
```
47277

48278
Java Versions
49279
-------------
@@ -53,7 +283,9 @@ Java 7 or above is required for using this client.
53283
Testing
54284
-------
55285

56-
<!-- TODO(mziccard): add this in once the RemoteGCBQMHelper class is functional -->
286+
This library has tools to help make tests for code using Cloud BigQuery.
287+
288+
See [TESTING] to read more about testing.
57289

58290
Versioning
59291
----------

0 commit comments

Comments
 (0)