Skip to content
This repository has been archived by the owner on Mar 7, 2024. It is now read-only.

Add bq_ingested_timestamp #9

Merged
merged 9 commits into from
Jun 15, 2021
Merged

Add bq_ingested_timestamp #9

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
ae6a873
add bq_ingested_timestamp
mariusandra May 27, 2021
f9cee04
fix some errors, clean up some
neilkakkar Jun 7, 2021
6b7941d
remove unused vars
neilkakkar Jun 7, 2021
f13861b
address comments
neilkakkar Jun 9, 2021
4e30b7e
adjust default for deduplication
neilkakkar Jun 9, 2021
ff8eed7
rephrase as question
neilkakkar Jun 9, 2021
3f7fc92
rephrase hint
neilkakkar Jun 9, 2021
46f6d9b
remove deduplicate option
neilkakkar Jun 10, 2021
55384cb
copyedit
neilkakkar Jun 10, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
address comments
  • Loading branch information
@neilkakkar
neilkakkar committed Jun 9, 2021
commit f13861b4c409644c47976b947fef9c6ad318a40a
7 changes: 2 additions & 5 deletions index.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,9 +1,7 @@
import { createBuffer } from '@posthog/plugin-contrib'
import { Plugin, PluginMeta, PluginEvent } from '@posthog/plugin-scaffold'
import { Plugin, PluginMeta, PluginEvent, RetryError } from '@posthog/plugin-scaffold'
import { BigQuery, Table, TableField, TableMetadata } from '@google-cloud/bigquery'

class RetryError extends Error {}

type BigQueryPlugin = Plugin<{
global: {
bigQueryClient: BigQuery
Expand Down Expand Up @@ -47,7 +45,7 @@ export const setupPlugin: BigQueryPlugin['setupPlugin'] = async (meta) => {
throw new Error('Table ID not provided!')
}

if (config.deduplicateEvents in ['true', 'True', 't']) {
if (config.deduplicateEvents === 'Yes') {
global.deduplicateEvents = true
} else {
global.deduplicateEvents = false
Expand Down Expand Up @@ -100,7 +98,6 @@ export const setupPlugin: BigQueryPlugin['setupPlugin'] = async (meta) => {
metadata.schema.fields = metadata.schema.fields.concat(fieldsToAdd)
;[result] = await global.bigQueryTable.setMetadata(metadata)
} catch (error) {
console.error(error)
const fieldsToStillAdd = global.bigQueryTableFields.filter(
({ name }) => !result.schema?.fields?.find((f: any) => f.name === name)
)
Expand Down
7 changes: 4 additions & 3 deletions plugin.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -49,9 +49,10 @@
{
"key": "deduplicateEvents",
"name": "Attempt deduplicating events",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Given we have yes and no, might make sense wording as a question?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You mean something like: "Attempt deduplicating events?"

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, for example

"type": "string",
"default": "true",
"hint": "Default true. We try to deduplicate events being sent to Bigquery if set to true. Set to false to increase Bigquery insertion quota limit"
"type": "choice",
"choices": ["Yes", "No"],
"default": "Yes",
"hint": "Default 'Yes'. We try to deduplicate events being sent to Bigquery if set to 'Yes'. Set to 'No' to increase Bigquery insertion quota limit"
}
]
}