Datastore Connectivity for BigQuery (bgc)

This library is compatible with Go 1.5+

Please refer to CHANGELOG.md if you encounter breaking changes.

• Usage
• License
• Credits and Acknowledgements

This library uses SQL mode and streaming API to insert data as default. To use legacy SQL please use the following /* USE LEGACY SQL */ hint, in this case you will not be able to fetch repeated and nested fields.

Configuration parameters

insertMethod

To control insert method just provide config.parameters with the following value:

_table_name_.insertMethod = "load"

Note that if streaming is used, currently UPDATE and DELETE statements are not supported.

insertIdColumn

For streaming you can specify which column to use as insertId with the following config.params

_table_name_.insertMethod = "stream"
_table_name_.insertIdColumn = "sessionId"

streamBatchCount

streamBatchCount controls row count in batch (default 9999)

insertWaitTimeoutInMs

When inserting data data this library checks upto 60 sec if data has been added. To control this behaviour you can set insertWaitTimeoutInMs (default 60 sec)

To disable this mechanism set: insertWaitTimeoutInMs: -1

insertMaxRetires

Retries insert when 503 internal error

datasetId

Default dataset

pageSize

Default 500

The maximum number of rows of data to return per page of results. In addition to this limit, responses are also limited to 10 MB.

Credentials

1. Google secrets for service account

a) set GOOGLE_APPLICATION_CREDENTIALS environment variable

b) credential can be a name with extension of the JSON secret file placed into ~/.secret/ folder

config.yaml

driverName: bigquerycredentials: bq # place your big query secret json to ~/.secret/bg.jsonparameters:
datasetId: myDataset

c) full URL to secret file

config.yaml

driverName: bigquerycredentials: file://tmp/secret/mySecret.jsonparameters:
datasetId: myDataset

Secret file has to specify the following attributes:

typeConfigstruct {
//google cloud credentialClientEmailstring`json:"client_email,omitempty"`TokenURLstring`json:"token_uri,omitempty"`PrivateKeystring`json:"private_key,omitempty"`PrivateKeyIDstring`json:"private_key_id,omitempty"`ProjectIDstring`json:"project_id,omitempty"`
}

2. Private key (pem)

config.yaml

driverName: bigquerycredentials: bq # place your big query secret json to ~/.secret/bg.jsonparameters:
serviceAccountId: "***@developer.gserviceaccount.com"datasetId: MyDatasetprojectId: spheric-arcadia-98015privateKeyPath: /tmp/secret/bq.pem

Usage:

The following is a very simple example of Reading and Inserting data

package main
import (
"github.com/viant/bgc""github.com/viant/dsc""time""fmt""log"
)
typeMostLikedCitystruct {
CitystringVisitsintSouvenirs []string
}
typeTravelerstruct {
IdintNamestringLastVisitTime time.TimeAchievements []stringMostLikedCityMostLikedCityVisitedCities []struct {
CitystringVisitsint
}
}
funcmain() {
config, err:=dsc.NewConfigWithParameters("bigquery", "",
"bq", // google cloud secret placed in ~/.secret/bg.jsonmap[string]string{
"datasetId":"MyDataset",
})
iferr!=nil {
log.Fatal(err)
}
factory:=dsc.NewManagerFactory()
manager, err:=factory.Create(config)
iferr!=nil {
log.Fatalf("Failed to create manager %v", err)
}
traveler:=Traveler{}
success, err:=manager.ReadSingle(&traveler, " SELECT id, name, lastVisitTime, visitedCities, achievements, mostLikedCity FROM travelers WHERE id = ?", []interface{}{4}, nil)
iferr!=nil {
panic(err.Error())
}
travelers:=make([]Traveler, 0)
err:=manager.ReadAll(&interest, "SELECT iid, name, lastVisitTime, visitedCities, achievements, mostLikedCity",nil, nil)
iferr!=nil {
panic(err.Error())
}
// ...inserted, updated, err:=manager.PersistAll(&travelers, "travelers", nil)
iferr!=nil {
panic(err.Error())
}
// ...//Custom reading handler with reading query info type to get CacheHit, TotalRows, TotalBytesProcessedvarresultInfo=&bgc.QueryResultInfo{}
varperf=make(map[string]int) err=manager.ReadAllWithHandler(`SELECT DATE(date), COUNT(*) FROM performance_agg WHERE DATE(date) = ? GROUP BY 1`, []interface{}{
"2018-05-03",
resultInfo,
}, func(scanner dsc.Scanner) (toContinuebool, errerror) {
vardatestringvarcountinterr=scanner.Scan(&date, &count)
iferr!=nil {
returnfalse, err
}
perf[date] =countreturntrue, nil
})
log.Printf("cache: %v, rows: %v, bytes: %v", resultInfo.CacheHit, resultInfo.TotalRows, resultInfo.TotalBytesProcessed)
dialect:=dsc.GetDatastoreDialect(config.DriverName)
DDL, err:=dialect.ShowCreateTable(manager, "performance_agg")
fmt.Printf("%v %v\n", DDL, err)
}

GoCover

License

The source code is made available under the terms of the Apache License, Version 2, as stated in the file LICENSE.

Individual files may be made available under their own specific license, all compatible with Apache License, Version 2. Please see individual files for details.

Credits and Acknowledgements

Library Author: Adrian Witas

Contributors: Mikhail Berlyant