Bigtable¶
Google Cloud Platform provides hosting of Google’s Bigtable database, the original inspiration of HBase and many NoSQL storage systems. Because HBase is so similar to Bigtable, running OpenTSDB 2.3 and later with Google’s backend is simple. Indeed, the schemas (see HBase Schema) are exactly the same so all you have to do is create your Bigtable instance, create your TSDB tables using the Bigtable HBase shell, and fire up the TSDs.
Note
The clients for Bigtable are in beta and undergoing a number of changes. Performance should improve as we adjust the code and uncover new tuning parameters. Please help us out on the mailing list or by modifying the code in GitHub.
Setup¶
To run under GKE, see Monitoring time-series data with OpenTSDB on Cloud Bigtable and GKE
Setup your Google Cloud Platform account.
Follow the steps in Creating a Cloud Bigtable Cluster.
Follow the steps in HBase Shell Quickstart, paying attention to where you download your JSON key file.
Set the HBASE_HOME environment variable to your Bigtable shell directory, make sure the HBASE_CLASSPATH, JAVA_HOME, and GOOGLE_APPLICATION_CREDENTIALS environment variables have been set according to the values in Creating a Cloud BigTable Cluster document, and run the src/create_table.sh script. If the script fails to launch the shell, try running the shell manually and execute the create statements substituting the proper values.
Build TSDB by executing sh build-bigtable.sh (or if you prefer Maven, sh build-bigtable.sh pom.xml)
Prepare the opentsdb.conf file with the required and/or optional configuration parameters below.
Run the TSD via build/tsdb tsd –config=<path>/opentsdb.conf
Configuration¶
The following is a table of required and optional parameters to run OpenTSDB with Bigtable. These are in addition to the standard TSD configuration parameters from Configuration.
Property |
Type |
Required |
Description |
Default |
|---|---|---|---|---|
google.bigtable.project.id |
String |
Required |
The project ID hosting your Bigtable cluster. |
|
google.bigtable.instance.id |
String |
Required |
The cluster ID you assigned to your Bigtable cluster at creation. Note that prior to AsyncBigtable 0.3.0 the value was |
|
google.bigtable.zone.id |
String |
Required |
The zone where your Bigtable cluster is operating; chosen at creation. Note that prior to AsyncBigtable 0.3.0 the value was |
|
hbase.client.connection.impl |
String |
Required |
The class that will be used to implement the HBase API AsyncBigtable will use as a shim between the Bigtable client and OpenTSDB. Set this to |
|
google.bigtable.auth.service.account.enable |
Boolean |
Required |
Whether or not to use a Google cloud service account to connect. Set this to true |
false |
google.bigtable.auth.json.keyfile |
String |
Required |
The full path to the JSON formatted key file associated with the service account you want to use for Bigtable access. Download this from your cloud console. |
|
google.bigtable.grpc.channel.count |
Integer |
Optional |
The number of sockets opened to the Bigtable API for handling RPCs. For higher throughput consider increasing the channel count. |
4 |
Note
In older version’s of the client, Google’s Bigtable client communicates with their servers over HTTP2 with TLS using ALPN. As Java 7 and 8 (dunno about 9) lack native ALPN support, a library must be loaded at JVM start to modify the JVM’s bytecode. The build script for OpenTSDB will attempt to detect your JDK version and download the proper version of ALPN but if you have a custom JVM or something other than Hotspot or OpenJDK you may run into issues. Try different versions of the alpn-boot JAR to see what works for you. With AsyncBigtable 0.3.0 a JDK agnostic library is used so ALPN is no longer required.