Teradata Direct connector in Kubernetes#

<< Return to section in k8s configuration documentation.

The Starburst Teradata Direct connector is supported for Kubernetes deployments in AWS EKS and in Azure AKS.

Warning

The configuration to use the Starburst Teradata Direct connector on Kubernetes is complex. You need significant Kubernetes and networking knowledge. Contact our Starburst Support team for assistance.

It relies on the Teradata cluster nodes, and the deployed table operator, to be able to address the SEP cluster nodes. Each worker node and the coordinator need an assigned IP address that is exposed outside the Kubernetes SEP cluster and is reachable from Teradata. This need translates into the following technical requirements:

A CNI (Container Network Interface) must be configured using this interface for AWS with SNAT disabled so that it is performed outside of Kubernetes, allowing pods to be accessed from outside the cluster by their assigned IP addresses.
Teradata must be deployed into the same Virtual Network via a VPC/VNet peering connection or VPN.
An adequate number of node instance network interfaces and IP address are available, taking autoscaling needs into account.

Thoroughly read the Teradata Direct connector documentation before you proceed to setup your environment, and create and verify catalogs.

AWS EKS setup#

AWS provides the vpc-cni addon to automate configuraiton of your EKS cluster. Follow the EKS Addons instructions for guidance.

After the EKS cluster is deployed, you must update its default security group with an inbound rule allowing the Teradata host to get back into the EKS cluster.

Azure AKS setup#

Ensure that Azure VNet CNI is applied to the cluster from your preferred management interface, such as Azure Portal or Terraform.

Only if you are using Terraform, the following is an example section:

resource "azurerm_kubernetes_cluster" "main" {
  network_profile {
    network_plugin    = "azure"
    network_mode      = "transparent"
  }
}

Catalog setup and verification#

Install the native table operator of the Teradata Direct connector on Teradata.
Add a catalog to your sep-prod-catalogs.yaml file, following this example:

catalogs:
  myteradatadirect: |-
    connector.name=teradata-direct
    connection-url=jdbc:teradata://<routable_ip_address>
    connection-user=test
    connection-password=test
    teradata-direct.table-operator.name=test.starburst_table_operator
    teradata-direct.http.port=9000
    teradata-direct.table-operator.logging-level=DEBUG

Verify all pods have IP assigned addresses in a shared network, and the following command does not return an error indicating that no IP addresses are available:

$ kubectl describe pod <my-deployment-xxxxxxxxxx-xxxxx> -n <my-namespace>

Failed to create pod sandbox: rpc error: code = Unknown desc = failed to
set up sandbox container "<LongHash> network for pod
"<my-deployment-59f5f68b58-c89wx>": networkPlugin cni failed to set up
pod "<my-deployment-59f5f68b58-c89wx_my-namespace>" network: add cmd:
failed to assign an IP address to container

Run the following Teradata Direct connector query through your preferred client:
```
select count(*) from myteradatadirect.sys_calendar.calendar;
```
Verify that the receivers are responding. The connector should return the result of the above query immediately. A brief check of the /tmp/starburstdata_table_operator*.log file should not display any sleep: entries, indicating that receivers could not respond or were not found:
```
2021-04-15T11:18:32 DEBUG src/main/cpp/util.cpp:211 sleep: 244
2021-04-15T11:18:32 DEBUG src/main/cpp/util.cpp:211 sleep: 500
2021-04-15T11:18:32 DEBUG src/main/cpp/util.cpp:211 sleep: 1000
```