Create an AKS cluster#

This page describes how to configure a new AKS cluster that ensures all SEP resources are co-located and follow best practices.

Warning

SEP has specific requirements for sizing, placement, and sharing of resources. You must ensure that your AKS cluster meets all requirements described in our cluster requirements section.

Prerequisites#

Ensure that you have the following tools, policies, and certificates before creating a Kubernetes cluster for SEP in AKS:

  • helm

  • kubectl

  • Azure CLI (az)

  • Azure resource group for the SEP nodes

  • Virtual network assigned for the resource group

  • IAM policies for ADLS, S3, as desired

  • CA-signed certificate for HTTPS/TLS (for a domain such as starburst.example.com) if using AD/LDAP authentication

Create your Azure cluster#

It is strongly recommended to have your SEP coordinator and workers share the same resource group. The following example az aks create command creates the sep-example cluster in the example-rg resource group:

$ az aks create --kubernetes-version <supported-version> --name sep-example --resource-group example-rg \
  --vnet-subnet-id /subscriptions/1234abcd-a1b2-c3d4-e5f6-example/resourceGroups/example-rg/providers/Microsoft.Network/virtualNetworks/example-network/subnets/default \
  --service-cidr 10.10.0.0/16 \
  --dns-service-ip 10.10.0.10 \
  --docker-bridge-address 172.16.0.1/16 \
  --location eastus \
  --zones 1 \
  --network-plugin azure \
  --node-vm-size standard_ds2_v2 \
  --enable-aad \
  --aad-admin-group-object-ids aabbccdd-1a2b-3c4d-5d6f-example \
  --assign-identity /subscriptions/1234abcd-a1b2-c3d4-e5f6-example/resourcegroups/example-rg/providers/Microsoft.ManagedIdentity/userAssignedIdentities/example-identity-policy \
  --enable-cluster-autoscaler \
  --node-count 1 \
  --min-count 1 \
  --max-count 3 \
  --nodepool-name systempool \
  --node-osdisk-size 64

You must adjust parameters such as the VM size for the nodes, the available disk space, node counts, and specific identifiers to your planned deployment.

Establish nodepools#

The best practice is to create one nodepool for your SEP coordinator and another for worker nodes. The following az aks nodepool add command creates a nodepool for a coordinator:

$ az aks nodepool add --cluster-name sep-example --resource-group example-rg \
  --name sep-coordinator \
  --labels apps=sep-coordinator \
  --node-vm-size standard_d8s_v3 \
  --eviction-policy Delete \
  --spot-max-price -1 \
  --enable-cluster-autoscaler \
  --node-count 1 \
  --min-count 1 \
  --max-count 2 \
  --node-osdisk-size 64 \
  --node-osdisk-type Ephemeral \
  --no-wait

The following command creates a scaling nodepool for a minimum of two workers:

$ az aks nodepool add --cluster-name sep-example --resource-group example-rg \
  --name sep-workers \
  --labels apps=sep-workers \
  --node-vm-size standard_d8s_v3 \
  --eviction-policy Delete \
  --spot-max-price -1 \
  --enable-cluster-autoscaler \
  --node-count 2 \
  --min-count 2 \
  --max-count 4 \
  --node-osdisk-size 64 \
  --node-osdisk-type Ephemeral \
  --no-wait

You must adjust parameters such as the VM size for the nodes, the available disk space, node counts, and specific identifiers to your planned deployment.

Configuring user-assigned managed identities with AKS#

SEP supports user-assigned managed identities via Workload Identity management to authenticate and access Azure resources like ADLS based on the permissions assigned to that identity.

Note

The following steps assume that you already created a user-assigned managed identity in Azure and the identity has adequate permissions to access a specific ADLS location.

  1. Enable workload identity management on an existing AKS cluster with the following command:

    az aks update \
--resource-group "${RESOURCE_GROUP}" \
--name "${CLUSTER_NAME}" \
--enable-oidc-issuer \
--enable-workload-identity

    Alternatively, create a new AKS cluster with workload identity management:

    az aks create \
--resource-group "${RESOURCE_GROUP}" \
--name "${CLUSTER_NAME}" \
--enable-oidc-issuer \
--enable-workload-identity \
--generate-ssh-keys

  2. Create a service account to use the user-assigned managed identity:

    kubectl apply -f -
apiVersion: v1
kind: ServiceAccount
metadata:
  annotations:
    azure.workload.identity/client-id: "${USER_ASSIGNED_CLIENT_ID}"
  name: "${SERVICE_ACCOUNT_NAME}"
  namespace: "${SERVICE_ACCOUNT_NAMESPACE}"

    If you need to fetch the user-assigned client ID, use the following command:

    export USER_ASSIGNED_CLIENT_ID="$(az identity show \
  --resource-group "${RESOURCE_GROUP}" \
  --name "${USER_ASSIGNED_IDENTITY_NAME}" \
  --query 'clientId' \
  --output tsv)"

  3. Create a federated credential to allow the identity to authenticate Kubernetes workloads:

    az identity federated-credential create \
   --name ${FEDERATED_IDENTITY_CREDENTIAL_NAME} \
   --identity-name "${USER_ASSIGNED_IDENTITY_NAME}" \
   --resource-group "${RESOURCE_GROUP}" \
   --issuer "${AKS_OIDC_ISSUER}" \
   --subject system:serviceaccount:"${SERVICE_ACCOUNT_NAMESPACE}":"${SERVICE_ACCOUNT_NAME}" \
   --audience api://AzureADTokenExchange

    If you need to fetch the OpenID Connect (OIDC) issuer, use the following command:

    export AKS_OIDC_ISSUER="$(az aks show --name "${CLUSTER_NAME}" \
 --resource-group "${RESOURCE_GROUP}" \
 --query "oidcIssuerProfile.issuerUrl" \
 --output tsv)"

  4. Deploy the Helm chart and ensure your workload uses the appropriate service account linked to the user-assigned managed identity:

    apiVersion: v1
kind: Pod
metadata:
  name: sample-workload-identity
  namespace: ${SERVICE_ACCOUNT_NAMESPACE}
  labels:
    azure.workload.identity/use: "true"  # Required. Only pods with this label can use workload identity.
spec:
  serviceAccountName: ${SERVICE_ACCOUNT_NAME}
  containers:
   - image: <image>
     name: <containerName>

For more information on creating a user-assigned managed identy, see the AKS documentation.