Deploying Kubernetes Applications on Azure Stack Edge
Based on the project experiences working with Azure Stack Edge in 2020 in this article we will share a point of view on deploying Kubernetes applications on Azure Stack Edge by Azure Arc for Kubernetes through GitOps with GitHub or GitLab.
If you are reading this article we assume you are already familiar with Azure Stack and Azure Stack Edge, in fact, you can always find more information about Azure Stack Edge here.
If interested, you can order an Azure Stack Edge device type available in your country for your organization on Azure Portal according to the current pricing. Specifically, we have got Azure Stack Edge Pro R with 24 cores, 1 GPU, 256 GB RAM and plenty of storage which allows us to deploy and run some sophisticated Artificial Intelligence (AI) apps on a single node Kubernetes cluster.
When implementing DevOps/DevSecOps and/or GitOps on the Software Development Lifecycle (SDLC) we will have to make certain technology choices. In this article we assume one of the feasible scenarios with the following details:
- App is a microservices-based containerized Artificial Intelligence (AI) app
- Code lives in a private GitHub repo or private GitLab repo
- Container images live in a private Azure Container Registry (ACR)
- App is deployed on Azure Stack Edge(s) by Azure Arc for Kubernetes through GitOps via Flux
End-to-end scenario
The app we deployed on Azure Stack Edge is an Artificial Intelligence (AI) app implementing a knowledge mining workload on the edge. The app takes advantage of Microsoft’s first-party Cognitive Services containers, containerized Open-Source Software (OSS) and purpose-built custom components which have also been containerized. All necessary container images have been stored in a private Azure Container Registry (ACR) in Azure Cloud. For the custom components we leveraged Azure DevOps pipeline(s) to build the code and package them into containers as well. There’re 4 paths depicted below:
- Yellow path depicts how the necessary infrastructure for storing container images in the Cloud has been provisioned via Terraform pipeline (YAML) and how the purpose-built custom components’ code has been built and the corresponding container images have been pushed to a private Azure Container Registry (ACR) in Azure Cloud via Code pipeline (YAML)
- Pink path depicts the activities performed to set up Azure Stack Edge device across the Cloud and Edge
- Blue path depicts the installation of Azure Arc for Kubernetes agent on Azure Stack Edge device
- Green path depicts the GitOps flow deploying your app on to Azure Stack Edge device
The following diagram illustrates the entire end-to-end sample scenario:
Noteworthy: When provisioning all the necessary resources across the Cloud and Edge for configuring your Azure Stack Edge and preparing container images for your app, many of the steps may be automated/scripted. For example, you may leverage Azure DevOps, Azure CLI and/or Azure REST API for that.
Azure Stack Edge device
Once you get Azure Stack Edge device in hand there’re detailed tutorials available on docs.microsoft.com to get you going with setting up, configuring, and activating your type of a device. Specifically, for Azure Stack Edge Pro R the tutorial can be found here.
Noteworthy: Please note that configuration management of Azure Stack Edge Pro R device is performed via local web User Interface (UI). To connect to device’s local web User Interface (UI) you configure Ethernet adapter on your computer with a specific static IP address for a specific subnet, and physically connect your computer to Ethernet PORT1 on Azure Stack Edge Pro R with a network cable directly or via a switch. Other Ethernet ports on Azure Stack Edge Pro R can be used to connect to the internet if/when necessary.
Azure Stack Edge Kubernetes cluster
Kubernetes cluster on Azure Stack Edge Pro R is created for you when you configure a compute role as a part of a device configuration. Typically, when you start working with Azure Stack Edge for implementing a specific workload, you create a custom namespace for your app(s), create a dedicated username and associate this username with the namespace created. These steps can be performed by using the following PowerShell cmdlets after you remotely connect to the cluster from your computer via kubectl as described here:
- New-HcsKubernetesNamespace
- New-HcsKubernetesUser
- Grant-HcsKubernetesNamespaceAccess
Noteworthy: Please note that the dedicated user you create per the above paragraph will have permissions to create Kubernetes resources in the associated custom namespace only. In case you need Administrative access to the Kubernetes cluster resources you will need to establish a support session to connect to the device (and there’s a separate procedure on how to retrieve EdgeSupport user password), typically, you won’t need Administrative access to deploy your apps though.
Azure Arc for Kubernetes agent on Azure Stack Edge
To enable Azure Arc for Kubernetes cluster on Azure Stack Edge Pro R you can follow the tutorial described here. Namely, among other steps the following PowerShell cmdlet will be used to install Azure Arc for Kubernetes agent on the device:
- Set-HcsKubernetesAzureArcAgent
Noteworthy: Please note that the corresponding Azure Arc for Kubernetes resource in Azure portal will be automatically created after you install Azure Arc for Kubernetes agent on the device per the above paragraph.
Code
According to the sample scenario the code will reside in a private GitHub repo or a private GitLab repo. The tutorial here describes how to deploy a sample stateless app via GitOps on Azure Arc enabled Kubernetes cluster on Azure Stack Edge. For the sake of simplicity, we will take advantage of the referenced in the tutorial public GitHub repo by Kanika Goyal: https://github.com/kagoyal/dbehaikudemo. The only difference will be that we will use a private GitHub repo (by creating a private fork of this public GitHub repo, for example) or a private GitLab repo (by importing this public GitHub repo into a private GitLab project, for example).
Noteworthy: Please note that for the code we use the namespace specified in Kubernetes objects (deployment, service) in code (yaml) will be the namespace where the corresponding pods will be deployed on Azure Stack Edge device. Practically, this will/may be the custom namespace for your app(s) you created while setting up Azure Stack Edge Kubernetes cluster.
Azure Arc for Kubernetes GitOps configuration
To deploy configurations using GitOps on Azure Arc enabled Kubernetes cluster you can follow the tutorial described here. One of the supported paths to deploy configurations using GitOps with private GitHub repo on Azure Arc enabled Kubernetes cluster is to leverage SSH keypair generated by Flux. This approach will also work with private GitLab repo and has the following steps:
- Create Azure Arc for Kubernetes GitOps configuration in Azure portal
- Retrieve the generated by Flux repository public key (in Azure portal)
- Add repository public key to GitHub or GitLab account as Deploy key
Noteworthy: When you create a new GitOps configuration it is important to note that you may have multiple choices of what you put in as Operator namespace and Operator scope. Specifically, in case you specify Operator namespace = “x” and Operator scope = “Namespace”, and the custom namespace you created for your app(s) was “y”, Flux won’t be able to create Kubernetes resources in your custom namespace “y” because of the lack of permissions. That’s why practically it may make sense to do one of the following:
- Specify Operator namespace = “x” and Operator scope = “Cluster” which would give Flux sufficient permissions to create Kubernetes resources in your custom namespace “y”
- Specify Operator namespace = “y” and Operator scope = “Namespace” which would also deploy Flux pods into your custom namespace “y” along with Kubernetes resources for your app
Deployment
Once everything is deployed via Flux you will be able to see the deployed pods in the corresponding namespace(s) in Kubernetes cluster on Azure Stack Edge via kubectl and start using your app as necessary.
Noteworthy: In case something went wrong, and the deployment has not taken place after the defined interval of time you may want to troubleshoot the error(s) by reviewing the logs of the Flux operator pod via kubectl. Please note that Flux operator and supporting memcached (and possibly Helm if enabled) pods will first be created in the Operator namespace right after Kubernetes GitOps configuration gets installed (Operator state = “Installed” in Azure portal) before Flux starts creating resources for your app(s).
Showcase
If you would like to see an example of a sophisticated Artificial Intelligence (AI) app also deployed on Azure Stack Edge using the method described above, you are welcome to watch a Channel 9 video here. Enriched Search Experience Project sample Kubernetes app on Azure Stack Edge leverages circa 30 different container images to implement a knowledge mining workload on the edge.
Disconnected mode and App sideloading
In case you work in a disconnected or air-gapped environment, you might be interested in considering a sample app sideloading procedure explained below:
The idea is to create a setup which you can leverage on-premises (for example, in a secure data center) or locally (for example, on a secure laptop), and possibly in the field without connectivity to a network (internet). In case we consider a containerized setup on a laptop, we assume that there is a direct connectivity to Azure Stack Edge (ASE) device via a network cable. For organization of the complete offline DevOps logically we would need the following components: 1) Private Repo; 2) DevOps CI/CD Pipeline(s); 3) Container Registry. Already mentioned in the above paragraphs GitLab may be deployed as a Docker container (say, on a secure laptop) or on Kubernetes (say, in a secure Azure Stack Hub (ASH) or Azure Modular Datacenter (MDC)), and may serve as a complete platform for offline DevOps providing private code repositories as (private) GitLab projects, DevOps CI/CD Pipeline(s) for building code and working with containers, and it (GitLab) integrates with Docker Container Registry. More information about offline deployments of GitLab can be found here. For the purposes of this scenario, we used GitLab Community Edition Docker image found here. However, you may also have other preferences when it comes to DevOps CI/CD Pipeline(s) and Container Registry. Thus, in addition to GitLab private code repo which contains the code for custom purpose-built components (marked as “code”) and declarative definitions of Kubernetes objects (marked as “k8s”), you may want to leverage Jenkins for DevOps CI/CD Pipeline(s) and a dedicated Docker Container Registry (v2) as your Container Registry. Jenkins can be connected to GitLab by means of Jenkins GitLab Plug-in (a suggested Plug-in) and by using GitLab API token (access token), and Jenkins can be connected to Docker Container Registry (v2) by means of Jenkins Docker Pipeline Plug-in (installed via Plug-in Manager). More information about installing Jenkins in an offline mode can be found here. For the purposes of this scenario, we used Jenkins Docker image found here. Docker Container Registry (v2) can be your local Container Registry running in a Docker container itself. More considerations for offline and air-gapped registries can be found here. For the purposes of this scenario, we used Docker Container Registry (v2) Docker image found here. For the complete offline DevOps cycle to take place you would need to connect all the containers to a local Docker network to enable internal communications. Once you have your code built and containers pushed to the Container Registry, you may sideload your app with the help of kubectl and/or helm from the terminal. It might also be beneficial to leverage Docker export/import capabilities if, for example, you choose to bring in prebuilt container images (as snapshots) on a secure media. To recap there’re 4 paths depicted above:
- Yellow path depicts how we build code from GitLab Repo and push container images to Container Registry by running Jenkins Pipeline(s)
- Pink path depicts the activities performed to set up Azure Stack Edge device across the Cloud and Edge (this is a one-time exercise performed upon the initial configuration of the device)
- Green path depicts the app sideloading on Azure Stack Edge device from a terminal on a secure laptop on a local network connected to GitLab (for Kubernetes objects definitions) and Container Registry (for custom purpose-built components Docker images)
- Blue path is an additional option which depicts the app sideloading from a terminal in one step in case you bring in Kubernetes objects definitions to a laptop using a secure media
For certain use cases it is important to be able to not only organize the complete offline DevOps flow(s) but also enable the complete offline MLOps flow(s), for example, for incremental improvements of initially deployed ML models in the field.
Noteworthy: The app sideloading procedure above has been simplified for clarity and the proper configuration of security is a subject of a separate/dedicated walkthrough (security hardening). In fact, each guide here, here and here (corresponding to GitLab, Jenkins and Docker Registry v2) covers the necessary details for their secure setup. Also, when setting up Jenkins and its Docker Pipeline Plug-in you will likely come across the challenge of running “Docker in a Docker container”. This is needed to be able to push the built images into Docker Container Registry v2 (which is also running in a container) from Jenkins Pipeline(s). On a laptop to set up a docker client inside of Jenkins container you may want to provide Jenkins access to Docker socket (/var/run/docker.sock) as described in the Using the remote Docker server guidance here. Unless set up properly you will see “docker: not found” message when running Jenkins Pipeline(s). Another challenge you may face might be with IP address and name resolution with a containerized Docker Registry v2. Should it appear, to solve this one, you may want to specify IP address explicitly using REGISTRY_HTTP_ADDR environmental variable when starting Docker container as described in the Customize the published port guidance here.
Disclaimer
Opinions expressed are solely of the author and do not express the views and opinions of author’s current employer, Microsoft.
