Kiyot

Kiyot is a Kubernetes CRI (container runtime interface) shim that uses Milpa, a nodeless container engine to schedule pods and containers. The most widely known CRI implementation is Docker, which is the default CRI as of today in Kubernetes.

Kubernetes CRI

Kubernetes CRI

Architecture

Kiyot makes it possible to schedule Kuberetes workloads (pods, deployments, etc) in a cloud-native manner via Milpa. When launching a pod in Kubernetes, Kiyot will direct Milpa to provision a right-sized cloud compute instance for your pod. Your pod will run on the compute instance (we call them Milpa Cells) and terminate the instance when Kubernetes stops the pod.

Under the hood, a Kiyot node uses a CRI proxy that enables our Kiyot CRI to exist alongside an existing Containerd CRI. Pods with the following annotation are directed to run in a nodeless manner via the Kiyot CRI:

kind: Pod
metadata:
  annotations:
    kubernetes.io/target-runtime: kiyot

Pods without a Kiyot runtime annotation are directed to the Containerd CRI and are run on the kubelet node. This splitting of CRI requests allows DaemonSets to run as expected (on the kubelet node) and also enables stateful workloads that are unsupported by Kiyot to run on the Kubelet node.

Kiyot Architecture

Kiyot Architecture

Installing Kiyot

Note: The Kiyot/Milpa installation process is currently streamlined for provisioning a new VPC network and kubernetes cluster inside an existing AWS account. This is useful for users who want to try out Kioyt/Milpa without integrating into an existing Kubernetes cluster. We will be publishing scripts to help in adding a nodeless worker to an existing kubernetes cluster in the near future.

To try out Elotl's Nodeless Kubernetes offering, use our terraform scripts available at www.github.com/elotl/kubeadm-aws. The install process requires the user to install terraform and the terraform AWS provider. Follow the instructions in the readme in that repo to provision a new nodeless kubernetes cluster in your AWS account.

Differences from Kubernetes with Docker

Kubernetes assumes that it launches containers, running locally on Kubelets (Kubernetes worker nodes), via the CRI. In reality, Kiyot workloads will be scheduled to run in cloud instances instead.

Since the Docker CRI (and other "local" CRIs) allows containers to interact directly with the host (i.e., the Kubelet node), there might still be a few surprises when running pods using Kiyot.

Necessary Annotation for Kiyot Pods

As stated earlier, to direct pods to run on Kiyot, you'll need to include the following annotation in your pod spec:

kind: Pod
metadata:
  annotations:
    kubernetes.io/target-runtime: kiyot

Additional Annotations for Kiyot/Milpa

Kiyot understands additional annotations that customize the type of cloud instance a pod will run on.

Launch Type

To specify a pod will be run on a spot instance, use the following annotation in the pod:

kind: Pod
metadata:
  annotations:
    pod.elotl.co/launch-type: Spot
    kubernetes.io/target-runtime: kiyot

Instance Type

To control the exact type of VM instance type that the pod will run on, use the instance-type annotation:

kind: Pod
metadata:
  annotations:
    pod.elotl.co/instance-type: p3.2xlarge
    kubernetes.io/target-runtime: kiyot

Viewing Cells Created by Kiyot/Milpa

Use kubectl get cells to list the running cells (nodes created by Milpa to run pods) in your cluster:

NAME       POD NAME                KUBELET                      LAUNCH TYPE   INSTANCE TYPE ...
f7697790   kube-proxy-76bbf9cff4   ip-10-0-31-99.ec2.internal   On-Demand     c5.large ...
44d98728   myserver                ip-10-0-31-99.ec2.internal   On-Demand     t3.nano ...

Interacting with Namespaces on the Host

Kubernetes allows privileged containers to interact with the network, pid or IPC namespaces of the host. For example, a pod might request to share the network namespace with the host:

apiVersion: v1
kind: Pod
metadata:
  name: influxdb
spec:
  hostNetwork: true
  containers:
    - name: influxdb
      image: influxdb

This is usually reserved for very specific use cases (e.g. Kubernetes networking plugins), and application pods from end users rarely use these features. When a pod runs on the VM created by Milpa/Kiyot, the pod will automatically use the host network of the VM it is launched onto.

Pulling Images

When pulling an image, the image will get downloaded to the cloud instance that gets allocated to run the pod. This has the side effect that problems with pulling the image (invalid image, authentication problems, etc) will only occur once a cloud instance has been allocated to the pod, and the cloud instance tries to pull the image.

Mounting Volumes into Containers

EmptyDirs are fully supported in Milpa and Kiyot, so containers in your pod can share data via an EmptyDir volume.

Other kinds of volumes that are mounted into containers are only partially supported currently: they are always read only, and any updates performed inside a container will not be reflected in other containers if the volume is shared across multiple containers or multiple pods.

Troubleshooting

When pods are failing to launch, kubectl can be used to get useful troubleshooting information from the cluster. Some of the common commands you might use are