Photo by Maximilian Weisbecker on Unsplash

Launch Kubernetes Job on-demand with Python

I want to launch specific tasks programmatically on Kubernetes using Kubernetes Job.

How can I launch on-demand jobs on Kubernetes?

In this tutorial, I define a simple Kubernetes Job and then show two ways to make it “on-demand”.

These are the steps we will go through:

1. Define the Job
2. Working with CronJob
3. More versatility with a Python app
4. Deploy of the Python app

Before we start

Before we start, notice that I’m using Minikube on Windows, a Kubernetes engine that can run locally on my PC. So, if you decide to try on developing something that runs on Kubernetes, you should definitely check it out.

Step 1 - Define the job

A Job is a Kubernetes concept that creates one (or more) Pods, and ensures that a specified number of them successfully terminates. For more details about what a Job is and what you can do with it, I suggest you read the Kubernetes documentation link.

For the sake of this tutorial, I’m creating a simple Job, named literally ‘simple-job’.

This Job creates a Kubernetes Pod from the Docker image busybox ([it] combines tiny versions of many common UNIX utilities into a single small executable, link) adding as arguments sleep 6.

A sample yaml defining our Job is the following one:

Aside from the standard items in the file, the more interesting ones are:

• kind: Job : this lets Kubernetes understand that what we are defining with this yaml file is a Job
• restartPolicy: OnFailure : let you specify the restart policy (the Pod stays on the node but the container is re-run). The possible values are OnFailure for cases in which the process exited with a non-zero exit code (or any other reason), Never to not re-run the container.

In order to try this out, having an active Kubernetes cluster you can run: kubectl apply -f simple-job.yaml. With this command, Kubernetes obtains the Job configuration and deploys it.

The response that kubectl will give you is: job.batch/simple-job created. Now we want to check the pods with kubectl get pods:

If you now want to delete the Job, you can simply launch kubectl delete job simple-job

With the job definition, we can run it with no problem. When trying to take this Job to the next level we hit a bottleneck. How can we run the job when needed? The first approach is to use a CronJob.

Step 2 - Working with CronJobs

With CronJob we can, as the name indicates, define Jobs that run on a schedule (written in Cron format).

The same job with a CronJob can be defined as:

The schedule here defined can be paraphrased as “every minute”.

Running kubectl apply -f simple-cron-job.yaml we deploy the CronJob to Kubernetes and then running kubectl get pods we will see

To delete the CronJob simply run kubectl delete cronjob simple-cron-job.

With the use of CronJob, we find a useful tool for scheduled tasks. Instead, if you want to run a Job when a specific situation (or we can say a custom trigger) happens, we can write a very simple Python script.

Step 3 - Define the application that will run

The use of custom triggers allows to launch jobs for every situation:

• A work queue on a database
• Local queue
• API calls
• etc

In order to fully customize the trigger of Jobs I write this simple python script:

A fast recap of what the code does:

• Inside the main() function we retrieve the Kubernetes configuration. Since I am running Kubernetes on Minikube on a Windows machine, my configuration file is C:\Users\<username>\.kube\config.
• The method create_job_object() returns the Job configuration (the same that we have seen in Step 1 simple-job.yaml definition).
• create_job() and delete_job() are the ones that actually create and delete the Job.

Executing locally the file with python local_app.py and launching (always the same) kubectl get pods we get these results:

This script lets us create many jobs whenever we want. The only limitation is that this script runs on our local machine and communicates with Kubernetes through its API using the config file. Its natural evolution is to be deployed on a Kubernetes Pod.

Step 4 - Deploy the app on Kubernetes

Firstly, I made some minor changes to the code, in order to make it runnable as a Pod:

In order to make the script runnable inside Kubernetes Pod we have to:

1. Create a Docker image that runs the script
2. Create a Kubernetes Service Account, Role, and RoleBinding to give to the Pod the job creation/deletion permission (and deploy them on Kubernetes)
3. Create a Pod running the Docker container (and deploy it on Kubernetes)

We define the docker image (inside a Dockerfile) as:

I build the image using the Minikube Docker environment running @FOR /f "tokens=*" %i IN ('minikube -p minikube docker-env') DO @%i (on Windows). To have more details about it, runminikube docker-env. At the end of the returned information, there should be a command that you can launch in order to point your shell to minikube's docker.

In order to build the docker image, I move to the folder in which I have the Python script and the Dockerfile and launch docker build . -t "tutorialk8sjob" and, as logs show, the image is being built.

For the creation of Service Account, Role, and RoleBinding I followed this guide (I suggest you do the same). However, the files that we need are the following:

Lastly, what we have to write is the Pod yaml that represents our Python script on the Kubernetes cluster:

In this small piece of code, we are specifying the name of our Pod, the Docker image (tutorialk8sjob) that we want this Pod to execute and the service-account (job-robot) to be used. Also, the imagePullPolicy directive is used in order to specify that the tutorialk8sjob Docker image has to be found on the local registry. We are now ready to deploy everything and check how it is working on Kubernetes.

• kubectl apply -f service_account.yaml
• kubectl apply -f role.yaml
• kubectl apply -f rolebinding.yaml
• kubectl apply -f app.yaml

Looking to the Pods being created/deleted we see:

We start from the definition of a single Kubernetes Job, then we moved to a CronJob that lets you schedule your Job at different times of day/month/ecc. After that, we saw how to trigger manually the execution of a Job by creating/deleting it through a Python script that we deployed on a Kubernetes Pod.

One last thing

We have seen how to deploy our application one step at a time, but if you want to deploy everything in a single shot it is very easy:

• Copy all of your yaml file in a single folder
• Launch kubectl apply -f <folderName>

You see that all the yaml files inside the folder are being applied, so all your Kubernetes application is now ready to go.

All the code that we have seen is available in the public repository available on GitHub.

If you have any tips, comments or questions, do not hesitate to ask me in the comments below.