Get best of the week

Create a TODO API for Golang with Kubernetes

Hello everyone! Ahead of the launch of the Kubernetes-based Infrastructure Platform course, we prepared a translation of another interesting material.


This article is intended for newcomers to Kubernetes, who will be interested in understanding a practical example of how to write a Golang API to manage the TODO list, and then how to deploy it to Kubernetes.

Every developer loves a good TODO list, right? How else could we organize ourselves otherwise?


Every developer loves a good TODO application, right?

We'll start by reviewing the list of necessary components, then go on to configure Kubernetes, provide the Postgresql database, and then install an application framework that will help us easily deploy the Go API in Kubernetes, without focusing on the details.

We will create two endpoints in the API - one to create a new TODO record, the other to select all TODO records.

All code from this tutorial is available on GitHub.

The list of necessary components:


  • Locally Installed Docker

Kubernetes runs code in container images, so you need to install Docker on your computer.

Install Docker: https://www.docker.com
Register a Docker Hub account to store your Docker images: https://hub.docker.com/

  • Kubernetes Cluster


You can choose either a local or remote cluster, but which one is better? Simplified options, such as k3d, work on any computer that Docker can run on, so running a local cluster no longer requires a lot of RAM. A remote cluster can also be a very efficient solution, but keep in mind that all of your Docker images will need to be uploaded and uploaded for each change.

Install k3d: https://github.com/rancher/k3d

k3d will not install kubectl (this is the CLI for Kubernetes), so install it separately from here: https://kubernetes.io/docs/tasks/tools / install-kubectl

  • Golang


You also need to install Golang with the IDE on your computer. Go is free and you can download it for MacOS, Windows or Linux from the following link: https://golang.org/dl

  • IDE


I would recommend using Visual Studio Code, it is free and has a set of plugins for Go that you can add. Some colleagues prefer Goland from Jetbrains. If you are a Java programmer, you probably would rather pay for Goland, as it will remind you of their other products.

Install VSCode or Golang .

Create a cluster


You need to install all the software specified in the previous section.

Create a new cluster using k3d:


k3d create

Configure kubectl so that it points to a new cluster. Remember that multiple clusters can be used from the same computer, therefore it is extremely important to point to the correct one:

export KUBECONFIG="$(k3d get-kubeconfig --name='k3s-default')"


Ensure that the cluster has at least one node. Here you can see that I have Kubernetes 1.17 installed - a relatively new version:

kubectl get node
NAME                     STATUS   ROLES    AGE   VERSION
k3d-k3s-default-server   Ready    master   48s   v1.17.0+k3s.1


We will store our TODO records in the database table, so now we need to install it. Postgresql is a popular relational database that we can install in a cluster using the Helm chart.

Install arkade


arkade is a CLI Go, similar to "brew" or "apt-get", but for Kubernetes applications. It uses a Helm, kubectl, or CLI project to install a project or product into your cluster.

curl -sLS https://dl.get-arkade.dev | sudo sh


Now install Postgresql

arkade install postgresql
===================================================================== = PostgreSQL has been installed.                                    =
=====================================================================


You will also see information about the connection string and how to start the Postgresql CLI through the Docker image inside the cluster.

You can get this information at any time using arkade info postgresql.

We will design the table layout

CREATE TABLE todo (
 id              INT GENERATED ALWAYS AS IDENTITY,
 description     text NOT NULL,
 created_date    timestamp NOT NULL,
 completed_date  timestamp NOT NULL
);


Run arkade info postgresqlto get connection information again. It should look something like this:

export POSTGRES_PASSWORD=$(kubectl get secret --namespace default postgresql -o jsonpath="{.data.postgresql-password}" | base64 --decode)
kubectl run postgresql-client --rm --tty -i --restart='Never' --namespace default --image docker.io/bitnami/postgresql:11.6.0-debian-9-r0 --env="PGPASSWORD=$POSTGRES_PASSWORD" --command -- psql --host postgresql -U postgres -d postgres -p 5432


Now you have at the command prompt:, postgres = #you can create the table by copying it inside, then run \dtto show the table:

postgres=# \dt
List of relations
Schema | Name | Type  |  Owner
--------+------+-------+----------
public | todo | table | postgres
(1 row)


Install application framework


Just as PHP developers accelerated their workflow using LAMP (Linux Apache + Mysql + PHP) and Rails developers using a pre-prepared stack, Kubernetes developers can use application frameworks.
The PLONK stack stands for Prometheus, Linux, OpenFaaS, NATS, and Kubernetes.
  • Prometheus provides metrics, auto-scaling, and observability to test the health of your system, allows it to respond to surges in demand and cut costs by providing metrics to make decisions about scaling to zero.
  • Linux, although not the only option for running workloads in Kubernetes, is standard and easiest to use.
  • Initially, OpenFaaS provided portable functions to developers, but it works great when deploying APIs and microservices. Its versatility means that any Docker containers with an HTTP server can be deployed and managed.
  • NATS is a popular CNCF project used for messaging and pub / sub. In the PLONK stack, it provides the ability to execute requests asynchronously and for queuing.
  • Kubernetes is the reason we are here. It provides a scalable, self-healing, and declarative infrastructure. And if you no longer need part of all this grandeur, its API is simplified with OpenFaaS.


Install the PLONK stack via arkade:

arkade install openfaas


Read the informational message and run each command:

  • Install faas-cli.
  • Get your password.
  • Redirect the OpenFaaS gateway UI (through port 8080)
  • And log in to the system through the CLI.


As before, you can receive an informational message using info openfaas.

Deploy Your First Go API


There are several ways to create a Go API using PLONK. The first option is to use the Dockerfile and manually determine the TCP port, health check, HTTP server, and so on. This can be done using faas-cli new --lang dockerfile API_NAME, but there is a simpler and more automated way.

The second way is to use the built-in templates offered by the Function Store:

faas-cli template store list | grep go
go                       openfaas           Classic Golang template
golang-http              openfaas-incubator Golang HTTP template
golang-middleware        openfaas-incubator Golang Middleware template


Since we want to create a traditional HTTP-style API, the golang-middleware template would be most appropriate.

At the beginning of the tutorial, you registered with your Docker Hub account to store your Docker images. Each Kubernetes workload must be embedded in a Docker image before it can be deployed.

Pull up the special pattern:

faas-cli template store pull golang-middleware


Use Scaffold for your API with golang-middleware and your username in the Docker Hub:

export PREFIX=alexellis2
export LANG=golang-middleware
export API_NAME=todo
faas-cli new --lang $LANG --prefix $PREFIX $API_NAME


You will see two generated files:

  • ./todo.yml - provides a way to configure, deploy, and install the template and name
  • ./todo/handler.go - here you write the code and add any other files or packages that you require


Let's do some editing and then expand the code.

package function
import (
   "net/http"
   "encoding/json"
)
type Todo struct {
   Description string `json:"description"`
}
func Handle(w http.ResponseWriter, r *http.Request) {
   todos := []Todo{}
   todos = append(todos, Todo{Description: "Run faas-cli up"})
   res, _ := json.Marshal(todos)
   w.WriteHeader(http.StatusOK)
   w.Header().Set("Content-Type", "application/json")
   w.Write([]byte(res))
}


If you do not use VScode and its plugins for editing and formatting the code, then run it after each change to make sure that the file is formatted correctly.

gofmt -w -s ./todo/handler.go


Now deploy the code - first create a new image, launch it in the Docker Hub and deploy it to the cluster using the OpenFaaS API:

Invoke your endpoint when ready:
curl http://127.0.0.1:8080/function/todo
{
"description": "Run faas-cli up"
}


Allow users to create new TODO records


Now let's allow users to create new entries in their TODO list. First you need to add the link or “dependency” for Go to the Postgresql library.

We can get it using the vending or Go modules that were introduced in Go 1.11 and installed by default in Go 1.13.

Edit handler.goand add the module that we need to access Postgresql:

import (
   "database/sql"
   _ "github.com/lib/pq"
...


In order for Go modules to detect dependencies, we need to declare something inside the file, which we will use later. If we do not, then VSCode will delete these lines when saved.

Add this under the imports in the file

var db *sql.DB


Always run these commands in the todofolder where it is located handler.go, and not at the root level c todo.yml.

Initialize the new Go module:

cd todo/
ls
handler.go
export GO111MODULE=on
go mod init


Now update the file using the pq library:

go get
go mod tidy
cat go.mod
module github.com/alexellis/todo1/todo
go 1.13
require github.com/lib/pq v1.3.0


Whatever is inside go.mod, copy its contents toGO_REPLACE.txt

cat go.mod > GO_REPLACE.txt


Now let's make sure that the assembly still works before adding the insert code.

faas-cli build -f todo.yml --build-arg GO111MODULE=on


You may notice that now we pass --build-argin to report a pattern for using Go modules.

During assembly, you will see that the modules are downloaded as needed from the Internet.

Step 16/29 : RUN go test ./... -cover
---> Running in 9a4017438500
go: downloading github.com/lib/pq v1.3.0
go: extracting github.com/lib/pq v1.3.0
go: finding github.com/lib/pq v1.3.0
?       github.com/alexellis/todo1/todo [no test files]
Removing intermediate container 9a4017438500


Configuring Secrets for Postgresql Access


We can create a connection pool in init (), a method that will only run once when the program starts.

// init       .   ,     ,   /   /

func init() {
       if _, err := os.Stat("/var/openfaas/secrets/password"); err == nil {
               password, _ := sdk.ReadSecret("password")
               user, _ := sdk.ReadSecret("username")
               host, _ := sdk.ReadSecret("host")
               dbName := os.Getenv("postgres_db")
               port := os.Getenv("postgres_port")
               sslmode := os.Getenv("postgres_sslmode")
               connStr := "postgres://" + user + ":" + password + "@" + host + ":" + port + "/" + dbName + "?sslmode=" + sslmode
var err error
               db, err = sql.Open("postgres", connStr)
               if err != nil {
                       panic(err.Error())
               }
               err = db.Ping()
               if err != nil {
                       panic(err.Error())
               }
       }
}


As you noticed, some information is extracted from what is os.Getenvread from the environment. I would consider these values ​​non-confidential, they are set in the file todo.y.

The rest, such as the password and host, which are confidential, are kept in Kubernetes secrets .

You can create them through faas-cli secret createor through kubectl create secret generic -n openfaas-fn.

The string sdk.ReadSecretcomes from OpenFaaS Cloud SDK, with the following import: github.com/openfaas/openfaas-cloud/sdk. It reads a secret file from disk and returns a value or error.

Get the secret values ​​from arkade info postgresql.

Now for each password, do the following:

export POSTGRES_PASSWORD=$(kubectl get secret --namespace default postgresql -o jsonpath="{.data.postgresql-password}" | base64 --decode)
export USERNAME="postgres"
export PASSWORD=$POSTGRES_PASSWORD
export HOST="postgresql.default"
faas-cli secret create username --from-literal $USERNAME
faas-cli secret create password --from-literal $PASSWORD
faas-cli secret create host --from-literal $HOST


Check the secrets for availability and compliance:

faas-cli secret ls
NAME
username
password
host
# And via kubectl:
kubectl get secret -n openfaas-fn
NAME                  TYPE                                  DATA   AGE
username              Opaque                                1      13s
password              Opaque                                1      13s
host                  Opaque                                1      12s


Edit our YAML file and add the following:

secrets:
   - host
   - password
   - username
   environment:
     postgres_db: postgres
     postgres_sslmode: "disable"
     postgres_port: 5432


Next, update the Go modules and run the build again:

cd todo
go get
go mod tidy
cd ..
faas-cli build -f todo.yml --build-arg GO111MODULE=on
Successfully built d2c609f8f559
Successfully tagged alexellis2/todo:latest
Image: alexellis2/todo:latest built.
[0] < Building todo done in 22.50s.
[0] Worker done.
Total build time: 22.50s


The assembly worked as expected, so let's run it faas-cliwith the same arguments to start and deploy the image. If the credentials and the SQL configuration are correct, we will not see errors in the logs, however, if they are incorrect, we will get the panic code in init ().

Check the logs:

faas-cli logs todo
2020-03-26T14:10:03Z Forking - ./handler []
2020-03-26T14:10:03Z 2020/03/26 14:10:03 Started logging stderr from function.
2020-03-26T14:10:03Z 2020/03/26 14:10:03 Started logging stdout from function.
2020-03-26T14:10:03Z 2020/03/26 14:10:03 OperationalMode: http
2020-03-26T14:10:03Z 2020/03/26 14:10:03 Timeouts: read: 10s, write: 10s hard: 10s.
2020-03-26T14:10:03Z 2020/03/26 14:10:03 Listening on port: 8080
2020-03-26T14:10:03Z 2020/03/26 14:10:03 Metrics listening on port: 8081
2020-03-26T14:10:03Z 2020/03/26 14:10:03 Writing lock-file to: /tmp/.lock


While everything looks good, now let's try calling the endpoint:

echo | faas-cli invoke todo -f todo.yml
2020-03-26T14:11:02Z 2020/03/26 14:11:02 POST / - 200 OK - ContentLength: 35


So, now we have established a successful connection to the database, and we can perform insert. How do we know that? Because it db.Ping ()returns an error, otherwise it would have thrown a panic:

err = db.Ping()
if err != nil {
   panic(err.Error())
}


Follow the link for more details on the database / sql package .

Writing the insert code


This code inserts a new row into the table todoand uses special syntax in which the value is not enclosed in quotation marks, but instead is replaced by the db.Query code. In the "old days" of LAMP programming, a common mistake that led to many systems becoming unsafe was the lack of sanitation of input data and the concatenation of user input directly into the SQL statement.

Imagine someone entering a description; drop table todoIt would not be very fun.

Therefore, we run db.Query, then pass the SQL statement using $1, $2etc. for each value, and then we can get the result and / or error. We must also close this result, so use defer for this.

func insert(description string) error {
       res, err := db.Query(`insert into todo (id, description, created_date) values (DEFAULT, $1, now());`,
               description)
       if err != nil {
               return err
       }
       defer res.Close()
       return nil
}


Now let's connect this to the code.

func Handle(w http.ResponseWriter, r *http.Request) {
       if r.Method == http.MethodPost && r.URL.Path == "/create" {
               defer r.Body.Close()
               body, _ := ioutil.ReadAll(r.Body)
               if err := insert(string(body)); err != nil {
                       http.Error(w, fmt.Sprintf("unable to insert todo: %s", err.Error()), http.StatusInternalServerError)
               }
       }
}


Let's deploy it and run it.

echo | faas-cli invoke todo -f todo.yml
curl http://127.0.0.1:8080/function/todo/create --data "faas-cli build"
curl http://127.0.0.1:8080/function/todo/create --data "faas-cli push"
curl http://127.0.0.1:8080/function/todo/create --data "faas-cli deploy"


Check the API logs:

faas-cli logs todo
2020-03-26T14:35:29Z 2020/03/26 14:35:29 POST /create - 200 OK - ContentLength: 0


Check the contents of the table using pgsql:

export POSTGRES_PASSWORD=$(kubectl get secret --namespace default postgresql -o jsonpath="{.data.postgresql-password}" | base64 --decode)
kubectl run postgresql-client --rm --tty -i --restart='Never' --namespace default --image docker.io/bitnami/postgresql:11.6.0-debian-9-r0 --env="PGPASSWORD=$POSTGRES_PASSWORD" --command -- psql --host postgresql -U postgres -d postgres -p 5432
postgres=# select * from todo;
id |   description   |        created_date        | completed_date
----+-----------------+----------------------------+----------------
1 | faas-cli build  | 2020-03-26 14:36:03.367789 |
2 | faas-cli push   | 2020-03-26 14:36:03.389656 |
3 | faas-cli deploy | 2020-03-26 14:36:03.797881 |


Congratulations, you now have a TODO API that can accept incoming requests through curlor any other HTTP client and write them to the database table.

Record Request


Let's create a new function to query TODO records from a table:

func selectTodos() ([]Todo, error) {
   var error err
   var todos []Todo
   return todos, err
}


We cannot name this select method because it is a reserved keyword for working with goroutines.

Now connect the method to the main handler:

} else if r.Method == http.MethodGet && r.URL.Path == "/list" {
   todos, err := selectTodos()
   if err != nil {
       http.Error(w, fmt.Sprintf("unable to get todos: %s", err.Error()), http.StatusInternalServerError)
   }
   out, _ := json.Marshal(todos)
   w.Header().Set("Content-Type", "application/json")
   w.Write(out)
}


Now that there are additional fields for dates in our data schema, update the Todo structure:

type Todo struct {
   ID int `json:"id"`
   Description   string `json:"description"`
   CreatedDate   *time.Time `json:"created_date"`
   CompletedDate *time.Time `json:"completed_date"`
}


Now let's add the selectTodos()request code to our method :

func selectTodos() ([]Todo, error) {
       rows, getErr := db.Query(`select id, description, created_date, completed_date from todo;`)
   if getErr != nil {
       return []Todo{}, errors.Wrap(getErr, "unable to get from todo table")
   }
   todos := []Todo{}
   defer rows.Close()
   for rows.Next() {
       result := Todo{}
       scanErr := rows.Scan(&result.ID, &result.Description, &result.CreatedDate, &result.CompletedDate)
       if scanErr != nil {
           log.Println("scan err:", scanErr)
       }
       todos = append(todos, result)
   }
   return todos, nil
}


As before, we need to delay closing the rows for the request. Each value is inserted into the new structure using the rows.Scan method. At the end of the method, we have a piece of Todo content.

Let's try:

faas-cli up -f todo.yml --build-arg GO111MODULE=on
curl http://127.0.0.1:8080/function/todo/list


Here is the result:

[
 {
   "id": 2,
   "description": "faas-cli build",
   "created_date": "2020-03-26T14:36:03.367789Z",
   "completed_date": null
 },
 {
   "id": 3,
   "description": "faas-cli push",
   "created_date": "2020-03-26T14:36:03.389656Z",
   "completed_date": null
 },
 {
   "id": 4,
   "description": "faas-cli deploy",
   "created_date": "2020-03-26T14:36:03.797881Z",
   "completed_date": null
 }
]


To remove the values, we can update the structure annotations by adding omitempty:

CompletedDate *time.Time `json:"completed_date,omitempty"`


To summarize the work done


We have not finished yet, but this is a good moment to stop and review what we have achieved so far. We:

  • Installed Go, Docker, kubectl and VSCode (IDE)
  • Deployed Kubernetes on our local computer
  • Installed Postgresql using arkade and helm3
  • Installed OpenFaaS and the PLONK stack for Kubernetes application developers
  • Created the initial static REST API using Go and golang-middlewarethe OpenFaaS template
  • Added “insert” functionality to our TODO API
  • Added select functionality to our TODO API


The full code sample that we have created so far is available on my GitHub account: alexellis / kubernetes-todo-go-app

Then we can do much more, for example:

  • Adding Authentication Using a Static Bearer Token
  • Create a web page using a static HTML or React template to render a TODO list and create new elements.
  • Adding multi-user support to the API


And much more. We could also delve into the PLONK stack and deploy the Grafana dashboard to begin to monitor our API and understand how many resources are being used with the Kubernetes dashboard or metric server (installed using arkade install).

Learn more about arkade: https://get-arkade.dev

Attend the OpenFaaS workshop to learn more about the above: https://github.com/openfaas/workshop/

You can subscribe to my premium Insiders Updates newsletter at
https: / /www.alexellis.io/
and on my twitter Alex Ellis


Learn more about the course

More articles:


All Articles