Information


Blog Posts


Collections



Contact


Things Ian Says

Exploring Kubernetes with K3d: Creating a REST API

I’ve been exploring Kubernetes recently, using the container-based k3d implementation as a way to do this easily on my PC. However, before I started looking at k3d itself, I thought I’d write an initial app I could deploy onto it. This article describes creating this first app.

The first thing I did was choose a problem domain to work in, which is banking. I thought I would create microservices for customers, their contact details, their accounts, and the transactions for those accounts. Although this was probably finer grained than if I were creating such a system for real, I thought this would give me enough services to make the deployment interesting, and the chattiness would also allow me to try out some monitoring tooling.

However, to make it easy to create the data I would need, I thought I would first build a testdata service. This would give one central place to for data to originate, which would make it easy to build the relationships between the data for each service. Then, each of the actual microservices can pull their data from this testdata service, when they initialise.

So, this article describes how I built the testdata service.

Setting up the dependencies

This testdata service will serve up JSON APIs from a web server, so the first thing I need is expressjs. I also want somewhere to store the data. I could set up a full-blown database for this, but instead I’m going to take the lighter weight option of an in-memory, Javascript-based database. I’ll use TaffyDB for this. I want to populate this with synthetic test data, so I’ll need a library to generate this. I found faker.js which allowed me to create the data. Finally, I wanted to used UUIDs as my keys, so I needed a library for that - I just used a standard UUID module. After installing these dependencies, I ended up with the following package.json:

{
  "name": "banking-mocks",
  "version": "0.0.1",
  "description": "Mocks for banking services to be used in example problems",
  "main": "index.js",
  "type": "module",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1",
    "start": "node index.js"
  },
  "author": "Ian Finch",
  "license": "Apache-2.0",
  "dependencies": {
    "express": "^4.17.1",
    "faker": "^5.5.3",
    "taffydb": "^2.7.3",
    "uuid": "^8.3.2"
  }
}

Generating the synthetic data

After including the faker.js module, I set the locale to en_GB. This means, for example, that when I generate an address, I’ll get a UK-style postcode rather than a US zip code. So, that looks like this:

import faker from "faker";
faker.locale = "en_GB";

The way you use faker.js is simply to call a function to generate data for a certain type (for example a name). Because it provides so many such functions, they are grouped by area (for example name, address, vehicle, etc). As an example, to generate a random last name, you would just use:

let lastName = faker.name.lastName();

Because I want to generate a lot of data, and store it in database records, I thought it would be easiest to define objects with the field names as keys, and the functions to call as the values. Then I could write a generic function to create a record, which would just be passed the appropriate one of these structures.

As an example, here is my structure for a person record, which will end up in my customers table (or collection):

const person = {
    title: faker.name.prefix,
    firstName: faker.name.firstName,
    lastName: faker.name.lastName
};

Here is my function to create the record:

const createRecord = (recordType) =>
    Object.keys(recordType).reduce((result, field) => {
        result[field] = recordType[field]();
        return result;
    }, {});

And it is called simply like this:

let myRecord = createRecord(person);

Unwrapping the createRecord() function, the key part of the work is done by the line:

result[field] = recordType[field]();

This sets a field in the result, by finding the matching field from the recordType object and calling the function it defines. We then wrap that in a reduce() function, so that we can build up a full object, and we key the reduce() function using the keys from the passed in recordType.

However, because I want to build REST APIs, I need a unique identifier for each of the records, so I can refer to them in API calls. So, for example, for my customer record, I will have a customerId which will be a UUID.

To begin with, I will include the UUID module. If you are not familiar with UUIDs, there are various standard ways of generating them which are supported by the module I used. For this service, I will use the v4 standard, so I include the UUID module like this:

import { v4 as uuid } from "uuid";

Now I can create my customer ID like this:

let recordId = { customerId: uuid() };

I’m going to modify my createRecord() function to accept this recordId, by using it to initialise the reduce() function:

const createRecord = (recordType, initialRecord) =>
    Object.keys(recordType).reduce((result, field) => {
        result[field] = recordType[field]();
        return result;
    }, initialRecord);

Now I can create a record like this:

let myRecord = createRecord(person, { customerId: uuid() });

I’m going to add a function to make this work in a bulk way. First I’ll create a function which generates a list of UUIDs. It takes a key for the UUID and the number of UUIDs to create as parameters.

const uuidList = (key, length) =>
    Array.from({length}, () => ({[key]: uuid()}));

So now I can create 100 customer IDs, for example:

let uuids = uuidList("customerId", 100);

Once I have a list of UUIDs like this, I can pass them into my createRecord() function to create a list of synthetic records:

let uuids = uuidList("customerId", 100);
let records = uuids.map(uuidObject => createRecord(person, uuidObject));

That’s the core of my data generation, but for completeness, here are the other record type definitions:

const contactDetails = {
    street: faker.address.streetAddress,
    city: faker.address.cityName,
    county: faker.address.county,
    postCode: faker.address.zipCode,
    phone: faker.phone.phoneNumber,
    email: faker.internet.email
};

const account = {
    sortCode: () => ("" + (faker.datatype.number() * 1000 +
                           faker.datatype.number()))
                     .replace(/(..)(..)(..).*/, '$1-$2-$3'),
    accountNumber: faker.finance.account,
    branch: () => faker.address.streetName() + ", " +
                  faker.address.cityName() + ", " +
                  faker.address.zipCode(),
    balance: faker.finance.amount
};

const transaction = {
    date: faker.date.recent,
    description: faker.finance.transactionDescription
};

The sort code field is a bit messy, and I should probably separate it into its own function, but for now I’ll live with the messiness. If you want to understand it, just note that faker.datatype.number() generates a 4 or 5 digit number, so I stick a couple of them together to make sure I have enough digits for a sort code, then take the first 6 digits from that number and arrange them in the sort code structure.

Now that we can generate our data, let’s think about how to store that in our TaffyDB database.

Putting the data in a database

As I said earlier, I decided to use TaffyDB as my database, and creating a database is as simple as passing an array of objects in the constructor. So, for example, let’s suppose I have the following two records:

const records = [
  {
    "customerId": "41d5fd8a-c42d-4b21-acc2-787d4ff6688b",
    "title": "Dr.",
    "firstName": "Ima",
    "lastName": "Yost"
  },
  {
    "customerId": "cf17572d-7187-4870-bb43-858574bf45a5",
    "title": "Miss",
    "firstName": "Alexander",
    "lastName": "Swift"
  }
];

If I want to create a database with these two records, I just pass those records into the constructor:

import { taffy as TAFFY } from "taffydb";
const db = TAFFY(records);

Combining this with our data generation from the previous section, I end up with this:

import { taffy as TAFFY } from "taffydb";

let uuids = uuidList("customerId", 100);
let records = uuids.map(uuidObject => createRecord(person, uuidObject));
const db = TAFFY(records);

Let’s make this into a function, so we can use it in a more general way:

const createDb = (key, recordType, length) =>
    {
        const uuids = uuidList(key, length);
        const records = uuids.map(uuidObject =>
                                    createRecord(recordType, uuidObject));
        return TAFFY(records);
    };

So I can now create a database, populated with generated data, by calling that function like this:

let db = createDb("customerId", person, 100);

Once you’ve created a database in this way, that variable (db in the above example) is actually a function which you call to query the database. The simplest call you can make is to retrieve all records:

let allRecords = db();

We can also pass in a key/value pair and it will only return records where the supplied key in the database record has the value passed in. So, for example, if we want to find a customer with the ID 41d5fd8a-c42d-4b21-acc2-787d4ff6688b we can make the following call:

let records = db({ "customerId": "41d5fd8a-c42d-4b21-acc2-787d4ff6688b" });

Once we’ve got the result, we can call different functions on the result to convert it to a standard Javascript variable, or to perform further processing on it.

let result = db().get();                // Return an array
let result = db().stringify();          // Return a JSON string
let result = db().first();              // Return the first result
let result = db().map(x => x.lastName); // Apply a function to each result

Because I want multiple collections within my database, I decided to create a db object, and then put all my collections in that object, keyed on the collection name:

const db = {
    customers: createDb("customerId", person, 100),
    contacts: createDb("contactId", contactDetails, 100),
    accounts: createDb("accountId", account, 150),
    transactions: createDb("transactionId", transaction, 3000)
};

So, for example, to get all the customers I can make this call:

let customers = db.customers().get();

That covers off the database element, so now let’s move on to the final part — the REST API server.

Adding in the Express API server

At the top level of my API, I want to return a list of all the collections. I’ll return two elements in the result — a list of collections, and a list of links to the collections.

To get the list of collections, we can just use Object.keys() on our db object. To convert that to a list of links, we just use map() to prefix each with the server string:

import express from "express";

const app = express();
const port = 3000;
const server = `http://localhost:${port}`;

app.get("/", (req, res) => {
    res.send({
        collections: Object.keys(db),
        _links: Object.keys(db).map(x => `${server}/${x}`)
    });
});

We also to start the server to be able to do anything with it:

app.listen(port, () => {
    console.log(`Server listening at ${server}`);
});

Now we can start the server:

ian@localhost $ npm start

> banking-mocks@0.0.1 start
> node index.js

Server listening at http://localhost:3000

Now, in another window I can query the server to see the results:

ian@localhost $ curl -s http://localhost:3000/ | jq .
{
  "collections": [
    "customers",
    "contacts",
    "accounts",
    "transactions"
  ],
  "_links": [
    "http://localhost:3000/customers",
    "http://localhost:3000/contacts",
    "http://localhost:3000/accounts",
    "http://localhost:3000/transactions"
  ]
}

Now I need to be able to respond to the API calls described in this response, for example /customers. First thing to do is check to make sure the collection being requested exists. If I define the route in the get requerst as /:collection, I will be able to get its value from the request parameters at req.params.collection. So I can then just check that db[req.params.collection] exists, and if it doesn’t then I return a 404 response.

If it does exist, then I can use a database query like the ones I described earlier to get the result to return, for example db[req.params.collection]().get().

So, putting that all together, I get:

app.get("/:collection", (req, res) => {

    if (!db[req.params.collection]) {
        res.sendStatus(404);
        return;
    }

    res.send(db[req.params.collection]().get());
});

And now I can check that it works through a call like this:

ian@localhost $ curl -s http://localhost:3000/customers | jq .
[
  {
    "customerId": "6dc4635f-5c27-49ad-9c0a-8f18d77699c4",
    "title": "Mr.",
    "firstName": "Crystel",
    "lastName": "Schiller",
    "___id": "T000002R000002",
    "___s": true
  },
  {
    "customerId": "beb88a39-9c8e-4af5-b774-371fa6d0d04f",
    "title": "Dr.",
    "firstName": "Maribel",
    "lastName": "Thiel",
    "___id": "T000002R000003",
    "___s": true
  },
  ...
]

As you can see, this returns the data we put in, plus a couple of the internal fields that TaffyDB uses. I could have used the internal ID field (___id), but decided to put in my own, so I put it in UUID format. I’ll probably remove the internal fields from the result when I start to add in more services.

Put it all in Docker

Because I want to deploy this on Kubernetes, I finished off my sticking it in a container. I just grabbed the standard node docker image (the alpine version), installed the various npm packages I need as dependencies, copy my files across (the index.js I’ve just described) and run it.

That looks like this:

FROM node:alpine
WORKDIR /usr/src/app
COPY package*.json ./
RUN npm install
COPY . .
EXPOSE 3000
CMD [ "node", "index.js" ]

Then I can build the container using a command like:

docker build -t banking-testdata .

And I can run it like this:

docker run -ti -p 3000:3000 banking-testdata

Summary

Firstly, I’m aware that I haven’t done anything with Kubernetes or k3d in this article. I’ll get onto that in the next one.

What I have done is build up an API server in the following three steps:

  • Create test data using faker.js
  • Put the test data into a TaffyDB database
  • Created an expressjs server to return the data as an API

In the next article I’ll actually spin up a Kubernetes platform to deploy this API onto.

If you want to see the code discussed in this article, it’s available at ianfinch/k8s-banking-testbed (commit #1506 …) .