Exploring Kubernetes with K3d: Creating a REST API
I’ve been exploring Kubernetes recently, using the container-based k3d
implementation as a way to do this easily on my PC. However, before I started
looking at k3d itself, I thought I’d write an initial app I could deploy onto
it. This article describes creating this first app.
The first thing I did was choose a problem domain to work in, which is banking. I thought I would create microservices for customers, their contact details, their accounts, and the transactions for those accounts. Although this was probably finer grained than if I were creating such a system for real, I thought this would give me enough services to make the deployment interesting, and the chattiness would also allow me to try out some monitoring tooling.
However, to make it easy to create the data I would need, I thought I would
first build a testdata service. This would give one central place to for
data to originate, which would make it easy to build the relationships between
the data for each service. Then, each of the actual microservices can pull
their data from this testdata service, when they initialise.
So, this article describes how I built the testdata service.
Setting up the dependencies
This testdata service will serve up JSON APIs from a web server, so the first
thing I need is expressjs. I also want somewhere
to store the data. I could set up a full-blown database for this, but instead
I’m going to take the lighter weight option of an in-memory, Javascript-based
database. I’ll use TaffyDB for this. I want to
populate this with synthetic test data, so I’ll need a library to generate
this. I found faker.js which allowed me
to create the data. Finally, I wanted to used UUIDs as my keys, so I needed a
library for that - I just used a standard UUID
module. After installing these dependencies,
I ended up with the following package.json:
{
"name": "banking-mocks",
"version": "0.0.1",
"description": "Mocks for banking services to be used in example problems",
"main": "index.js",
"type": "module",
"scripts": {
"test": "echo \"Error: no test specified\" && exit 1",
"start": "node index.js"
},
"author": "Ian Finch",
"license": "Apache-2.0",
"dependencies": {
"express": "^4.17.1",
"faker": "^5.5.3",
"taffydb": "^2.7.3",
"uuid": "^8.3.2"
}
}
Generating the synthetic data
After including the faker.js module, I set the locale to en_GB. This means,
for example, that when I generate an address, I’ll get a UK-style postcode
rather than a US zip code. So, that looks like this:
import faker from "faker";
faker.locale = "en_GB";
The way you use faker.js is simply to call a function to generate data for a
certain type (for example a name). Because it provides so many such functions,
they are grouped by area (for example name, address, vehicle, etc). As an
example, to generate a random last name, you would just use:
let lastName = faker.name.lastName();
Because I want to generate a lot of data, and store it in database records, I thought it would be easiest to define objects with the field names as keys, and the functions to call as the values. Then I could write a generic function to create a record, which would just be passed the appropriate one of these structures.
As an example, here is my structure for a person record, which will end up in
my customers table (or collection):
const person = {
title: faker.name.prefix,
firstName: faker.name.firstName,
lastName: faker.name.lastName
};
Here is my function to create the record:
const createRecord = (recordType) =>
Object.keys(recordType).reduce((result, field) => {
result[field] = recordType[field]();
return result;
}, {});
And it is called simply like this:
let myRecord = createRecord(person);
Unwrapping the createRecord() function, the key part of the work is done by the line:
result[field] = recordType[field]();
This sets a field in the result, by finding the matching field from the
recordType object and calling the function it defines. We then wrap that in
a reduce() function, so that we can build up a full object, and we key the
reduce() function using the keys from the passed in recordType.
However, because I want to build REST APIs, I need a unique identifier for each
of the records, so I can refer to them in API calls. So, for example, for my
customer record, I will have a customerId which will be a UUID.
To begin with, I will include the UUID module. If you are not familiar with
UUIDs, there are various standard ways of generating them which are supported
by the module I used. For this service, I will use the v4 standard, so I
include the UUID module like this:
import { v4 as uuid } from "uuid";
Now I can create my customer ID like this:
let recordId = { customerId: uuid() };
I’m going to modify my createRecord() function to accept this recordId, by
using it to initialise the reduce() function:
const createRecord = (recordType, initialRecord) =>
Object.keys(recordType).reduce((result, field) => {
result[field] = recordType[field]();
return result;
}, initialRecord);
Now I can create a record like this:
let myRecord = createRecord(person, { customerId: uuid() });
I’m going to add a function to make this work in a bulk way. First I’ll create
a function which generates a list of UUIDs. It takes a key for the UUID and
the number of UUIDs to create as parameters.
const uuidList = (key, length) =>
Array.from({length}, () => ({[key]: uuid()}));
So now I can create 100 customer IDs, for example:
let uuids = uuidList("customerId", 100);
Once I have a list of UUIDs like this, I can pass them into my createRecord()
function to create a list of synthetic records:
let uuids = uuidList("customerId", 100);
let records = uuids.map(uuidObject => createRecord(person, uuidObject));
That’s the core of my data generation, but for completeness, here are the other record type definitions:
const contactDetails = {
street: faker.address.streetAddress,
city: faker.address.cityName,
county: faker.address.county,
postCode: faker.address.zipCode,
phone: faker.phone.phoneNumber,
email: faker.internet.email
};
const account = {
sortCode: () => ("" + (faker.datatype.number() * 1000 +
faker.datatype.number()))
.replace(/(..)(..)(..).*/, '$1-$2-$3'),
accountNumber: faker.finance.account,
branch: () => faker.address.streetName() + ", " +
faker.address.cityName() + ", " +
faker.address.zipCode(),
balance: faker.finance.amount
};
const transaction = {
date: faker.date.recent,
description: faker.finance.transactionDescription
};
The sort code field is a bit messy, and I should probably separate it into its
own function, but for now I’ll live with the messiness. If you want to
understand it, just note that faker.datatype.number() generates a 4 or 5
digit number, so I stick a couple of them together to make sure I have enough
digits for a sort code, then take the first 6 digits from that number and
arrange them in the sort code structure.
Now that we can generate our data, let’s think about how to store that in our
TaffyDB database.
Putting the data in a database
As I said earlier, I decided to use TaffyDB as my database, and creating a
database is as simple as passing an array of objects in the constructor. So,
for example, let’s suppose I have the following two records:
const records = [
{
"customerId": "41d5fd8a-c42d-4b21-acc2-787d4ff6688b",
"title": "Dr.",
"firstName": "Ima",
"lastName": "Yost"
},
{
"customerId": "cf17572d-7187-4870-bb43-858574bf45a5",
"title": "Miss",
"firstName": "Alexander",
"lastName": "Swift"
}
];
If I want to create a database with these two records, I just pass those records into the constructor:
import { taffy as TAFFY } from "taffydb";
const db = TAFFY(records);
Combining this with our data generation from the previous section, I end up with this:
import { taffy as TAFFY } from "taffydb";
let uuids = uuidList("customerId", 100);
let records = uuids.map(uuidObject => createRecord(person, uuidObject));
const db = TAFFY(records);
Let’s make this into a function, so we can use it in a more general way:
const createDb = (key, recordType, length) =>
{
const uuids = uuidList(key, length);
const records = uuids.map(uuidObject =>
createRecord(recordType, uuidObject));
return TAFFY(records);
};
So I can now create a database, populated with generated data, by calling that function like this:
let db = createDb("customerId", person, 100);
Once you’ve created a database in this way, that variable (db in the above
example) is actually a function which you call to query the database. The
simplest call you can make is to retrieve all records:
let allRecords = db();
We can also pass in a key/value pair and it will only return records where the
supplied key in the database record has the value passed in. So, for example,
if we want to find a customer with the ID
41d5fd8a-c42d-4b21-acc2-787d4ff6688b we can make the following call:
let records = db({ "customerId": "41d5fd8a-c42d-4b21-acc2-787d4ff6688b" });
Once we’ve got the result, we can call different functions on the result to convert it to a standard Javascript variable, or to perform further processing on it.
let result = db().get(); // Return an array
let result = db().stringify(); // Return a JSON string
let result = db().first(); // Return the first result
let result = db().map(x => x.lastName); // Apply a function to each result
Because I want multiple collections within my database, I decided to create a
db object, and then put all my collections in that object, keyed on the
collection name:
const db = {
customers: createDb("customerId", person, 100),
contacts: createDb("contactId", contactDetails, 100),
accounts: createDb("accountId", account, 150),
transactions: createDb("transactionId", transaction, 3000)
};
So, for example, to get all the customers I can make this call:
let customers = db.customers().get();
That covers off the database element, so now let’s move on to the final part — the REST API server.
Adding in the Express API server
At the top level of my API, I want to return a list of all the collections. I’ll return two elements in the result — a list of collections, and a list of links to the collections.
To get the list of collections, we can just use Object.keys() on our db
object. To convert that to a list of links, we just use map() to prefix each
with the server string:
import express from "express";
const app = express();
const port = 3000;
const server = `http://localhost:${port}`;
app.get("/", (req, res) => {
res.send({
collections: Object.keys(db),
_links: Object.keys(db).map(x => `${server}/${x}`)
});
});
We also to start the server to be able to do anything with it:
app.listen(port, () => {
console.log(`Server listening at ${server}`);
});
Now we can start the server:
ian@localhost $ npm start
> banking-mocks@0.0.1 start
> node index.js
Server listening at http://localhost:3000
Now, in another window I can query the server to see the results:
ian@localhost $ curl -s http://localhost:3000/ | jq .
{
"collections": [
"customers",
"contacts",
"accounts",
"transactions"
],
"_links": [
"http://localhost:3000/customers",
"http://localhost:3000/contacts",
"http://localhost:3000/accounts",
"http://localhost:3000/transactions"
]
}
Now I need to be able to respond to the API calls described in this response,
for example /customers. First thing to do is check to make sure the
collection being requested exists. If I define the route in the get requerst
as /:collection, I will be able to get its value from the request parameters
at req.params.collection. So I can then just check that
db[req.params.collection] exists, and if it doesn’t then I return a 404
response.
If it does exist, then I can use a database query like the ones I described
earlier to get the result to return, for example
db[req.params.collection]().get().
So, putting that all together, I get:
app.get("/:collection", (req, res) => {
if (!db[req.params.collection]) {
res.sendStatus(404);
return;
}
res.send(db[req.params.collection]().get());
});
And now I can check that it works through a call like this:
ian@localhost $ curl -s http://localhost:3000/customers | jq .
[
{
"customerId": "6dc4635f-5c27-49ad-9c0a-8f18d77699c4",
"title": "Mr.",
"firstName": "Crystel",
"lastName": "Schiller",
"___id": "T000002R000002",
"___s": true
},
{
"customerId": "beb88a39-9c8e-4af5-b774-371fa6d0d04f",
"title": "Dr.",
"firstName": "Maribel",
"lastName": "Thiel",
"___id": "T000002R000003",
"___s": true
},
...
]
As you can see, this returns the data we put in, plus a couple of the internal
fields that TaffyDB uses. I could have used the internal ID field (___id),
but decided to put in my own, so I put it in UUID format. I’ll probably remove
the internal fields from the result when I start to add in more services.
Put it all in Docker
Because I want to deploy this on Kubernetes, I finished off my sticking it in a
container. I just grabbed the standard node docker image (the alpine
version), installed the various npm packages I need as dependencies, copy my
files across (the index.js I’ve just described) and run it.
That looks like this:
FROM node:alpine
WORKDIR /usr/src/app
COPY package*.json ./
RUN npm install
COPY . .
EXPOSE 3000
CMD [ "node", "index.js" ]
Then I can build the container using a command like:
docker build -t banking-testdata .
And I can run it like this:
docker run -ti -p 3000:3000 banking-testdata
Summary
Firstly, I’m aware that I haven’t done anything with Kubernetes or k3d in this article. I’ll get onto that in the next one.
What I have done is build up an API server in the following three steps:
- Create test data using
faker.js - Put the test data into a
TaffyDBdatabase - Created an
expressjsserver to return the data as an API
In the next article I’ll actually spin up a Kubernetes platform to deploy this API onto.
If you want to see the code discussed in this article, it’s available at ianfinch/k8s-banking-testbed (commit #1506 …) .