Tutorial
Hello World!
So you're new on your job and your boss wants you to create an API for accessing programming language code snippets.
The design and frontend teams are already working on the user interface and can't wait to access the API you're about to create.
There you go! This tutorial helps you create a REST API for "Hello World" code snippets based on FLAT.
Tools
FLAT is supplied as a Docker image, so please make sure you have Docker installed on your system.
We assume you have the curl
command line tool installed, too, which comes in handy when working with REST APIs. The jq
JSON processor is a useful tool when working with JSON.
Running the FLAT Docker image is dead easy thanks to the FLAT CLI.
Please download the script from the FLAT CLI repository, make it executable and put the script in your $PATH
:
📎 You could also put
flat
into your~/bin/
directory. If that is not already in your$PATH
, you can add it withexport PATH="$PATH:~/bin"
.
If you do not have bash
installed on your system, you can use docker-compose
as an alternative to the flat-cli. In addition to all the configuration files used in this tutorial, the repository FLAT tutorial files also provides docker-compose.yml
files to get you started. Refer to the (accompanying documentation)[https://github.com/sevenval/flat-tutorial-files/blob/main/README.md] for more information.
Getting Started
Let's create a workspace for our little project. We call it hello-world
and create a directory with that name:
Let's try to start FLAT:
Before we can start FLAT, we need an API definition. The default location is ./swagger.yaml
. To get the server up and running, an empty one will do the job:
Now you can start FLAT like this
or from within the working directory:
📎 You can choose a port other than 8080 with the
-p
option, for exampleflat start -p 8000
📎 If you haven't already guessed: You can stop FLAT at any time as usual by pressing Ctrl + C in its terminal.
Point your web browser to http://localhost:8080/ and see FLAT in action:
Or use curl
on the command line in a new terminal:
OpenAPI
Looks like an empty definition isn't all that useful after all.
We will need at least a minimal OpenAPI definition in our swagger.yaml
to avoid that error page.
Currently, FLAT supports OpenAPI 2.0 also known as Swagger:
📎 OpenAPI definitions can be written in YAML or JSON. We recommend YAML for brevity and readability.
Now we get a different error message:
We'll get rid of it in the next section.
In the Flow
The flow feature of FLAT gives us full control over request and response processing. Let's create a simple flow definition in hello.xml
with an echo
action that produces a JSON snippet:
Then we assign that flow to the path /
in our swagger.yaml
. With x-flat-flow
we tell FLAT to start the hello.xml
flow whenever /
is being requested:
📎 File paths are resolved relative to the location of the file they are referenced in. For
swagger.yaml
andhello.xml
are in the same directory, we can simply use the filename here.
There we have our first "Hello World" snippet in JSON format:
Okay, let's parametrize our requests and add a second language for "Hello World" snippets: CSS. First, we add a path parameter for the language
to the OpenAPI definition. That is, the user has to provide the desired language in the request URI:
Then in our hello.xml
we evaluate that path parameter to return the appropriate code snippet. In the flow, path parameters can be accessed via $request/params/…
. With the <if>
/<elseif>
/<else>
control structure we check the value of $request/params/language
and echo
the respective snippet as JSON:
Did you notice how we change the HTTP status code to 404 when the requested language is unsupported?
📝 Exercise: Enhance the above flow so that it can output a JavaScript code snippet as well.
💡 Hint… ```xml …{"code": "console.log('Hello World')"} … ```
Working with Files
Now we could add another language to our "Hello World" collection. And yet another, and… – Wait! Someone must have done this before, right?
Right! "Hello World" code snippets for many computer languages can of course be found on GitHub. The frontend developer team agrees that it would suffice if our API simply provides the URLs pointing to the code snippets rather than the snippets itself, for example
To give the other team something to base their work on, we'll quickly set up some mock responses. To keep our flow readable and separate concerns let's put the JSON responses into separate files:
We replace our previous flow in hello.xml
with a new one that accesses these files:
With the help of the concat
function, the copy
action assembles a filename from the requested language and loads the file's content into the $json
variable. If $json
is not empty, the dump
action then sends the content of $json
as response. Otherwise the echo
action emits an error.
Great! The frontend guys are happy that they can already request our rudimentary API while they are developing the UI for it.
Let's take another look at the code. Is it completely safe to use the $request/params/language
as we did above? Of course it's not! Using input from the client without proper sanitizing beforehand is always dangerous!
📝 Exercise: "Hack" our API to gain access to arbitrary files in your app directory, for example the
swagger.yaml
file.💡 Hint… Access to any file in our `~/hello-world` directory is possible by stuffing the whole file path into the `language` path parameter so that the filename assembled by the `concat` function above looks like `files/../swagger.yaml#.json`. Forward slashes have to be URL encoded as `%2f`, the hash tag `#` for stripping off the `.json` filename extension has to be encoded as `%23`. Instead of `%23` you could also use `%3f` for `?` or `%3b` for `;` here: ```bash $ curl http://localhost:8080/..%2fswagger.yaml%23 swagger: "2.0" info: version: "1.0" title: Hello World! … ``` Never use user input without checking it first!
Request Validation
We should fix that security threat by denying such potentially malicious requests. One simple solution would be to allow alphanumeric characters for the language parameter only. The matches
function helps with that:
Now we get:
Okay, that works! But request validation is something that is supposed to be done with the help of the OpenAPI definition. So we better put it in our swagger.yaml
:
Then we have to enable the request validation feature in the top-level section of swagger.yaml
:
Let's try again:
📎 Note how we "pipe" the response from
curl
through thejq
command to get formatted JSON output.
Having verified that the built-in error message was triggered, we can now remove our homegrown request validation from the flow.
Working with Templates
It's time to gradually move away from our local mock responses and deal with some real data from the hello-world
repository at GitHub. A suitable search request against the GitHub API for the C programming language for example looks like:
Whoa – that's a lot to take in! Most of that 22 kB JSON response is irrelevant for us. Only the html_url
field of the top-most ranked search result is the one that's interesting.
As a first step, we store that JSON as a mock for an upstream response:
In the flow, we load that mock.json
file with copy
and throw away all unwanted fields and search results with the help of a template
action:
Let's take a closer look. The copy
action has no explicit out="…"
and the following template
has no in
. That means, the latter operates on the output of copy
which is the content of our mock.json
representing the body of the upstream response. The template
action evaluates the expression between {{
and }}
and stores its result in the variable $url
as set by out
. The expression itself extracts the wanted html_url
property from the first object of the items
array (items/value[1]
). If that property does not exist, the Null Coalescing Operator ??
sets the empty string ""
as the result.
The rest of the flow is only slightly modified: We now check if we have content in $url
and create the final JSON response with a second template
. If $url
is empty, we send the error response as before.
📎 Instead of using the
$url
variable as input (in="$url"
) in the second template and operating on its content, we could have accessed$url
directly inside the template:Note that we now explicitly omit the input using
in=""
to prevent the unwanted default input from being loaded – which happens to be the huge JSON output from thecopy
action.
Here's the result:
Upstream Requests
Now let's send a real request. Thanks to FLAT that is easy: The request
action sends an HTTP request to an upstream server and provides us with the response. The upstream URL is built with concat
: As with the curl
command from above we search for files containing the word hello
in the leachim6/hello-world
repository on GitHub. Now we want the filename and the language to match our $request/params/language
parameter. For starters we simply dump
the upstream response to see what we're dealing with. We also remove the copy
action from above for now – we'll get back to it later on:
📎
dump
has no explicitin
here and just takes the output ofrequest
.
Requesting /C
should result in the same 22 kB JSON response as before – again formatted with jq
for readability:
However, requests for other languages such as /html
or /Java
are now possible, too. So let's remove that dump
action again and send another request:
Almost! We now get in fact a filtered response from GitHub, but we've missed a tiny detail: The URL points to the HTML page with the code snippet on it instead of the snippet itself.
📝 Exercise: Use another
template
action with appropriate string functions to modify$url
so that it points to the Raw view onraw.githubusercontent.com
.💡 Hint… ```xml … {{ concat('https://raw.githubusercontent.com/leachim6/hello-world', substring-after(., '/blob')) }}{"url": {{ . }}} … ``` Or, if you prefer using regular expressions: ```xml {{ replace(., "^.*/blob", "https://raw.githubusercontent.com/leachim6/hello-world") }} ```
Great! Now that we've fixed that and $url
references the right destination, we can use curl
to fetch a URL – and immediately load the respective "Hello World" code snippet afterwards:
Mock, mock!
Often, the upstream API responses we build our applications upon are not cacheable at all. Therefore, they have to be fetched over and over again in every test run during development. Downloading large responses takes some time. Especially when dealing with a small scale upstream server, for example a test system, heavy testing may result in significant load and traffic on that system.
Hence, loading a local mock file with copy
instead of waiting for a complete request/response cycle of a real remote API call can speed up development notably. So it might pay off to work with local mocks whenever possible. The following flow snippet featuring our copy
action from before illustrates how this could look like:
📎 Note that in
$request/headers
the header field names are lower-case.
Both
and
now return the URL from our local mock.json
actually pointing to the C version of "Hello World".
📎 Sending the header
Mock: true
will lead to different behavior as it enables FLAT's built-in mock feature which allows you to specify mock responses in the OpenAPI defintion.
Response Validation
To ensure that we don't break our API while we are developing it further, let's validate our responses, too. To do so, we add another schema for our own API responses to our swagger.yaml
. That schema requires that the url
property exist in the response and point to raw.githubusercontent.com
.
As before, the validation must be enabled in the top-level section:
If we now alter the domain name in the second template of our flow to flaw.githubusercontent.com
, we get a validation error:
Request Configuration
Instead of assembling the request URL with concat
, we can configure it. For better readability, we first move the request part into a sub-flow
– a separate flow file that can be called from others flows like a subroutine:
The request
action now resides in upstream_request.xml
:
We can change that into the following, more readable code by using the join
function on the individual $query_parameters
to set the query
property of our request
action:
Upstream Validation
Besides validating incoming requests and the response our API generates, we can also enable validation for our upstream request to GitHub and the associated response.
So let's enhance our request configuration in upstream_request.xml
with some validation options
:
Furthermore, we need a second OpenAPI definition file (upstream.yaml
) that describes valid upstream requests along with the expected responses:
So far, this one checks that the query string actually contains a q
parameter with an arbitrary string value.
📝 Exercise: Change the regular expression in
pattern
so that any invalid request query string is rejected with aPattern constraint violated in query for q
error in the FLAT runner log.💡 Hint… `pattern: ^hello repo:leachim6/hello-world filename:\w+ language:\w+$`
Excellent! Now the validation will ruthlessly reveal our mistakes, when we continue working on our request configuration.
Next, we'll take care of the responses: We enable validation for upstream responses
which we expect to be something like { "total_count": <integer>, "items": [ … ] }
:
Now we'd immediately notice if GitHub changed relevant parts of its API that affect our "Hello World" API built on top of it.
To abort the flow in case the upstream request or response is invalid or the request fails outright, we add the exit-on-error
request option:
To see the effect, shorten the hello-world
of the repository name to just hello
in upstream_request.xml
:
If we request our API
instead of the output
we now get
If you prefer to provide a custom error document, you can configure an error flow. Just create error.xml
:
and reference it on the top level in swagger.yaml
:
We now get
Now revert the change to upstream_request.xml
:
Improving the Configuration
Currently we send an error response if our template expression {{ items/value[1]/html_url ?? "" }}
yields an empty string. A more straightforward way would be to simply check the number of results (total_count
) that are returned:
As before, the template
for $count
operates on the result of the previous request and therefore has no in="…"
.
Now we can completely do away with the else
block if we change the above condition into 0 >= $count
as the echo
action already finishes the request:
📝 Exercise: Maybe the condition
0 >= $count
hurts your eyes, too. Swap the operands and use an appropriate comparison operator:$count … 0
Beware – we're in XML land here!💡 Hint… To keep the XML well-formed when checking `$count… … ``` Yeah – that's even worse! Less accurate, but probably sufficient may be just `$count = 0`. Or maybe `not($count > 0)`. You get the idea. Let's stick with `$count = 0`.
Next, we are going to make the lengthy concat
expression more readable by introducing template variables. That leaves us with one single template
action producing the final output:
Or, if you chose the regular expression solution above, you might prefer keeping the replace
function:
When Things Go South
Consider the following situation: After a few weeks of live operation, your manager calls and informs you that the "Hello World" API has stopped functioning. He urges you to fix it as soon as possible. Let's see for ourselves:
Sigh! So how do we find out what is going wrong?
In the default setup, the FLAT runner only logs severe errors to the log file and outputs them on the command line. If we want to get more information about what's happening, we have to change FLAT's debug settings. On startup, we can either set the environment variable $FLAT_DEBUG
or use the -d
option
📎 Mind the
'
quoting to prevent the shell from expanding*
.
Both commands start FLAT with the debug log level lowered from error
to debug
. The *
stands for any topic. log
means that debug information will be written to the log file.
📎
'*:debug:log'
is in fact the same as if we've just written'*:debug'
or even'*'
.
So when we now request our API, a lot of information from the log file will be output on the terminal where we started FLAT. We can now browse through that pile of text or use a more specific topic. For example, if we were only interested in messages related to our template
and echo
actions we could set the debug topic to template,echo
:
However, changing the default debug settings and restarting FLAT every time is not the proper way to go. Instead, we can simply send an appropriate Debug
header with our request:
If we also set the debug sink to inline
or append
, the output will be included in the HTTP response rather than the log file, for example:
The latter is the only reasonable way to debug on a production system where we usually can't access the log file. While the flat
command line tool enables header debugging by default, the appropriate environment variables must be set to enable it in other environments, typically $FLAT_DEBUG_AUTH
.
📎 If the
$FLAT_DEBUG_AUTH
environment variable is set (this is a must on production systems!), FLAT requires a password to enable debugging by means of theDebug
header, for example--header 'Debug: *:warn:append; auth=Pa5sw0rd'
.
Besides the built-in debug information, custom debug output can help understand the flow process. The debug
action enables us to insert our own log lines into the flow, for example
Configuration Files
The complete configuration files can be found in the companion repository FLAT tutorial files.
Last updated