A Lightweight OCaml Webapp Tutorial

This tutorial aims to guide readers familiar with OCaml along one course to a backend for a webapp. The app is lightweight in that it doesn’t take much code to define and in that it probably shouldn’t be used for any heavy, industrial applications.

The app implements two kinds of functionality:

  1. An embellished echo server, responding to path parameters
  2. An interface to a rudimentary database of author excerpts

The tutorial covers the following topics:

(The OCaml ecosystem also has great support for the frontend, thanks to Js_of_ocaml and Bucklescript, but we won’t be covering that here.)

Feedback, improvements, and corrections are most welcome!

Table of Contents

1 Tips on Reading this Tutorial

  • This is a more “advanced” tutorial, covering a wide range of functionality. For a very basic intro using a simple echo server, see the README.md in the Opium repo.
  • All file paths are given relative to the project’s root directory, unless explicitly noted otherwise.
  • This repository contains the complete code for the app. The tutorial consists of a tour of the source code, with some notes on setup. So I encourage you to clone the app and play with the code as you read.
  • For quick reference to code examples, see the Index of Listings
  • Skip or skim “Explanation of the code” sections if the code is clear to you.
  • Skip to the sections that are useful to you.

2 Clone This Repository

The best way to read this tutorial is to clone the repository

git clone https://gitlab.com/shonfeder/ocaml_webapp

And play with the code as you read along.

(If you’re org-mode user, you may even prefer to just read the ./tutorial/tutorial.org file locally.)

3 Setup

3.1 Dependencies

We will be using the following OCaml libraries to build the app:

  • Opium: A Sinatra-like web toolkit
  • Caqti: A library for interfacing with PostgreSQL
  • ppx_rapper: A syntax extension to ease writing sql queries for Caqti
  • TyXML: A library for generating correct HTML
  • Logs: A logging library
  • Lwt: A concurency library
  • Core: A (standard) standard library alternative

There are many other excellent and powerful web libraries we won’t be using. See the OCamlverse page on Web and Networking for a catalog.

3.2 Prerequisites

This section is for readers who don’t currently have a well configured OCaml setup in place. Skip to Configuration if you’re already setup with the modern suite of tools.

3.2.1 Install and initialize opam

opam is the OCaml package manager.

To install opam, run

sh <(curl -sL https://raw.githubusercontent.com/ocaml/opam/master/shell/install.sh)

or choose another method.

Initialize opam and build your main working environment with

opam init
# Install any dependencies called for
# Answer yes to the prompts to configure your shell environment

opam switch 4.08.1
# Sets the system ocaml compiler to use ocaml 4.08.1
# Use a different version at your own discretion

3.2.2 Install dune

Dune is the standard build system for OCaml. We’ll need it to initialize our project below.

opam install dune

3.2.3 Install editor support and initialize your configurations

For the basic tooling, run

opam install user-setup merlin ocamlformat ocp-indent
opam user-setup install
  1. Emacs

    Install the OCaml mode for emacs

    opam install tuareg
    

    If you aren’t already an emacs user but want to give it a spin, I highly recommend the spacemacs distribution for a beginner friendly approach.

    I use doom-emacs, and find the IDE experience phenomenal.

  2. For Vim or VSCode

3.2.4 Install PostgreSQL

This tutorial currently1 uses postgres, and you’ll need postgres installed to run the app successfully.

If you don’t already have postgres set up, see the installation instructions from postgresql.org and/or the docs on installing postgres for your distro or OS.

Make sure you also have configured your user with access to create and update databases. The Arch wiki has a short section on creating your first database and user.

3.3 Configuration

This section is for users who aren’t very familiar with configuring an OCaml project. Skip to Create an opam Switch if you’re familiar with wrangling dune and opam.

We’ll walk through the configuration2 of the project in this section. However, if you cloned the repository, as advised, I encourage you to skip to the next section and then refer back here later as needed, and for reference when you’re making your own project.

3.3.1 Project initialization

The basic skeleton was generated with the dune init subcommand:

dune init proj ocaml_webapp \
    --libs core,opium,caqti,caqti-driver-postgresql,caqti-lwt,tyxml,lwt.unix \
    --ppx ppx_rapper

This creates a new directory named ocaml_webapp, with the following subdirectories corresponding to self-contained components:

  • lib: The library component, where all the business logic should live
  • bin: The executable component, this will be a client of our lib
  • test: You can guess (we won’t be using this in our tutorial)

Each directory has it’s own dune file configuring the component, (which includes a libraries stanza declaring the component’s dependencies).

3.3.2 Configuration of the package and its dependencies

We use dune to generate the opam package configuration. We always want to configure a package for our project so that dependencies are managed via configuration as code throughout.

The package is configured in ./dune-project:

(lang dune 2.0)
(generate_opam_files true)

(name ocaml_webapp)
(version 0.1.0)

(authors "Shon Feder")
(license MIT) ; TODO
(source (uri "git+https://gitlab.com/shonfeder/ocaml_webapp.git")) ; TODO
(maintainers "shon.feder@gmail.com") ; TODO
(homepage "https://gitlab.com/shonfeder/ocaml_webapp") ; TODO
(bug_reports "https://gitlab.com/shonfeder/ocaml_webapp/issues") ; TODO
(documentation "https://shonfeder.gitlab.io/ocaml_webapp/") ; TODO

(package
 (name ocaml_webapp)
 (synopsis "A minimal example of a lightweight webapp in OCaml") ; TODO
 (description "A minimal webapp using Opium, Catqi, and tyXML") ; TODO
 (depends
  ;; General system dependencies
  (dune (>= 2))
  (ocaml (>= 4.08.0))

  ;; Standard library replacement
  (core (>= v0.12.2))

  ;; Web toolkit
  (opium (>= 0.17.1))

  ;; Database interface
  (caqti (>= 1.2.1))
  (caqti-lwt (>= 1.2.0))
  (caqti-driver-postgresql (>= 1.2.1))
  (ppx_rapper (>= 0.9.2))

  ;; HTML generation
  (tyxml (>= 4.3.0))

  ;; Logging
  (logs (>= 0.7.0))

  ;; Dev dependencies
  (utop :dev)
  (merlin :dev)
  (ocamlformat :dev)
  (ocp-indent :dev)
  (tuareg :dev) ;; rm if not using emacs
))

The fields that you should customize in your own app are marked with TODOs, and their meaning should be straightforward. See the dune docs if you want more details on this configuration.

3.3.3 The Makefile

These make rules help to expedite and standardize some common development tasks and commands. (If you prefer to manage these manually, you can use the makefile as a recipe book.)

# The name of your project (as given in the dune-project file)
# TODO
project_name = ocaml_webapp

# The opam package configuration file
opam_file = $(project_name).opam

.PHONY: deps run run-debug migrate rollback

# Alis to update the opam file and install the needed deps
deps: $(opam_file)

# Build and run the app
run:
    dune exec $(project_name)

# Build and run the app with Opium's internal debug messages visible
run-debug:
    dune exec $(project_name) -- --debug

# Run the database migrations defined in migrate/migrate.ml
migrate:
    dune exec migrate_ocaml_webapp

# Run the database rollback defined in migrate/rollback.ml
rollback:
    dune exec rollback_ocaml_webapp

# Update the package dependencies when new deps are added to dune-project
$(opam_file): dune-project
    -dune build @install        # Update the $(project_name).opam file
    -git add $(opam_file)       # opam uses the state of master for it updates
    -git commit $(opam_file) -m "Updating package dependencies"
    opam install . --deps-only  # Install the new dependencies

3.4 Create an opam Switch

An opam “switch” is a sandboxed ocaml environment.

3.4.1 A fresh opam switch for the project

Create a new opam switch for the project with:

# From within the project root directory
opam switch create . 4.08.1 --deps-only
# I recommend 4.08.1 because all basic tools are compatible with it as of 2019/12/30
# if this should need updating, please make an issue or PR

Developing projects in their own switch helps to ensure smoother deployment and guards against dependency conflicts arising between unrelated projects on your local system.

To verify that the switch has been created and that you are now working in the sandbox, you can run

opam switch
# →/home/you/path/to/ocaml_webapp  ocaml-base-compiler.4.08.1  /home/you/path/to/ocaml_webapp

If you do not see the switch associated with your project directory indicated by a , then something has gone wrong.

3.4.2 Install dune in the switch

We installed dune previously at the system level, but we’ll need it in our project sandbox too:

opam install dune

3.4.3 Install and update the project dependencies

Run

make deps
# install all dependencies

(If you opted not to use the suggested Makefile, manually execute the deps rule defined in the Makefile)

NOTE: This assumes you are working inside a git repository If you did not clone this repo, then make sure to git init prior to installing the deps.

3.4.4 Confirm that the app skeleton can build

dune build

If no errors are reported, then setup is complete!

3.5 Create the Postgres database

Assuming you have postgresql installed, you can use the createdb utility to create a new database for the app:

createdb ocaml_webapp
# Replace 'ocaml_webapp' with your app's name if you're working with custom code

Then run the database migration to create the needed tables

make migrate

3.5.1 Troubleshooting problems with the database port

The app is configured to look for postgres at the localhost ip on its default port of 5432. If you have trouble connecting, ensure that the database is configured to match this configuration, or update the configuration in ./lib/db.ml to match your postgres setup:

11: (** Configuration of the connection *)
12: let url = "localhost"
13: let port = 5432
14: let database = "ocaml_webapp"

4 Tour of the Application

4.1 The Executable

We’ll define our app’s main executable entrypoint in the aptly named ./bin/main.ml:

 1: open Opium.Std
 2: 
 3: open Ocaml_webapp
 4: 
 5: let static =
 6:   Middleware.static
 7:     ~local_path:"./static"
 8:     ~uri_prefix:"/static"
 9:     ()
10: 
11: (** Build the Opium app  *)
12: let app : Opium.App.t =
13:   App.empty
14:   |> App.cmd_name "Ocaml Webapp Tutorial"
15:   |> middleware static
16:   |> Db.middleware
17:   |> Route.add_routes
18: 
19: let log_level = Some Logs.Debug
20: 
21: (** Configure the logger *)
22: let set_logger () =
23:   Logs.set_reporter (Logs_fmt.reporter ());
24:   Logs.set_level log_level
25: 
26: (** Run the application *)
27: let () =
28:   set_logger ();
29:   (* run_command' generates a CLI that configures a deferred run of the app *)
30:   match App.run_command' app with
31:   (* The deferred unit signals the deferred execution of the app *)
32:   | `Ok (app : unit Lwt.t ) -> Lwt_main.run app
33:   | `Error                  -> exit 1
34:   | `Not_running            -> exit 0

Run the app with

make run

and visit http://localhost:3000/.

4.1.1 Explanation of the code

From the bottom up…

  1. Execution

    The first thing we do is set up logging with the set_logger function3. Doing this first ensures that any important log messages that may be generated during subsequent actions will be recorded.

    App.run_command' app generates a commandline interface for launching our application app. It returns a value of type unit Lwt.t, signifying the deferred result of running the program.

    You can read the docs for the generated command line interface by running

    dune exec ocaml_webapp -- --help
    

    We pass the deferred result to Lwt_main.run, which kicks of the Lwt scheduler, and runs our app (see the Lwt_main docs for more info).

  2. Setting the logger

    The set_logger function itself merely uses the default logs log reporter. It sets the log_level, which we’ve hard coded to Logs.Debug. We use the Lwt binding operator to sequence our actions here, just as we do in the run function.

    Note that for this logging configuration to work, we rely on the library dependencies logs, logs.fmt, and logs.lwt declared in ./bin/dune:

     9:    logs         ;; The logs library
    10:    logs.fmt     ;; The logs format helper
    11:    logs.lwt     ;; The logs concurrency helper
    

    To see the debug logging output (very useful for tracking down routing problems) run the app with

    make run-debug
    

    This will output all the incoming and outgoing HTTP requests to your terminal.

    NOTE: The --debug flag used in the invocation of Opium here only affects the internal logging from the Opium library. It is the log_level we set that effects the output level from the logger.

  3. Building the app

    let app = ... defines our application. Applications are built up by pushing a base App.empty through a pipeline of functions of type App.builder. App.builder is just a synonym for functions of type App.t -> App.t, which take in an application, add something to it, and pass along the new application. In this case, we build the app with the follow steps:

    1. Set the name for the command line interface with App.cmd_name
    2. Add the middleware to serve static files
    3. Add the database connection to the environment with the Db.middleware function
    4. Add all of the route handlers with Route.add_routes.

    The helper static just configures a piece of middleware to serve static files, using Opium’s Middleware.static. It serves the contents of the ./static directory at the "/static" route.

4.2 Simple Route Handlers

The Route.add_routes of ./bin/main.ml adds all of our route handlers to the app. This is defined in ./lib/route.ml:

72: let routes =
73:   [ root
74:   ; hello
75:   ; hello_fallback
76:   ; excerpts
77:   ; get_excerpts_add
78:   ; post_excerpts_add
79:   ; excerpts_by_author
80:   ; excerpts
81:   ]
82: 
83: let add_routes app =
84:   Core.List.fold ~f:(fun app route -> route app) ~init:app routes

All of our routes are defined as App.builder functions, and listed in the routes list.

Let’s look at the simplest handlers, that just serve a basic page and act as an echo server:

 1: open Opium.Std
 2: (* HTML generation library *)
 3: open Tyxml
 4: 
 5: (** The route handlers for our app *)
 6: 
 7: (** Defines a handler that replies to GET requests at the root endpoint *)
 8: let root = get "/"
 9:     begin fun _req ->
10:       respond' @@
11:       Content.welcome_page
12:     end
13: 
14: (** Defines a handler that takes a path parameter from the route *)
15: let hello = get "/hello/:lang"
16:     begin fun req ->
17:       let lang = param req "lang" in
18:       respond' @@
19:       Content.hello_page lang
20:     end
21: 
22: (** Fallback handler in case the endpoint is called without a language parameter *)
23: let hello_fallback = get "/hello"
24:     begin fun _req ->
25:       respond' @@
26:       Content.basic_page Html.[p [txt "Hiya"]]
27:     end

4.2.1 Explanation of the code

  1. Responding to requests

    Each function is defined in terms of a route combinator that takes a string encoding the route and a handler function. The handler function takes an incoming request (req) to an outgoing response.

    The root and hello_fallback handlers don’t use the request. Instead, they use respond’ to reply with the relevant page defined in the Content module. The ' on the respond' function indicates that the respone is a deferred Lwt.t value. We’ll use this for all our respones, and Lwt will handle all the concurrency for us.

    Note that the hello_fallback handler offers the first appearance of HTML generation: Html.[p [txt "Hiya"]] invokes the p and txt functions from the Tyxml’s Html module to generate a paragraph HTML element holding the text "Hiya". We will explore this deeper in the section on Content Generation.

  2. Handling path parameters

    The hello function handles routes that include a segment after the "hello". The :lang component of the path will cause a "lang" parameter to be added to the req. This parameter will hold a string recording the value of this segment of the path for any incoming request that matches the pattern. "/hello/Hindi" matches this pattern but "/hello/foo/bar" and "/hello" do not.

    Here, we just pass the lang parameter on to our page generation function Content.hello_page.

4.3 Content Generation

All of the route handlers call functions from the Content module. This is where we’ve collected all the functions responsible for generating HTML. I’ll walk through some highlights of the code, but see the Tyxml documentation for more information and instruction on how to use it’s Html module (there is also a ppx syntax extension letting you write HTML with embedded OCaml).

There are other options for HTML templating, so if you prefer, you may want to check out ocaml-mustache or ecaml.

4.3.1 Basics of Tyxml as Seen in the <head> Element

For the sake of a deeper dive into a short piece of HTML generating code, consider the head element we define for use in all the following content:

 1: open Core
 2: open Tyxml
 3: 
 4: (** A <head> component shared by all pages *)
 5: let default_head =
 6:   let open Html in
 7:   head
 8:     (title (txt "OCaml Webapp Tutorial"))
 9:     [ meta ~a:[a_charset "UTF-8"] ()
10:     ; link ~rel:[`Stylesheet] ~href:"/static/style.css" () ]
  1. Explanation of the code

    Tyxml.Html aims to provide type-safe combinators for every valid HTML element. The name of the combinator is generally the same as the tag for the element itself. Most combinators take an optional named ~a argument that accepts a list of attributes, and a required argument with a list of values of type Tyxml.Html.elt which specify the element’s inner HTML.

    In the default_head function, we see head, which generates <head>...</head> elements. It is an exception to the general pattern of element combinators: since every well formed head element should have a title, head takes a required title argument, and then the usual list of other elements. The title element itself takes a terminal txt expression.

    meta takes a () instead of a list of elements. This is because it is a terminal element. The meta element shows an example of setting attributes: Txyml.Html attributes combinators all begin with the a_ prefix. In this case, we declare the charset attribute for our HTML documents.

    The above code generates the following HTML:

    <head>
      <title>OCaml Webapp Tutorial</title>
      <meta charset="UTF-8"/>
      <link rel="stylesheet" href="/static/style.css"/>
    </head>
    

4.3.2 Generating HTML for Use by Opium

opium has no understanding of Tyxml: when we reply with HTML, opium just expects the HTML to be represented as a string. Thus, we’ll need to render Tyxml’s well-typed HTML AST into a string before responding to requests. In the following code, we do that conversation in the basic_page function:

11: (** The basic page layout, emitted as an [`Html string] which Opium can use as a
12:     response *)
13: let basic_page content =
14:   let raw_html =
15:     let open Html in
16:     html default_head (body content)
17:     |> Format.asprintf "%a" (Html.pp ())
18:   in
19:   `Html raw_html
  1. Explanation of the code

    This function is responsible for the following preparation of its content:

    1. Wrap the content in a <body> tag
    2. Construct an html element with the default_head
    3. Serialize the AST of type _ Tyxml.Html.elt into a well formatted string
    4. Wrap the raw_html in the `Html polymorphic variant tag expected by opium

    All the HTML we construct for our app will ultimately go trough basic_page prior to becoming part of an opium response.

4.3.3 Parametric Templates

Let’s look at a template which takes a parameter, and see how this parameter is threaded through from the route handler.

Consider the hello handler again

14: (** Defines a handler that takes a path parameter from the route *)
15: let hello = get "/hello/:lang"
16:     begin fun req ->
17:       let lang = param req "lang" in
18:       respond' @@
19:       Content.hello_page lang
20:     end

It retrieves the value supplied as a path parameter in the :lang position from the request, req, and forwards it to the Content.hello_page function. This function will then generate our HTML, which we send back as our response via respond'

Content.hello_page simply pattern matches on the language supplied and generates a page with an appropriate greeting:

41: let hello_page lang =
42:   let greeting = match lang with
43:     | "中文"    -> "你好,世界!"
44:     | "Deutsch" -> "Hallo, Welt!"
45:     | "English" -> "Hello, World!"
46:     | _         -> "Language not supported :(\nYou can add a language via PR to https://gitlab.com/shonfeder/ocaml_webapp"
47:   in
48:   basic_page Html.[p [txt greeting]]

4.3.4 Forms that Submit Post Requests

Finally, let’s see how we can generate a form that will submit a post request.

The add_excerpt_page generates a page serving a simple form that collects information describing an excerpt from a book, article, or other source:

50: let add_excerpt_page =
51:   let txt_input name =
52:     Html.[ label ~a:[a_label_for name] [txt (String.capitalize name)]
53:          ; input ~a:[a_input_type `Text; a_name name] () ]
54:   in
55:   let excerpt_input =
56:     let name = "excerpt" in
57:     Html.[ label ~a:[a_label_for name] [txt (String.capitalize name)]
58:          ; textarea ~a:[a_name name] (txt "") ]
59:   in
60:   let submit =
61:     Html.[input ~a:[ a_input_type `Submit; a_value "Submit"] () ]
62:   in
63:   basic_page
64:     Html.[ form ~a:[a_method `Post; a_action "/excerpts/add"]
65:              (List.map ~f:p
66:                 [ txt_input "author"
67:                 ; excerpt_input
68:                 ; txt_input "source"
69:                 ; txt_input "page"
70:                 ; submit
71:                 ])]

Lines 51 through 62 define self-documenting helpers that we use to build up the parts of the form. We construct the page with the form using the named attribute argument a to configure the form to send a `Post request to our /excerpts/add endpoint.

We serve the add_excerpt_page in response to GET requests to /excerpts/add:

29: let get_excerpts_add = get "/excerpts/add" begin fun _req ->
30:     respond' @@
31:     Content.add_excerpt_page
32:   end

But we respond to POST requests at /excerpts/add via a different handler. Examining this handler will lead us into our interactions with the database, so before discussing how to use the Db api in a post route, we’ll walk through the database code.

4.4 Interfacing with the Database

The database interface is entirely encapsulated within the ./lib/db.ml module. Let’s review the API defined in ./lib/db.mli:

 1: open Opium.Std
 2: 
 3: (** {{1} Type aliases for clearer documentation and explication} *)
 4: 
 5: type 'err caqti_conn_pool =
 6:   (Caqti_lwt.connection, [> Caqti_error.connect] as 'err) Caqti_lwt.Pool.t
 7: 
 8: type ('res, 'err) query =
 9:   Caqti_lwt.connection -> ('res, [< Caqti_error.t ] as 'err) result Lwt.t
10: 
11: (** {{1} API for the Opium app database middleware }*)
12: 
13: val middleware : App.builder
14: (** [middleware app] equips the [app] with the database pool needed by the
15:     functions in [Update] and [Get]. It cannot (and should not) be accessed
16:     except through the API in this module. *)
17: 
18: module Get : sig
19:   (** Execute queries that fetch from the database *)
20: 
21:   val excerpts_by_author : string -> Request.t -> (Excerpt.t list, string) Lwt_result.t
22:   (** [excerpts_by_author author req] is the [Ok excerpts] list of all the
23:       [excerpts] by the [author], if it succeeds. *)
24: 
25:   val authors : Request.t -> (string list, string) Lwt_result.t
26:   (** [authors req] is the [Ok authors] list of all the [authors] in the
27:       database, if it succeeds. *)
28: 
29: end
30: 
31: module Update : sig
32:   (** Execute queries that update the database *)
33: 
34:   val add_excerpt : Excerpt.t -> Request.t -> (unit, string) Lwt_result.t
35:   (** [add_excerpt excerpt req] is [Ok ()] if the new excerpt can be inserted
36:       into the database. *)
37: 
38: end
39: 
40: (** {{1} API for database migrations } *)
41: 
42: module Migration : sig
43:   (** Interface for executing database migrations *)
44: 
45:   type 'a migration_error =
46:     [< Caqti_error.t > `Connect_failed `Connect_rejected `Post_connect ] as 'a
47: 
48:   type 'a migration_operation =
49:     Caqti_lwt.connection -> unit -> (unit, 'a migration_error) result Lwt.t
50: 
51:   type 'a migration_step = string * 'a migration_operation
52: 
53:   val execute : _ migration_step list -> (unit, string) result Lwt.t
54:   (** [execute steps] is [Ok ()] if all the migration tasks in [steps] can be
55:       executed or [Error err] where [err] explains the reason for failure. *)
56: 
57: end

This API encapsulates all the database operations so that no code in the rest of the app can read from or write to the database.

Now let’s walk through the implementation.

4.4.1 Implementing the Database Connection

To implement the database connection, we only need to construct the connection URI and feed it to Caqti. We use Caqti_lwt to create a pool of threads that can access this connection:

11: (** Configuration of the connection *)
12: let url = "localhost"
13: let port = 5432
14: let database = "ocaml_webapp"
15: let connection_uri = Printf.sprintf "postgresql://%s:%i/%s" url port database
16: 
17: (* [connection ()] establishes a live database connection and is a pool of
18:    concurrent threads for accessing that connection. *)
19: let connect () =
20:   connection_uri
21:   |> Uri.of_string
22:   |> Caqti_lwt.connect_pool ~max_size:10
23:   |> function | Ok pool   -> pool
24:               | Error err -> failwith (Caqti_error.show err)
25: 
26: (* [query_pool query pool] is the [Ok res] of the [res] obtained by executing
27:    the database [query], or else the [Error err] reporting the error causing
28:    the query to fail. *)
29: let query_pool query pool =
30:   Caqti_lwt.Pool.use query pool
  1. Explanation of the code

    The purpose of the connect function is largely to seal in the gnarly, polymorphic variant error type used by Caqti, so that the rest of the app doesn’t need to be aware of it. We map the error type to a string representation of it, but a productionized app you’d likely want to create a more robust error handler. In a production app, we’d want more robust error handling than just throwing failwith (probably using a result type). The same goes for the err handling in the query_pool helper:

    NOTE: The configuration variables hard code our database connection parameters into the source, which is expedient for this tutorial, but not a good practice for a real project, where we’d want to take these out of a configuration file or environment variable.

4.4.2 Implementing the database middleware

We need to make the database connection available at any route, so we can query and update it as needed. This is achieved by defining a piece of Opium middleware that initiates the connection pool and adds it to the environment of our opium app:

40: (* Seal the key type with a non-exported type, so the pool cannot be retrieved
41:    outside of this module *)
42: type 'err db_pool = 'err caqti_conn_pool
43: let key : _ db_pool Opium.Hmap.key = Opium.Hmap.Key.create ("db pool" , fun _ -> sexp_of_string "db_pool")
44: 
45: (* Initiate a connection pool and add it to the app environment *)
46: let middleware app =
47:   let pool = connect ()
48:   in
49:   let filter handler (req : Request.t) =
50:     let env = Opium.Hmap.add key pool (Request.env req) in
51:     handler {req with env}
52:   in
53:   let m = Rock.Middleware.create ~name:"database connection pool" ~filter in
54:   middleware m app
55: 
56: (* Execute a query on the database connection pool stored in the request
57:    environment *)
58: let query_db query req =
59:   Request.env req
60:   |> Opium.Hmap.get key
61:   |> query_pool query
  1. Explanation of the code

    The key is used to look up the database connection from the environment (the environment being a heterogeneous valued hashmap implemented using Hmap). The db_pool type alias is kept private, and we do not export the key itself, so there is no way to retrieve the connection from the environment outside of the Db module.

    A piece of middleware is of type Opium.App.builder, i.e., Opium.App.t -> Optium.App.t. All of the Db middleware logic is in its inline filter function. In general, such filters take a handler and a request, req, do some stuff to construct a new request, and then handle the new request with the given handler. In this case, all we’re doing is making sure that the database connection pool is available in the environment of the request. We use opium’s Rock.Middleware.create to create our new middleware and apply it to the given app using Opium.Std.middleware.

    The query_db function is just a helper used to execute queries on the database connection stored in a request. If we wanted to let the client code execute arbitrary queries on the database connection, we could expose this function in the API. However, we choose to keep it hidden, so that the database can only be accessed through the API we define below in the Get and Update submodules.

4.4.3 Writing Database Queries using ppx_rapper

This project uses Roddy MacSween’s ppx_rapper for writing the SQL queries. It abstracts away some awkward boilerplate required by Caqti (tho writing queries directly in Caqti is not diffcult. See the Additional Resources for material covering this):

ppx_rapper generates functions to form queries based on a simple templating syntax on top of PostgreSQL:

  • Input parameters are indicated with %type{param_name}, which will produce a named argument for the function, ~param_name:type. When the flag record_in is supplied, these parameters are taken from record fields of the same name.
  • Outputs are indicated with @type{expression}, which, when the record_out flag is supplied, will return a record {expression : type; ...} including fields for all indicated values of the specified types.

To help illustrate how this works, compare the queries below with the functions in the Get and Update submodules which execute the queries:

 56: (** Collects all the SQL queries *)
 57: module Query = struct
 58:   type ('res, 'err) query_result = ('res, [> Caqti_error.call_or_retrieve ] as 'err) result Lwt.t
 59: 
 60:   let add_excerpt
 61:     : Caqti_lwt.connection -> Excerpt.t -> (unit, 'err) query_result =
 62:     let open Excerpt in
 63:     [%rapper execute
 64:         {sql|
 65:         INSERT INTO excerpts(author, excerpt, source, page)
 66:         VALUES (%string{author}, %string{excerpt}, %string{source}, %string?{page})
 67:         |sql}
 68:         record_in
 69:     ]
 70: 
 71:   let get_excerpts_by_author
 72:     : Caqti_lwt.connection -> author:string -> (Excerpt.t list, 'err) query_result =
 73:     let open Excerpt in
 74:     [%rapper get_many
 75:         {sql|
 76:         SELECT @string{author}, @string{excerpt}, @string{source}, @string?{page}
 77:         FROM excerpts
 78:         WHERE author = %string{author}
 79:         |sql}
 80:         record_out
 81:     ]
 82: 
 83:   let get_authors
 84:     : Caqti_lwt.connection -> unit -> (string list, 'err) query_result =
 85:     [%rapper get_many
 86:         {sql|
 87:         SELECT DISTINCT @string{author}
 88:         FROM excerpts
 89:         |sql}
 90:     ]
 91: end
 92: 
 93: (* Execute queries for fetching data *)
 94: module Get = struct
 95:   let excerpts_by_author author : Request.t -> (Excerpt.t list, string) Lwt_result.t =
 96:     query_db (Query.get_excerpts_by_author ~author)
 97: 
 98:   let authors : Request.t -> (string list, string) Lwt_result.t =
 99:     query_db (fun c -> Query.get_authors c ())
100: end
101: 
102: (* Execute queries for updating data *)
103: module Update = struct
104:   let add_excerpt excerpt : Request.t -> (unit, string) Lwt_result.t =
105:     query_db (fun c -> Query.add_excerpt c excerpt)
106: end
  1. Explanation of the code

    Query.add_excerpt will execute the query defined in the sql string. The function generated by ppx_rapper has no output values, as indicated by the absence of any leading @, so it will return a unit. The SQL template takes four input values, %string{author}, %string{excerpt}, %string{source}, and string?{page}. %string?{page} indicates that page will be of type string option. The use of the record_in dictates that these values will be taken from a record with the corresponding field names and types; i.e., a record of type Excerpt.t:

    1: type t =
    2:   { author: string
    3:   ; excerpt: string
    4:   ; source: string
    5:   ; page: string option
    6:   }
    

    The resulting type of Query.add_excerpt expresses that the function takes a connection to the database and inserts the excerpt into it, returning an Ok () if all goes well.

    In the query function get_excerpts_by_author we see a query which will get_many results. Each result will be a record_out, with fields and types as dictated by the terms prefixed with a @, i.e., the Excerpt.t record. We also see that the query takes one input parameter, the %string{author}. Since we do not specify that we’re expecting a record_in, the resulting function will take a named parameter ~author:string.

    These query functions are used in the Get and Update submodules exposed in the external API. Each function in the external API ends up taking a Request.t from the Opium App.t and returning any fetched values as an Lwt.t (Ok res) (or an error).

4.4.4 Using the Db API in a Route

The handler for requests to post "/excerpts/add" illustrates usage of the API:

34: let respond_or_err resp = function
35:   | Ok v      -> respond' @@ resp v
36:   | Error err -> respond' @@ Content.error_page err
37: 
38: let excerpt_of_form_data data =
39:   let find data key =
40:     let open Core in
41:     (* NOTE Should handle error in case of missing fields *)
42:     List.Assoc.find_exn ~equal:String.equal data key |> String.concat
43:   in
44:   let author  = find data "author"
45:   and excerpt = find data "excerpt"
46:   and source  = find data "source"
47:   and page    = match find data "page" with "" -> None | p -> Some p
48:   in
49:   Lwt.return Excerpt.{author; excerpt; source; page}
50: 
51: let post_excerpts_add = post "/excerpts/add" begin fun req ->
52:     let open Lwt in
53:     (* NOTE Should handle possible error arising from invalid data *)
54:     App.urlencoded_pairs_of_body req  >>=
55:     excerpt_of_form_data              >>= fun excerpt ->
56:     Db.Update.add_excerpt excerpt req >>=
57:     respond_or_err (fun () -> Content.excerpt_added_page excerpt)
58:   end
  1. Explanation of the code

    The respond_or_err helper deals with possible Error err values of a result, passing Ok values back as responses if possible, or else responding with an error page.

    The excerpt_of_form_data function takes an association list of key-value pairs representing form data and constructs an Excerpt.t record.

    Finally, the post_excerpts_add handler decodes a response body containing form data (encoded according to the application/x-www-form-urlencoded content type) into an association list, uses the previous helper to build an Excerpt.t, and then gives the Excerpt.t to the Db.Update.add_excerpt function, which inserts the Excerpt.t into the database via the connection stored in the environment of the req.

4.4.5 Implementation of a Rudimentary Database Migration Utility

This app includes naive, bespoke migration and rollback utilities. They are defined in the migration directory, and make use of the Db.Migration module:

108: module Migration = struct
109:   type 'a migration_error =
110:     [< Caqti_error.t > `Connect_failed `Connect_rejected `Post_connect ] as 'a
111: 
112:   type 'a migration_operation =
113:     Caqti_lwt.connection -> unit -> (unit, 'a migration_error) result Lwt.t
114: 
115:   type 'a migration_step = string * 'a migration_operation
116: 
117:   let execute migrations =
118:     let open Lwt in
119:     let rec run migrations pool = match migrations with
120:       | [] -> Lwt_result.return ()
121:       | (name, migration) :: migrations ->
122:         Lwt_io.printf "Running: %s\n" name        >>= fun () ->
123:         query_pool (fun c -> migration c ()) pool >>= function
124:         | Ok () -> run migrations pool
125:         | Error err -> return (Error err)
126:     in
127:     return (connect ()) >>= run migrations
128: end

A migration is just a pair of a string (the name of the migration step) and a query. The execute function takes a list of migrations, initiates a connection, and then runs each query on the pool, so long as they result in Ok values.

See ./migrate/migrate.ml and ./migrate/rollback.ml for examples of this simplistic scheme. This code has no awareness of the current version of the database, so it will dumbly try to run all migrations: something to improve in a productionized app.

5 Conclusion

This tutorial has taken us from initial installation of an OCaml development environment, through configuration and setup of a project, to development of a toy webapp, complete with an interface to PostgreSQL. There is still much more to cover, and many more OCaml libraries to explore and demo. But hopefully this material provides enough footholds to help a novice Caml rider get going.

6 Corrections, Suggestions, and Feedback

If you have any questions or feedback, please

Thanks in advance for any contributions!

7 Additional Resources

7.1 Documentation

7.2 Tutorials

7.3 Examples

Footnotes:

1

I plan to add an optional sqlite backend, which should make it easier to set up an run.

2

Project configuration in OCaml is currently more involved than anyone would like. We’re making steady and rapid progress on improving the situation, and this section should be updated to keep pace. If you read this tutorial and find the setup or configuration out of date, please open a PR or file an issue.

3

This probably isn’t the ideal way to handle the logging configuration, but it is straightforward and suffices for the context of this tutorial. See https://gitlab.com/shonfeder/ocaml_webapp/merge_requests/1 for a discussion of some alternatives and the benefits and drawbacks of this approach.

Author: Shon Feder

Created: 2020-01-10 Fri 14:26