Datomic Antipatterns: Eager Conn

  • clojure
  • datomic

For a long time, I wrote Datomic-based applications in a style where a conn value was globally available in my core “db” namespace. It just seamed easy; no futzing about connecting to a URI (connect) or creating a database (create-database) just to get started interacting with a database from the REPL.

It looks a little something like this:

(ns my.app.db
  (:require [datomic.api :as d/)

(def uri "...")

(def conn
  (d/connect uri))

;; and you're on with your day

After toiling with this approach for some time, I’ve come to realize this is an anti-pattern, for one big reason:

Early connections bloat application launch time

Costs

Connecting to a database in Datomic realizes a number of costs:

  • Your peer must handshake with and connect to storage (time).
  • Your peer draws portions of the index into local memory (time).
  • To connect, you must have an existing database.
  • Your application launch immediately consumes a peer slot.
  • You’ll be tempted to bake-in bootstrapping/schema-migrations to your connection logic (more time).

All of this adds up; applications, more specifically, your REPL, will take longer to launch, and decrease Clojure’s already tenuous launch times.

Doing it right

The solution is simple, both conceptually and in practice: defer connecting to and setting up a Datomic database until it is actually needed.

In web applications, this means setting up a connection at service launch, rather than process launch. I generally call this function bootstrap!, and within it I wrap up all aspects of database connection and initialization.

The following sample creates a database, connects to it, and transacts all of the relevant schema (using conformity):

(defn bootstrap!
  "Bootstrap schema into the database."
  [uri]
  ;; Create the database
  (d/create-database uri) ;; an idempotent call
  ;; Connect to it
  (let [conn (d/connect uri)]
    ;; and transact our application's schema...
    (doseq [rsc ["bouncy-castle.edn"]]
      (let [norms (c/load-schema-rsc rsc)]
        (c/ensure-conforms conn norms)))))

Once you have such a function in hand, wrap it up in the code that launches your application. (Ensuring, of course, that launching a REPL doesn’t actually fully launch your application.)

Stuart Sierra’s component makes this rather easy, but it’s just as well to do this closer to your application’s lifecycle:

;; By defining a component...
(defrecord DatomicDatabase [uri]
  component/Lifecycle
  (start [component]
    (bootstrap! uri)
    (let [conn (d/connect uri)]
      (assoc component :conn conn))

  (stop [component]
   (dissoc component :conn))))

;; Or, more directly (a Pedestal example)...
(defn start [service]
  (db/bootstrap! db/uri)
  (alter-var-root #'server (fn [_] (http/create-server service)))
  (http/start server))

A Conn in Every House

Once your application is up and running, the last thing you need to worry about is providing an active connection to your application.

Despite my warnings, you shouldn’t be afraid to code your APIs to Datomic URIs (rather than connections). Although connect is a relatively expensive call, it is only expensive once. Datomic caches connections internally, thus each successive call to connect is essentially inert.

That said, I find the above a little ugly, so I opt instead to inject a conn variable via middleware. The following Pedestal interceptor injects a conn and most-recent database value into every request:

(defbefore insert-db [context]
  (let [conn (d/connect uri)]
    (assoc context
           :conn conn
           :db (d/db conn))))

And the REPL…

I’ve side-stepped one last important detail. While my REPLs launch quickly, I don’t have direct access to a conn. Unfortunately, for this, I don’t yet have an ideal solution.

For the time being, I get around this by defining a comment block (as below) that defines a conn var in my database namespace. Whenever I launch a REPL, I invoke this def, and gain access to a more immediate conn than connecting ad hoc.

(comment
  (def conn
    (doto uri
      bootstrap!
      d/connect)))

How about yourself? Have you noticed connecting to a Datomic database bogging down your REPL launches? How do you pass around connections in your own applications?

Like this post? Subscribe to my newsletter.

Get fresh content on Clojure, Architecture and Software Development, each and every week.

comments powered by Disqus