Litestream Writable VFS

Full Fact Overview

<div class="lead"><p><strong class="font-semibold text-navy-950">I’m Ben Johnson, and I work on Litestream at Fly.io. Litestream is the missing backup/restore system for SQLite. It’s free, open-source software that should run anywhere, and</strong> <a href="/blog/litestream-v050-is-here/" title=""><strong class="font-semibold text-navy-950">you can read more about it here</strong></a><strong class="font-semibold text-navy-950">.</strong></p> </div> <p>Each time we write about it, we get a little bit better at golfing down a description of what Litestream is. Here goes: Litestream is a Unix-y tool for keeping a SQLite database synchronized with S3-style object storage. It&rsquo;s a way of getting the speed and simplicity wins of SQLite without exposing yourself to catastrophic data loss. Your app doesn&rsquo;t necessarily even need to know it&rsquo;s there; you can just run it as a tool in the background.</p> <p>It&rsquo;s been a busy couple weeks!</p> <p>We recently <a href='/blog/design-and-implementation/' title=''>unveiled Sprites</a>. If you don&rsquo;t know what Sprites are, you should just <a href='https://sprites.dev/' title=''>go check them out</a>. They&rsquo;re one of the coolest things we&rsquo;ve ever shipped. I won&rsquo;t waste any more time selling them to you. Just, Sprites are a big deal, and so it&rsquo;s a big deal to me that Litestream is a load-bearing component for them.</p> <p>Sprites rely directly on Litestream in two big ways.</p> <p>First, Litestream SQLite is the core of our global Sprites orchestrator. Unlike our flagship Fly Machines product, which relies on a centralized Postgres cluster, our Elixir Sprites orchestrator runs directly off S3-compatible object storage. Every organization enrolled in Sprites gets their own SQLite database, synchronized by Litestream.</p> <p>This is a fun design. It takes advantage of the &ldquo;many SQLite databases&rdquo; pattern, which is under-appreciated. It&rsquo;s got nice scaling characteristics. Keeping that Postgres cluster happy as Fly.io grew has been a major engineering challenge.</p> <p>But as far as Litestream is concerned, the orchestrator is boring, and so that&rsquo;s all I&rsquo;ve got to say about it. The second way Sprites use Litestream is much more interesting.</p> <p>Litestream is built directly into the disk storage stack that runs on every Sprite.</p> <p>Sprites launch in under a second, and every one of them boots up with 100GB of durable storage. That&rsquo;s a tricky bit of engineering. We&rsquo;re able to do this because the root of storage for Sprites is S3-compatible object storage, and we&rsquo;re able to make it fast by keeping a database of in-use storage blocks that takes advantage of attached NVMe as a read-through cache. The system that does this is JuiceFS, and the database — let&rsquo;s call it &ldquo;the block map&rdquo; — is a rewritten metadata store, based (you guessed it) on BoltDB.</p> <p>I kid! It&rsquo;s Litestream SQLite, of course.</p> <h2 id='sprite-storage-is-fussy' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'><a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#sprite-storage-is-fussy' aria-label='Anchor'></a><span class='plain-code'>Sprite Storage Is Fussy</span></h2> <p>Everything in a Sprite is designed to come up fast.</p> <p>If the Fly Machine underneath a Sprite bounces, we might need to reconstitute the block map from object storage. Block maps aren&rsquo;t huge, but they&rsquo;re not tiny; maybe low tens of megabytes worst case.</p> <p>The thing is, this is happening while the Sprite boots back up. To put that in perspective, that&rsquo;s something that can happen in response to an incoming web request; that is, we have to finish fast enough to generate a timely response to that request. The time budget is small.</p> <p>To make this even faster, we are integrating Litestream VFS to improve start times.The VFS is a dynamic library you load into your app. Once you do, you can do stuff like this:</p> <div class="highlight-wrapper group relative bash"> <button type="button" class="bubble-wrap z-20 absolute right-9 -mr-0.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none" data-wrap-target="#code-9mamlh2i" > <svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35" stroke-linecap="round" stroke-linejoin="round"><g buffered-rendering="static"><path d="M9.912 8.037h2.732c1.277 0 2.315-.962 2.315-2.237a2.325 2.325 0 00-2.315-2.31H2.959m10.228 9.01H2.959M6.802 8H2.959" /><path d="M11.081 6.466L9.533 8.037l1.548 1.571" /></g></svg> <span class="bubble-sm bubble-tl [--offset-l:-9px] tail text-navy-950"> Wrap text </span> </button> <button type="button" class="bubble-wrap z-20 absolute right-1.5 top-1.5 text-transparent group-hover:text-gray-400 group-hover:hocus:text-white focus:text-white bg-transparent group-hover:bg-gray-900 group-hover:hocus:bg-gray-700 focus:bg-gray-700 transition-colors grid place-items-center w-7 h-7 rounded-lg outline-none focus:outline-none" data-copy-target="sibling" > <svg class="w-4 h-4 pointer-events-none" viewBox="0 0 16 16" fill="none" stroke="currentColor" stroke-width="1.35"><g buffered-rendering="static"><path d="M10.576 7.239c0-.995-.82-1.815-1.815-1.815H3.315c-.995 0-1.815.82-1.815 1.815v5.446c0 .995.82 1.815 1.815 1.815h5.446c.995 0 1.815-.82 1.815-1.815V7.239z" /><path d="M10.576 10.577h2.109A1.825 1.825 0 0014.5 8.761V3.315A1.826 1.826 0 0012.685 1.5H7.239c-.996 0-1.815.819-1.816 1.815v1.617" /></g></svg> <span class="bubble-sm bubble-tl [--offset-l:-6px] tail [--tail-x:calc(100%-30px)] text-navy-950"> Copy to clipboard </span> </button> <div class='highlight relative group'> <pre class='highlight '><code id="code-9mamlh2i">sqlite> .open file:///my.db?vfs<span class="o">=</span>litestream sqlite> PRAGMA litestream_time <span class="o">=</span> <span class="s1">'5 minutes ago'</span><span class="p">;</span> sqlite> SELECT <span class="k">*</span> FROM sandwich_ratings ORDER BY RANDOM<span class="o">()</span> LIMIT 3 <span class="p">;</span> 22|Veggie Delight|New York|4 30|Meatball|Los Angeles|5 168|Chicken Shawarma Wrap|Detroit|5 </code></pre> </div> </div> <p>Litestream VFS lets us run point-in-time SQLite queries hot off object storage blobs, answering queries before we&rsquo;ve downloaded the database.</p> <p>This is good, but it&rsquo;s not perfect. We had two problems:</p> <ol> <li>We could only read, not write. People write to Sprite disks. The storage stack needs to write, right away. </li><li>Running a query off object storage is a godsend in a cold start where we have no other alternative besides downloading the whole database, but it&rsquo;s not fast enough for steady state. </li></ol> <p>These are fun problems. Here&rsquo;s our first cut at solving them.</p> <h2 id='writable-vfs' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'><a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#writable-vfs' aria-label='Anchor'></a><span class='plain-code'>Writable VFS</span></h2> <p>The first thing we&rsquo;ve done is made the VFS optionally read-write. This feature is pretty subtle; it&rsquo;s interesting, but it&rsquo;s not as general-purpose as it might look. Let me explain how it works, and then explain why it works this way.</p> <div class="callout"><p>Keep in mind as you read this that this is about the VFS in particular. Obviously, normal SQLite databases using Litestream the normal way are writeable.</p> </div> <p>The VFS works by keeping an index of <code>(file,offset, size)</code> for every page of the database in object storage; the data comprising the index is stored, <a href='https://github.com/superfly/ltx' title=''>in LTX files</a>, so that it&rsquo;s efficient for us to reconstitute it quickly when the VFS starts, and lookups are heavily cached. When we queried <code>sandwich_ratings</code> earlier, our VFS library intercepted the SQLite read method, looked up the requested page in the index, fetched it, and cached it.</p> <p>This works great for reads. Writes are harder.</p> <p>Behind the scenes in read-only mode, Litestream polls, so that we can detect new LTX files created by remote writers to the database. This supports a handy use case where we&rsquo;re running tests or doing slow analytical queries of databases that need to stay fast in prod.</p> <p>In write mode, we don&rsquo;t allow multiple writers, because multiple-writer distributed SQLite databases are the <a href='https://hellraiser.fandom.com/wiki/Lament_Configuration' title=''>Lament Configuration</a> and we are not explorers over great vistas of pain. So the VFS in write-mode disables polling. We assume a single writer, and no additional backups to watch.</p> <p>Next, we buffer. Writes go to a local temporary buffer (&ldquo;the write buffer&rdquo;). Every second or so (or on clean shutdown), we sync the write buffer with object storage. Nothing written through the VFS is truly durable until that sync happens.</p> <div class="right-sidenote"><p>Most storage block maps are much smaller than this, but still.</p> </div> <p>Now, remember the use case we&rsquo;re looking to support here. A Sprite is cold-starting and its storage stack needs to serve writes, milliseconds after booting, without having a full copy of the 10MB block map. This writeable VFS mode lets us do that.</p> <p>Critically, we support that use case only up to the same durability requirements that a Sprite already has. All storage on a Sprite shares this &ldquo;eventual durability&rdquo; property, so the terms of the VFS write make sense here. They probably don&rsquo;t make sense for your application. But if for some reason they do, have at it! To enable writes with Litestream VFS, just set the <code>LITESTREAM_WRITE_ENABLED</code> environment variable <code>"true"</code>.</p> <p><img src="/blog/litestream-writable-vfs/assets/write-path.png" /></p> <h2 id='hydration' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'><a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#hydration' aria-label='Anchor'></a><span class='plain-code'>Hydration</span></h2> <p>The Sprite storage stack uses SQLite in VFS mode. In our original VFS design, most data is kept in S3. Again: fine at cold start, not so fine in steady state.</p> <p>To solve this problem, we shoplifted a trick from <a href='https://docs.kernel.org/admin-guide/device-mapper/dm-clone.html' title=''>systems like dm-clone</a>: background hydration. In hydration designs, we serve queries remotely while running a loop to pull the whole database. When you start the VFS with the <code>LITESTREAM_HYDRATION_PATH</code> environment variable set, we&rsquo;ll hydrate to that file.</p> <p>Hydration takes advantage of <a href='https://fly.io/blog/litestream-revamped#point-in-time-restores-but-fast' title=''>LTX compaction</a>, writing only the latest versions of each page. Reads don&rsquo;t block on hydration; we serve them from object storage immediately, and switch over to the hydration file when it&rsquo;s ready.</p> <p><img src="/blog/litestream-writable-vfs/assets/timeline.png" /></p> <p>As for the hydration file? It&rsquo;s simply a full copy of your database. It&rsquo;s the same thing you get if you run <code>litestream restore</code>.</p> <p>Because this is designed for environments like Sprites, which bounce a lot, we write the database to a temporary file. We can&rsquo;t trust that the database is using the latest state every time we start up, not without doing a full restore, so we just chuck the hydration file when we exit the VFS. That behavior is baked into the VFS right now. This feature&rsquo;s got what Sprites need, but again, maybe not what your app wants.</p> <h2 id='putting-it-all-together' class='group flex items-start whitespace-pre-wrap relative mt-14 sm:mt-16 mb-4 text-navy-950 font-heading'><a class='inline-block align-text-top relative top-[.15em] w-6 h-6 -ml-6 after:hash opacity-0 group-hover:opacity-100 transition-all' href='#putting-it-all-together' aria-label='Anchor'></a><span class='plain-code'>Putting It All Together</span></h2> <p>This is a post about two relatively big moves we&rsquo;ve made with our open-source Litestream project, but the features are narrowly scoped for problems that look like the ones our storage stack needs. If you think you can get use out of them, I&rsquo;m thrilled, and I hope you&rsquo;ll tell me about it.</p> <p>For ordinary read/write workloads, you don&rsquo;t need any of this mechanism. Litestream works fine without the VFS, with unmodified applications, just running as a sidecar alongside your application. The whole point of that configuration is to efficiently keep up with writes; that&rsquo;s easy when you know you have the whole database to work with when writes happen.</p> <p>But this whole thing is, to me, a valuable case study in how Litestream can get used in a relatively complicated and demanding problem domain. Sprites are very cool, and it&rsquo;s satisfying to know that every disk write that happens on a Sprite is running through Litestream.</p>