Origin Private File System (OPFS) Database with the RxDB OPFS-RxStorage

With the RxDB OPFS storage you can build a fully featured database on top of the Origin Private File System (OPFS) browser API. Compared to other storage solutions, it has a way better performance.

What is OPFS

The Origin Private File System (OPFS) is a native browser storage API that allows web applications to manage files in a private, sandboxed, origin-specific virtual filesystem. Unlike IndexedDB and LocalStorage, which are optimized as object/key-value storage, OPFS provides more granular control for file operations, enabling byte-by-byte access, file streaming, and even low-level manipulations. OPFS is ideal for applications requiring high-performance file operations (3x-4x faster compared to IndexedDB) inside of a client-side application, offering advantages like improved speed, more efficient use of resources, and enhanced security and privacy features.

OPFS limitations

From the beginning of 2023, the Origin Private File System API is supported by all modern browsers like Safari, Chrome, Edge and Firefox. Only Internet Explorer is not supported and likely will never get support.

It is important to know that the most performant synchronous methods like read() and write() of the OPFS API are only available inside of a WebWorker. They cannot be used in the main thread, an iFrame or even a SharedWorker. The OPFS createSyncAccessHandle() method that gives you access to the synchronous methods is not exposed in the main thread, only in a Worker.

While there is no concrete data size limit defined by the API, browsers will refuse to store more data at some point. If no more data can be written, a QuotaExceededError is thrown which should be handled by the application, like showing an error message to the user.

How the OPFS API works

The OPFS API is pretty straightforward to use. First you get the root filesystem. Then you can create files and directories on that. Notice that whenever you synchronously write to, or read from a file, an ArrayBuffer must be used that contains the data. It is not possible to synchronously write plain strings or objects into the file. Therefore the TextEncoder and TextDecoder API must be used.

Also notice that some of the methods of FileSystemSyncAccessHandle have been asynchronous in the past, but are synchronous since Chromium 108. To make it less confusing, we just use await in front of them, so it will work in both cases.

// Access the root directory of the origin's private file system.
const root = await navigator.storage.getDirectory();

// Create a subdirectory.
const diaryDirectory = await root.getDirectoryHandle('subfolder', {
  create: true,
});  

// Create a new file named 'example.txt'.
const fileHandle = await diaryDirectory.getFileHandle('example.txt', {
  create: true,
});

// Create a FileSystemSyncAccessHandle on the file.
const accessHandle = await fileHandle.createSyncAccessHandle();

// Write a sentence to the file.
let writeBuffer = new TextEncoder().encode('Hello from RxDB');
const writeSize = accessHandle.write(writeBuffer);

// Read file and transform data to string.
const readBuffer = new Uint8Array(writeSize);
const readSize = accessHandle.read(readBuffer, { at: 0 });  
const contentAsString = new TextDecoder().decode(readBuffer);

// Write an exclamation mark to the end of the file.
writeBuffer = new TextEncoder().encode('!');
accessHandle.write(writeBuffer, { at: readSize });

// Truncate file to 10 bytes.
await accessHandle.truncate(10);

// Get the new size of the file.
const fileSize = await accessHandle.getSize();

// Persist changes to disk.
await accessHandle.flush();

// Always close FileSystemSyncAccessHandle if done, so others can open the file again.
await accessHandle.close();

A more detailed description of the OPFS API can be found on MDN.

OPFS performance

Because the Origin Private File System API provides low-level access to binary files, it is much faster compared to IndexedDB or localStorage. According to the storage performance test, OPFS is up to 2x times faster on plain inserts when a new file is created on each write. Reads are even faster.

A good comparison about real world scenarios, are the performance results of the various RxDB storages. Here it shows that reads are up to 4x faster compared to IndexedDB, even with complex queries:

Using OPFS as RxStorage in RxDB

The OPFS RxStorage itself must run inside a WebWorker. Therefore we use the Worker RxStorage and let it point to the prebuild opfs.worker.js file that comes shipped with RxDB Premium 👑.

Notice that the OPFS RxStorage is part of the RxDB Premium 👑 plugin that must be purchased.

import {
    createRxDatabase
} from 'rxdb';
import { getRxStorageWorker } from 'rxdb-premium/plugins/storage-worker';

const database = await createRxDatabase({
    name: 'mydatabase',
    storage: getRxStorageWorker(
        {
            /**
             * This file must be statically served from a webserver.
             * You might want to first copy it somewhere outside of
             * your node_modules folder.
             */
            workerInput: 'node_modules/rxdb-premium/dist/workers/opfs.worker.js'
        }
    )
});

Using OPFS in the main thread instead of a worker

The createSyncAccessHandle method from the Filesystem API is only available inside of a Webworker. Therefore you cannot use getRxStorageOPFS() in the main thread. But there is a slightly slower way to access the virtual filesystem from the main thread. RxDB support the getRxStorageOPFSMainThread() for that. Notice that this uses the createWritable function which is not supported in safari.

Using OPFS from the main thread can have benefits because not having to cross the worker bridge can reduce latence in reads and writes.

import { createRxDatabase } from 'rxdb';
import { getRxStorageOPFSMainThread } from 'rxdb-premium/plugins/storage-worker';

const database = await createRxDatabase({
    name: 'mydatabase',
    storage: getRxStorageOPFSMainThread()
});

Building a custom worker.js

When you want to run additional plugins like storage wrappers or replication inside of the worker, you have to build your own worker.js file. You can do that similar to other workers by calling exposeWorkerRxStorage like described in the worker storage plugin.

// inside of the worker.js file
import { getRxStorageOPFS } from 'rxdb-premium/plugins/storage-opfs';
import { exposeWorkerRxStorage } from 'rxdb-premium/plugins/storage-worker';

const storage = getRxStorageOPFS();
exposeWorkerRxStorage({
    storage
});

Setting usesRxDatabaseInWorker when a RxDatabase is also used inside of the worker

When you use the OPFS inside of a worker, it will internally use strings to represent operation results. This has the benefit that transferring strings from the worker to the main thread, is way faster compared to complex json objects. The getRxStorageWorker() will automatically decode these strings on the main thread so that the data can be used by the RxDatabase.

But using a RxDatabase inside of your worker can make sense for example when you want to move the replication with a server. To enable this, you have to set usesRxDatabaseInWorker to true:

// inside of the worker.js file
import { getRxStorageOPFS } from 'rxdb-premium/plugins/storage-opfs';
const storage = getRxStorageOPFS({
  usesRxDatabaseInWorker: true
});

If you forget to set this and still create and use a RxDatabase inside of the worker, you might get the error messageorUncaught (in promise) TypeError: Cannot read properties of undefined (reading 'length')`.

Setting jsonPositionSize to increase the maximum database size.

By default the jsonPositionSize value is set to 8 which allows the database to get up to 100 megabytes in size (per collection). This is ok for most use cases but you might want to just increase jsonPositionSize to 14. In the next major RxDB version the default will be set to 14, but this was not possible without introducing a breaking change.

note

If you have already stored data, you cannot just change the jsonPositionSize value because your stored binary data will not be compatible anymore.

Also there is a opfs-big.worker.js file that has jsonPositionSize set to 14 already.

OPFS in Electron, React-Native or Capacitor.js

Origin Private File System is a browser API that is only accessible in browsers. Other JavaScript like React-Native or Node.js, do not support it.

Electron has two JavaScript contexts: the browser (chromium) context and the Node.js context. While you could use the OPFS API in the browser context, it is not recommended. Instead you should use the Filesystem API of Node.js and then only transfer the relevant data with the ipcRenderer. With RxDB that is pretty easy to configure:

• In the main.js, expose the Node Filesystem storage with the exposeIpcMainRxStorage() that comes with the electron plugin
• In the browser context, access the main storage with the getRxStorageIpcRenderer() method.

React Native (and Expo) does not have an OPFS API. You could use the ReactNative Filesystem to directly write data. But to get a fully featured database like RxDB it is easier to use the SQLite RxStorage which starts an SQLite database inside of the ReactNative app and uses that to do the database operations.

Capacitor.js is able to access the OPFS API.

Difference between File System Access API and Origin Private File System (OPFS)

Often developers are confused with the differences between the File System Access API and the Origin Private File System (OPFS).

• The File System Access API provides access to the files on the device file system, like the ones shown in the file explorer of the operating system. To use the File System API, the user has to actively select the files from a filepicker.
• Origin Private File System (OPFS) is a sub-part of the File System Standard and it only describes the things you can do with the filesystem root from navigator.storage.getDirectory(). OPFS writes to a sandboxed filesystem, not visible to the user. Therefore the user does not have to actively select or allow the data access.

Learn more about OPFS:

• WebKit: The File System API with Origin Private File System
• Browser Support
• Performance Test Tool