# RxDB Documentation
> Authoritative reference documentation for RxDB, a reactive, local-first NoSQL database for JavaScript with offline support and explicit replication.
This file contains all documentation content in a single document following the llmstxt.org standard.
## RxDB Docs
import { Overview } from '@site/src/components/overview';
# RxDB Documentation
---
## π Quickstart
import {Steps} from '@site/src/components/steps';
import {TriggerEvent} from '@site/src/components/trigger-event';
import {Tabs} from '@site/src/components/tabs';
import {NavbarDropdownSyncList} from '@site/src/components/navbar-dropdowns';
# RxDB Quickstart
Welcome to the RxDB Quickstart. Here we'll learn how to create a simple real-time app with the RxDB database that is able to store and query data persistently in a browser and does realtime updates to the UI on changes.
### Installation
Install the RxDB library and the RxJS dependency:
```bash
npm install rxdb rxjs
```
### Pick a Storage
RxDB is able to run in a wide range of JavaScript runtimes like browsers, mobile apps, desktop and servers. Therefore different storage engines exist that ensure the best performance depending on where RxDB is used.
#### LocalStorage
Use this for the simplest browser setup and very small datasets. It has a tiny bundle size and works anywhere [localStorage](./articles/localstorage.md) is available, but is not optimized for large data or heavy writes.
```ts
import {
getRxStorageLocalstorage
} from 'rxdb/plugins/storage-localstorage';
let storage = getRxStorageLocalstorage();
```
#### IndexedDB π
The premium [IndexedDB storage](./rx-storage-indexeddb.md) is a high-performance, browser-native storage with a smaller bundle and faster startup compared to Dexie-based IndexedDB. Recommended when you have [π premium](/premium/) access and care about performance and bundle size.
```ts
import {
getRxStorageIndexedDB
} from 'rxdb-premium/plugins/storage-indexeddb';
let storage = getRxStorageIndexedDB();
```
#### Dexie.js
[Dexie.js](./rx-storage-dexie.md) is a friendly wrapper around IndexedDB and is a great default for browser apps when you don't use premium. It's reliable, works well for medium-sized datasets, and is free to use.
```ts
import {
getRxStorageDexie
} from 'rxdb/plugins/storage-dexie';
let storage = getRxStorageDexie();
```
#### SQLite
[SQLite](./rx-storage-sqlite.md) is ideal for React Native, Capacitor, Electron, Node.js and other hybrid or native environments. It gives you a fast, durable database on disk. Use the π premium storage for production; a trial version exists for quick experimentation.
**Premium SQLite (Node.js example)**
```ts
import {
getRxStorageSQLite,
getSQLiteBasicsNode
} from 'rxdb-premium/plugins/storage-sqlite';
// Provide the sqliteBasics adapter for your runtime, e.g. Node.js, React Native, etc.
// For example in Node.js you would derive sqliteBasics from a sqlite3-compatible library:
import sqlite3 from 'sqlite3';
const storage = getRxStorageSQLite({
sqliteBasics: getSQLiteBasicsNode(sqlite3)
});
```
**SQLite trial storage (Node.js, free)**
```ts
import {
getRxStorageSQLiteTrial,
getSQLiteBasicsNodeNative
} from 'rxdb/plugins/storage-sqlite';
import { DatabaseSync } from 'node:sqlite';
const storage = getRxStorageSQLiteTrial({
sqliteBasics: getSQLiteBasicsNodeNative(DatabaseSync)
});
```
#### And more...
There are many more storages such as [MongoDB](./rx-storage-mongodb.md), [DenoKV](./rx-storage-denokv.md), [Filesystem](./rx-storage-filesystem-node.md), [Memory](./rx-storage-memory.md), [Memory-Mapped](./rx-storage-memory-mapped.md), [FoundationDB](./rx-storage-foundationdb.md) and more. [Browse the full list of storages](/rx-storage.html).
Which storage should I use?
RxDB provides a wide range of storages depending on your JavaScript runtime and performance needs.
In the Browser: Use the LocalStorage storage for simple setup and small build size. For bigger datasets, use either the dexie.js storage (free) or the IndexedDB RxStorage if you have π premium access which is a bit faster and has a smaller build size.
In Electron and React Native: Use the SQLite RxStorage if you have π premium access or the SQLite Trial RxStorage for tryouts.
In Capacitor: Use the SQLite RxStorage if you have π premium access, otherwise use the LocalStorage storage.
### Dev-Mode
When you use RxDB in development, you should always enable the [dev-mode plugin](./dev-mode.md), which adds helpful checks and validations, and tells you if you do something wrong.
```ts
import { addRxPlugin } from 'rxdb/plugins/core';
import { RxDBDevModePlugin } from 'rxdb/plugins/dev-mode';
addRxPlugin(RxDBDevModePlugin);
```
### Schema Validation
[Schema validation](./schema-validation.md) is required when using dev-mode and recommended (but optional) in production. Wrap your storage with the AJV schema validator to ensure all documents match your schema before being saved.
```ts
import { wrappedValidateAjvStorage } from 'rxdb/plugins/validate-ajv';
storage = wrappedValidateAjvStorage({ storage });
```
### Create a Database
A database is the topβlevel container in RxDB, responsible for managing collections, coordinating persistence, and providing reactive change streams.
```ts
import { createRxDatabase } from 'rxdb/plugins/core';
const myDatabase = await createRxDatabase({
name: 'mydatabase',
storage: storage
});
```
### Add a Collection
An RxDatabase contains [RxCollection](./rx-collection.md)s for storing and querying data. A collection is similar to an SQL table, and individual records are stored in the collection as JSON documents. An [RxDatabase](./rx-database.md) can have as many collections as you need.
Add a collection with a [schema](./rx-schema.md) to the database:
```ts
await myDatabase.addCollections({
// name of the collection
todos: {
// we use the JSON-schema standard
schema: {
version: 0,
primaryKey: 'id',
type: 'object',
properties: {
id: {
type: 'string',
maxLength: 100 // <- the primary key must have maxLength
},
name: {
type: 'string'
},
done: {
type: 'boolean'
},
timestamp: {
type: 'string',
format: 'date-time'
}
},
required: ['id', 'name', 'done', 'timestamp']
}
}
});
```
### Insert a Document
Now that we have an RxCollection we can store some [documents](./rx-document.md) in it.
```ts
const myDocument = await myDatabase.todos.insert({
id: 'todo1',
name: 'Learn RxDB',
done: false,
timestamp: new Date().toISOString()
});
```
### Run a Query
Execute a [query](./rx-query.md) that returns all found documents once:
```ts
const foundDocuments = await myDatabase.todos.find({
selector: {
done: {
$eq: false
}
}
}).exec();
```
### Update a Document
In the first found document, set `done` to `true`:
```ts
const firstDocument = foundDocuments[0];
await firstDocument.patch({
done: true
});
```
### Delete a Document
Delete the document so that it can no longer be found in queries:
```ts
await firstDocument.remove();
```
### Observe a Query
Subscribe to data changes so that your UI is always up-to-date with the data stored on disk. RxDB allows you to subscribe to data changes even when the change happens in another part of your application, another browser tab, or during database [replication/synchronization](./replication.md):
```ts
const observable = myDatabase.todos.find({
selector: {
done: {
$eq: false
}
}
}).$ // get the observable via RxQuery.$;
observable.subscribe(notDoneDocs => {
console.log('Currently have ' + notDoneDocs.length + ' things to do');
// -> here you would re-render your app to show the updated document list
});
```
### Observe a Document Value
You can also subscribe to the fields of a single RxDocument. Add the `$` sign to the desired field and then subscribe to the returned observable.
```ts
myDocument.done$.subscribe(isDone => {
console.log('done: ' + isDone);
});
```
### Sync the Client
RxDB has multiple [replication plugins](./replication.md) to replicate database state with a server.
#### HTTP
```ts
import {
replicateHTTP,
pullQueryBuilderFromRxSchema,
} from "rxdb/plugins/replication-http";
replicateHTTP({
collection: db.todos,
push: {
handler: async (rows) => {
return fetch("https://example.com/api/todos/push", {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify(rows),
}).then((res) => res.json());
},
},
pull: {
handler: async (lastCheckpoint) => {
return fetch(
"https://example.com/api/todos/pull?" +
new URLSearchParams({
checkpoint: JSON.stringify(lastCheckpoint)
}),
).then((res) => res.json());
},
},
});
```
#### GraphQL
```ts
import { replicateGraphQL } from 'rxdb/plugins/replication-graphql';
replicateGraphQL({
collection: db.todos,
url: 'https://example.com/graphql',
push: { batchSize: 50 },
pull: { batchSize: 50 }
});
```
#### WebRTC (P2P)
The easiest way to replicate data between your clients' devices is the [WebRTC replication plugin](./replication-webrtc.md) that replicates data between devices without a centralized server. This makes it easy to try out replication without having to host anything:
```ts
import {
replicateWebRTC,
getConnectionHandlerSimplePeer
} from 'rxdb/plugins/replication-webrtc';
replicateWebRTC({
collection: myDatabase.todos,
connectionHandlerCreator: getConnectionHandlerSimplePeer({}),
topic: '', // <- set any app-specific room id here.
secret: 'mysecret',
pull: {},
push: {}
})
```
#### CouchDB
```ts
import { replicateCouchDB } from 'rxdb/plugins/replication-couchdb';
replicateCouchDB({
collection: db.todos,
url: 'http://example.com/todos/',
push: {},
pull: {}
});
```
#### And more...
Explore all [replication plugins](/replication.html), including advanced conflict handling and custom protocols.
## Next steps
You are now ready to dive deeper into RxDB.
- Start reading the full documentation [here](./install.md).
- There is a full implementation of the [quickstart guide](https://github.com/pubkey/rxdb-quickstart) so you can clone that repository and play with the code.
- For frameworks and runtimes like Angular, React Native and others, check out the list of [example implementations](https://github.com/pubkey/rxdb/tree/master/examples).
- Also please continue reading the documentation, join the community on our [Discord chat](/chat/), and star the [GitHub repo](https://github.com/pubkey/rxdb).
- If you are using RxDB in a production environment and are able to support its continued development, please take a look at the [π Premium package](/premium/) which includes additional plugins and utilities.
---
## Installation
# Install RxDB
## npm
To install the latest release of `rxdb` and its dependencies and save it to your `package.json`, run:
`npm i rxdb --save`
## peer-dependency
You also need to install the peer-dependency `rxjs` if you have not installed it before.
`npm i rxjs --save`
## polyfills
RxDB is coded with ES8 and transpiled to ES5. This means you have to install [polyfills](https://developer.mozilla.org/en-US/docs/Glossary/Polyfill) to support older browsers. For example you can use the babel-polyfills with:
`npm i @babel/polyfill --save`
If you need polyfills, you have to import them in your code.
```typescript
import '@babel/polyfill';
```
## Polyfill the `global` variable
When you use RxDB with [Angular](./articles/angular-database.md) or other **Webpack** based frameworks, you might get the error `Uncaught ReferenceError: global is not defined`.
This is because some dependencies of RxDB assume a Node.js-specific `global` variable that is not added to browser runtimes by some bundlers.
You have to add them manually, like we do [here](https://github.com/pubkey/rxdb/blob/master/examples/angular/src/polyfills.ts).
```ts
(window as any).global = window;
(window as any).process = {
env: { DEBUG: undefined },
};
```
## Project Setup and Configuration
In the [examples](https://github.com/pubkey/rxdb/tree/master/examples) folder you can find CI tested projects for different frameworks and use cases, while in the [/config](https://github.com/pubkey/rxdb/tree/master/config) folder base configuration files for Webpack, Rollup, Mocha, Karma, TypeScript are exposed.
Consult [package.json](https://github.com/pubkey/rxdb/blob/master/package.json) for the versions of the packages supported.
## Installing the latest RxDB build
If you need the latest development state of RxDB, add it as git dependency into your `package.json`.
```json
"dependencies": {
"rxdb": "git+https://git@github.com/pubkey/rxdb.git#commitHash"
}
```
Replace `commitHash` with the hash of the latest [build-commit](https://github.com/pubkey/rxdb/search?q=build&type=Commits).
## Import
To import `rxdb`, add this to your JavaScript file to import the default bundle that contains the RxDB core:
```typescript
import {
createRxDatabase, // ./rx-database.md
/* ... */
} from 'rxdb';
```
---
## Development Mode
import {Steps} from '@site/src/components/steps';
# Dev Mode
The dev-mode plugin adds many checks and validations to RxDB.
This ensures that you use the RxDB API properly and so the dev-mode plugin should always be used when
using RxDB in development mode.
- Adds readable error messages.
- Ensures that `readonly` JavaScript objects are not accidentally mutated.
- Adds validation check for validity of schemas, queries, [ORM](./orm.md) methods and document fields.
- Notice that the `dev-mode` plugin does not perform schema checks against the data see [schema validation](./schema-validation.md) for that.
:::warning
The dev-mode plugin will increase your build size and decrease the performance. It must **always** be used in development. You should **never** use it in production.
:::
### Import the dev-mode Plugin
```javascript
import { RxDBDevModePlugin } from 'rxdb/plugins/dev-mode';
import { addRxPlugin } from 'rxdb/plugins/core';
```
## Add the Plugin to RxDB
```javascript
addRxPlugin(RxDBDevModePlugin);
```
## Usage with Node.js
```ts
async function createDb() {
if (process.env.NODE_ENV !== "production") {
await import('rxdb/plugins/dev-mode').then(
module => addRxPlugin(module.RxDBDevModePlugin)
);
}
const db = createRxDatabase( /* ... */ );
}
```
## Usage with [Angular](./articles/angular-database.md)
```ts
import { isDevMode } from '@angular/core';
async function createDb() {
if (isDevMode()){
await import('rxdb/plugins/dev-mode').then(
module => addRxPlugin(module.RxDBDevModePlugin)
);
}
const db = createRxDatabase( /* ... */ );
// ...
}
```
## Usage with webpack
In the `webpack.config.js`:
```ts
module.exports = {
entry: './src/index.ts',
/* ... */
plugins: [
// set a global variable that can be accessed during runtime
new webpack.DefinePlugin({ MODE: JSON.stringify("production") })
]
/* ... */
};
```
In your source code:
```ts
declare var MODE: 'production' | 'development';
async function createDb() {
if (MODE === 'development') {
await import('rxdb/plugins/dev-mode').then(
module => addRxPlugin(module.RxDBDevModePlugin)
);
}
const db = createRxDatabase( /* ... */ );
// ...
}
```
## Disable the dev-mode warning
When the dev-mode is enabled, it will print a `console.warn()` message to the console so that you do not accidentally use the dev-mode in production. To disable this warning you can call the `disableWarnings()` function.
```ts
import { disableWarnings } from 'rxdb/plugins/dev-mode';
disableWarnings();
```
## Disable the tracking iframe
When used in localhost and in the browser, the dev-mode plugin can add a tracking iframe to the DOM. This is used to track the effectiveness of marketing efforts of RxDB.
If you have [premium access](/premium/) and want to disable this iframe, you can call `setPremiumFlag()` before creating the database.
```js
import { setPremiumFlag } from 'rxdb-premium/plugins/shared';
setPremiumFlag();
```
---
## TypeScript Setup
import {Steps} from '@site/src/components/steps';
# Using RxDB with TypeScript
In this tutorial you will learn how to use RxDB with TypeScript.
We will create a basic database with one collection and several ORM-methods, fully typed!
RxDB directly comes with its typings and you do not have to install anything else, however the latest version of RxDB requires that you are using Typescript v3.8 or newer.
Our way to go is
- First define what the documents look like
- Then define what the collections look like
- Then define what the database looks like
## Declare the types
First you import the types from RxDB.
```typescript
import {
createRxDatabase,
RxDatabase,
RxCollection,
RxJsonSchema,
RxDocument,
} from 'rxdb/plugins/core';
```
## Create the base document type
First we have to define the TypeScript type of the documents of a collection:
**Option A**: Create the document type from the schema
```typescript
import {
toTypedRxJsonSchema,
ExtractDocumentTypeFromTypedRxJsonSchema,
RxJsonSchema
} from 'rxdb';
export const heroSchemaLiteral = {
title: 'hero schema',
description: 'describes a human being',
version: 0,
keyCompression: true,
primaryKey: 'passportId',
type: 'object',
properties: {
passportId: {
type: 'string',
maxLength: 100 // <- the primary key must have set maxLength
},
firstName: {
type: 'string'
},
lastName: {
type: 'string'
},
age: {
type: 'integer'
}
},
required: ['firstName', 'lastName', 'passportId'],
indexes: ['firstName']
} as const; // <- It is important to set 'as const' to preserve the literal type
const schemaTyped = toTypedRxJsonSchema(heroSchemaLiteral);
// aggregate the document type from the schema
export type HeroDocType = ExtractDocumentTypeFromTypedRxJsonSchema;
// create the typed RxJsonSchema from the literal typed object.
export const heroSchema: RxJsonSchema = heroSchemaLiteral;
```
**Option B**: Manually type the document type
```typescript
export type HeroDocType = {
passportId: string;
firstName: string;
lastName: string;
age?: number; // optional
};
```
**Option C**: Generate the document type from schema during build time
If your schema is in a `.json` file or generated from somewhere else, you might generate the typings with the [json-schema-to-typescript](https://www.npmjs.com/package/json-schema-to-typescript) module.
## Types for the ORM methods
We also add some ORM-methods for the document.
```typescript
export type HeroDocMethods = {
scream: (v: string) => string;
};
```
## Create [RxDocument](../rx-document.md) Type
We can merge these into our HeroDocument.
```typescript
export type HeroDocument = RxDocument;
```
## Create [RxCollection](../rx-collection.md) Type
Now we can define type for the collection which contains the documents.
```typescript
// we declare one static ORM-method for the collection
export type HeroCollectionMethods = {
countAllDocuments: () => Promise;
}
// and then merge all our types
export type HeroCollection = RxCollection<
HeroDocType,
HeroDocMethods,
HeroCollectionMethods
>;
```
## Create [RxDatabase](../rx-database.md) Type
Before we can define the database, we make a helper-type which contains all collections of it.
```typescript
export type MyDatabaseCollections = {
heroes: HeroCollection
}
```
Now the database.
```typescript
export type MyDatabase = RxDatabase;
```
## Using the types
Now that we have declare all our types, we can use them.
```typescript
/**
* create database and collections
*/
const myDatabase: MyDatabase = await createRxDatabase({
name: 'mydb',
storage: getRxStorageLocalstorage()
});
const heroSchema: RxJsonSchema = {
title: 'human schema',
description: 'describes a human being',
version: 0,
keyCompression: true,
primaryKey: 'passportId',
type: 'object',
properties: {
passportId: {
type: 'string'
},
firstName: {
type: 'string'
},
lastName: {
type: 'string'
},
age: {
type: 'integer'
}
},
required: ['passportId', 'firstName', 'lastName']
};
const heroDocMethods: HeroDocMethods = {
scream: function(this: HeroDocument, what: string) {
return this.firstName + ' screams: ' + what.toUpperCase();
}
};
const heroCollectionMethods: HeroCollectionMethods = {
countAllDocuments: async function(this: HeroCollection) {
const allDocs = await this.find().exec();
return allDocs.length;
}
};
await myDatabase.addCollections({
heroes: {
schema: heroSchema,
methods: heroDocMethods,
statics: heroCollectionMethods
}
});
// add a postInsert-hook
myDatabase.heroes.postInsert(
function myPostInsertHook(
this: HeroCollection, // own collection is bound to the scope
docData: HeroDocType, // documents data
doc: HeroDocument // RxDocument
) {
console.log('insert to ' + this.name + '-collection: ' + doc.firstName);
},
false // not async
);
/**
* use the database
*/
// insert a document
const hero: HeroDocument = await myDatabase.heroes.insert({
passportId: 'myId',
firstName: 'piotr',
lastName: 'potter',
age: 5
});
// access a property
console.log(hero.firstName);
// use a orm method
hero.scream('AAH!');
// use a static orm method from the collection
const amount: number = await myDatabase.heroes.countAllDocuments();
console.log(amount);
/**
* clean up
*/
myDatabase.close();
```
---
## RxDatabase - The Core of Your Realtime Data
# RxDatabase
An RxDatabase Object contains your [collections](./rx-collection.md) and handles the synchronization of change events.
## Creation
The database is created by the asynchronous `.createRxDatabase()` function of the core RxDB module. It has the following parameters:
```javascript
import { createRxDatabase } from 'rxdb/plugins/core';
import { getRxStorageLocalstorage } from 'rxdb/plugins/storage-localstorage';
const db = await createRxDatabase({
name: 'heroesdb', // <- name
storage: getRxStorageLocalstorage(), // <- RxStorage
/* Optional parameters: */
password: 'myPassword', // <- password (optional)
multiInstance: true, // <- multiInstance (optional, default: true)
eventReduce: true, // <- eventReduce (optional, default: false)
cleanupPolicy: {} // <- custom cleanup policy (optional)
});
```
### name
The database name is a string which uniquely identifies the database. When two RxDatabases have the same name and use the same `RxStorage`, their data can be assumed as equal and they will share events between each other.
Depending on the storage or adapter this can also be used to define the filesystem folder of your data.
### storage
RxDB works on top of an implementation of the [RxStorage](./rx-storage.md) interface. This interface is an abstraction that allows you to use different underlying databases that actually handle the documents. Depending on your use case you might use a different `storage` with different tradeoffs in performance, bundle size or supported runtimes.
There are many `RxStorage` implementations that can be used depending on the JavaScript environment and performance requirements.
For example you can use the [LocalStorage RxStorage](./rx-storage-localstorage.md) in the browser or use the [MongoDB RxStorage](./rx-storage-mongodb.md) in Node.js.
- [List of RxStorage implementations](./rx-storage.md)
```javascript
// use the LocalStorage that stores data in the browser.
import { getRxStorageLocalstorage } from 'rxdb/plugins/storage-localstorage';
const db = await createRxDatabase({
name: 'mydatabase',
storage: getRxStorageLocalstorage()
});
// ...or use the MongoDB RxStorage in Node.js.
import { getRxStorageMongoDB } from 'rxdb/plugins/storage-mongodb';
const dbMongo = await createRxDatabase({
name: 'mydatabase',
storage: getRxStorageMongoDB({
connection: 'mongodb://localhost:27017,localhost:27018,localhost:27019'
})
});
```
### password
`(optional)`
If you want to use encrypted fields in the collections of a database, you have to set a password for it. The password must be a string with at least 12 characters.
[Read more about encryption here](./encryption.md).
### multiInstance
`(optional=true)`
When you create more than one instance of the same database in a single javascript-runtime, you should set `multiInstance` to ```true```. This will enable the event sharing between the two instances. For example when the user has opened multiple browser windows, events will be shared between them so that both windows react to the same changes.
`multiInstance` should be set to `false` when you have single instances like a single Node.js process, a React Native app, a Cordova app or a single-window [Electron](./electron-database.md) app which can decrease the startup time because no instance coordination has to be done.
### eventReduce
`(optional=false)`
One big benefit of having a realtime database is that big performance optimizations can be done when the database knows a query is observed and the updated results are needed continuously. RxDB uses the [EventReduce Algorithm](https://github.com/pubkey/event-reduce) to optimize observer or recurring queries.
For better performance, you should always set `eventReduce: true`. This will also be the default in the next major RxDB version.
### ignoreDuplicate
`(optional=false)`
If you create multiple RxDatabase-instances with the same name and same adapter, it's very likely that you have done something wrong.
To prevent this common mistake, RxDB will throw an error when you do this.
In some rare cases like unit-tests, you want to do this intentionally by setting `ignoreDuplicate` to `true`. Because setting `ignoreDuplicate: true` in production will decrease the performance by having multiple instances of the same database, `ignoreDuplicate` is only allowed to be set in [dev-mode](./dev-mode.md).
```js
const db1 = await createRxDatabase({
name: 'heroesdb',
storage: getRxStorageLocalstorage(),
ignoreDuplicate: true
});
const db2 = await createRxDatabase({
name: 'heroesdb',
storage: getRxStorageLocalstorage(),
ignoreDuplicate: true // this create-call will not throw because you explicitly allow it
});
```
### closeDuplicates
`(optional=false)`
Closes all other RxDatabase instances that have the same storage+name combination.
```js
const db1 = await createRxDatabase({
name: 'heroesdb',
storage: getRxStorageLocalstorage(),
closeDuplicates: true
});
const db2 = await createRxDatabase({
name: 'heroesdb',
storage: getRxStorageLocalstorage(),
closeDuplicates: true // this create-call will close db1
});
// db1 is now closed.
```
### hashFunction
By default, RxDB will use `crypto.subtle.digest('SHA-256', data)` for hashing. If you need a different hash function or the `crypto.subtle` API is not supported in your JavaScript runtime, you can provide your own hash function instead. A hash function gets a `string`, `ArrayBuffer`, or `Blob` as input and returns a `Promise` that resolves a string. When a `Blob` is received (for attachment digest hashing), convert it to a string or ArrayBuffer before hashing.
```ts
// example hash function that runs in plain JavaScript
import { sha256 } from 'ohash';
import { blobToBase64String } from 'rxdb';
async function myOwnHashFunction(input: string | ArrayBuffer | Blob) {
if (typeof Blob !== 'undefined' && input instanceof Blob) {
input = await blobToBase64String(input);
} else if (input instanceof ArrayBuffer) {
input = new TextDecoder().decode(new Uint8Array(input));
}
return sha256(input);
}
const db = await createRxDatabase({
hashFunction: myOwnHashFunction
/* ... */
});
```
If you get the error message `TypeError: Cannot read properties of undefined (reading 'digest')` this likely means that you are neither running on `localhost` nor on `https` which is why your browser might not allow access to `crypto.subtle.digest`.
## Methods
### Observe with $
Calling this will return an [RxJS Observable](http://reactivex.io/documentation/observable.html) which streams all write events of the `RxDatabase`.
```javascript
myDb.$.subscribe(changeEvent => console.dir(changeEvent));
```
### exportJSON()
Use this function to create a JSON export from every piece of data in every collection of this database. You can pass `true` as a parameter to decrypt the encrypted data fields of your document.
Before `exportJSON()` and `importJSON()` can be used, you have to add the `json-dump` plugin.
```javascript
import { addRxPlugin } from 'rxdb';
import { RxDBJsonDumpPlugin } from 'rxdb/plugins/json-dump';
addRxPlugin(RxDBJsonDumpPlugin);
```
```javascript
myDatabase.exportJSON()
.then(json => console.dir(json));
```
### importJSON()
To import the JSON dumps into your database, use this function.
```javascript
// import the dump to the database
emptyDatabase.importJSON(json)
.then(() => console.log('done'));
```
### backup()
Writes the current (or ongoing) database state to the filesystem. [Read more](./backup.md)
### waitForLeadership()
Returns a Promise which resolves when the RxDatabase becomes [elected leader](./leader-election.md).
### requestIdlePromise()
Returns a promise which resolves when the database is in idle. This works similar to [requestIdleCallback](https://developer.mozilla.org/de/docs/Web/API/Window/requestIdleCallback) but tracks the idleness of the database instead of the CPU.
Use this for semi-important tasks like cleanups which should not affect the speed of important tasks.
```javascript
myDatabase.requestIdlePromise().then(() => {
// this will run at the moment the database has nothing else to do
myCollection.customCleanupFunction();
});
// with timeout
myDatabase.requestIdlePromise(1000 /* time in ms */).then(() => {
// this will run at the moment the database has nothing else to do
// or the timeout has passed
myCollection.customCleanupFunction();
});
```
### close()
Closes the database's object instance. This is to free up memory and stop all observers and replications.
Returns a `Promise` that resolves when the database is closed.
Closing a database will not remove the database's data. When you create the database again with `createRxDatabase()`, all data will still be there.
```javascript
await myDatabase.close();
```
### remove()
Wipes all documents from the storage. Use this to free up disk space.
```javascript
await myDatabase.remove();
// database instance is now gone
```
You can also clear a database without removing its instance by using `removeRxDatabase()`. This is useful if you want to migrate data or reset the user's state by renaming the database. Then you can remove the previous data with `removeRxDatabase()` without creating a RxDatabase first. Notice that this will only remove the
stored data on the storage. It will not clear the cache of any [RxDatabase](./rx-database.md) instances.
```javascript
import { removeRxDatabase } from 'rxdb';
removeRxDatabase('mydatabasename', 'localstorage');
```
### isRxDatabase
Returns true if the given object is an instance of RxDatabase. Returns false if not.
```javascript
import { isRxDatabase } from 'rxdb';
const is = isRxDatabase(myObj);
```
### collections$
Emits events whenever an [RxCollection](./rx-collection.md) is added or removed to the instance of the RxDatabase. Notice that this only emits the JavaScript instance of the RxCollection class, it does not emit events across browser tabs.
```javascript
const sub = myDatabase.collections$.subscribe(event => {
console.dir(event);
});
await myDatabase.addCollections({
heroes: {
schema: mySchema
}
});
// -> emits the event
sub.unsubscribe();
```
---
## Design Perfect Schemas in RxDB
# RxSchema
Schemas define the structure of the documents of a collection. Which field should be used as the primary key, which fields should be used as indexes, and what should be encrypted. Every collection has its own schema. With RxDB, schemas are defined with the [JSON Schema](https://json-schema.org/blog/posts/rxdb-case-study) standard which you might know from other projects.
## Example
In this example-schema we define a hero-collection with the following settings:
- the version-number of the schema is 0
- the name-property is the **primaryKey**. This means it's a unique, indexed, required `string` which can be used to definitely find a single document.
- the color-field is required for every document
- the healthpoints-field must be a number between 0 and 100
- the secret-field stores an encrypted value
- the birthyear-field is final which means it is required and cannot be changed
- the skills-attribute must be an array of objects which contain the name and the damage-attribute. There is a maximum of 5 skills per hero.
- Allows adding attachments and storing them encrypted
```json
{
"title": "hero schema",
"version": 0,
"description": "describes a simple hero",
"primaryKey": "name",
"type": "object",
"properties": {
"name": {
"type": "string",
"maxLength": 100 // <- the primary key must have set maxLength
},
"color": {
"type": "string"
},
"healthpoints": {
"type": "number",
"minimum": 0,
"maximum": 100
},
"secret": {
"type": "string"
},
"birthyear": {
"type": "number",
"final": true,
"minimum": 1900,
"maximum": 2050
},
"skills": {
"type": "array",
"maxItems": 5,
"uniqueItems": true,
"items": {
"type": "object",
"properties": {
"name": {
"type": "string"
},
"damage": {
"type": "number"
}
}
}
}
},
"required": [
"name",
"color"
],
"encrypted": ["secret"],
"attachments": {
"encrypted": true
}
}
```
## Create a collection with the schema
```javascript
await myDatabase.addCollections({
heroes: {
schema: myHeroSchema
}
});
console.dir(myDatabase.heroes.name);
// heroes
```
## version
The `version` field is a number, starting with `0`.
When the version is greater than 0, you have to provide the `migrationStrategies` to create a collection with this schema.
## primaryKey
The `primaryKey` field contains the fieldname of the property that will be used as primary key for the whole collection.
The value of the primary key of the document must be a `string`, unique, final and required.
### composite primary key
You can define a composite primary key which is composed from multiple properties of the document data.
```javascript
const mySchema = {
keyCompression: true, // set this to true, to enable the keyCompression
version: 0,
title: 'human schema with composite primary',
primaryKey: {
// where should the composed string be stored
key: 'id',
// fields that will be used to create the composed key
fields: [
'firstName',
'lastName'
],
// separator which is used to concat the fields values.
separator: '|'
},
type: 'object',
properties: {
id: {
type: 'string',
maxLength: 100 // <- the primary key must have set maxLength
},
firstName: {
type: 'string'
},
lastName: {
type: 'string'
}
},
required: [
'id',
'firstName',
'lastName'
]
};
```
You can then find a document by using the relevant parts to create the composite primaryKey:
```ts
// inserting with composite primary
await myRxCollection.insert({
// id, <- do not set the id, it will be filled by RxDB
firstName: 'foo',
lastName: 'bar'
});
// find by composite primary
const id = myRxCollection.schema.getPrimaryOfDocumentData({
firstName: 'foo',
lastName: 'bar'
});
const myRxDocument = myRxCollection.findOne(id).exec();
```
## Indexes
RxDB supports secondary indexes which are defined at the schema-level of the collection.
Indexes are only allowed on field types `string`, `integer` and `number`. Some RxStorages allow to use `boolean` fields as index.
Depending on the field type, you must have set some meta attributes like `maxLength` or `minimum`. This is required so that RxDB
is able to know the maximum string representation length of a field, which is needed to craft custom indexes in several `RxStorage` implementations.
:::note
RxDB will always append the `primaryKey` to all indexes to ensure a deterministic sort order of query results. You do not have to add the `primaryKey` to any index.
:::
### Index-example
```javascript
const schemaWithIndexes = {
version: 0,
title: 'human schema with indexes',
keyCompression: true,
primaryKey: 'id',
type: 'object',
properties: {
id: {
type: 'string',
maxLength: 100 // <- the primary key must have set maxLength
},
firstName: {
type: 'string',
maxLength: 100 // <- string-fields that are used as an index, must have set maxLength.
},
lastName: {
type: 'string'
},
active: {
type: 'boolean'
},
familyName: {
type: 'string'
},
balance: {
type: 'number',
// number fields that are used in an index, must have set minimum, maximum and multipleOf
minimum: 0,
maximum: 100000,
multipleOf: 0.01
},
creditCards: {
type: 'array',
items: {
type: 'object',
properties: {
cvc: {
type: 'number'
}
}
}
}
},
required: [
'id',
'active' // <- boolean fields that are used in an index must be required.
],
indexes: [
'firstName', // <- this will create a simple index for the `firstName` field
['active', 'firstName'], // <- this will create a compound-index for these two fields
'active'
]
};
```
# internalIndexes
When you use RxDB on the server-side, you might want to use internalIndexes to speed up internal queries. [Read more](./rx-server.md#server-only-indexes)
## attachments
To use attachments in the collection, you have to add the `attachments`-attribute to the schema. [See RxAttachment](./rx-attachment.md).
## default
Default values can only be defined for first-level fields.
Whenever you insert a document unset fields will be filled with default-values.
```javascript
const schemaWithDefaultAge = {
version: 0,
primaryKey: 'id',
type: 'object',
properties: {
id: {
type: 'string',
maxLength: 100 // <- the primary key must have set maxLength
},
firstName: {
type: 'string'
},
lastName: {
type: 'string'
},
age: {
type: 'integer',
default: 20 // <- default will be used
}
},
required: ['id']
};
```
## final
By setting a field to `final`, you make sure it cannot be modified later. Final fields are always required.
Final fields cannot be observed because they will not change.
Advantages:
- With final fields you can ensure that no-one accidentally modifies the data.
- When you enable the `eventReduce` algorithm, some performance-improvements are done.
```javascript
const schemaWithFinalAge = {
version: 0,
primaryKey: 'id',
type: 'object',
properties: {
id: {
type: 'string',
maxLength: 100 // <- the primary key must have set maxLength
},
firstName: {
type: 'string'
},
lastName: {
type: 'string'
},
age: {
type: 'integer',
final: true
}
},
required: ['id']
};
```
## Non allowed properties
The schema is not only used to validate objects before they are written into the database, but also used to map getters to observe and populate single fieldnames, key compression and other things. Therefore you can not use every schema which would be valid for the spec of [json-schema.org](http://json-schema.org/).
For example, fieldnames must match the regex `^[a-zA-Z](?:[[a-zA-Z0-9_]*]?[a-zA-Z0-9])?$` and `additionalProperties` is always set to `false`. But don't worry, RxDB will instantly throw an error when you pass an invalid schema into it.
Also the following class properties of `RxDocument` cannot be used as top level fields because they would clash when the RxDocument property is accessed:
```json
[
"collection",
"_data",
"_propertyCache",
"isInstanceOfRxDocument",
"primaryPath",
"primary",
"revision",
"deleted$",
"deleted$$",
"deleted",
"getLatest",
"$",
"$$",
"get$",
"get$$",
"populate",
"get",
"toJSON",
"toMutableJSON",
"update",
"incrementalUpdate",
"updateCRDT",
"putAttachment",
"putAttachmentBase64",
"getAttachment",
"allAttachments",
"allAttachments$",
"modify",
"incrementalModify",
"patch",
"incrementalPatch",
"_saveData",
"remove",
"incrementalRemove",
"close",
"deleted",
"synced"
]
```
## FAQ
How can I store a Date?
With RxDB you can only store plain JSON data inside of a document. You cannot store a JavaScript `new Date()` instance directly. This is for performance reasons and because `Date` is a mutable object where changing it at any time might cause strange problems that are hard to debug.
To store a date in RxDB, you have to define a string field with a `format` attribute:
```json
{
"type": "string",
"format": "date-time"
}
```
When storing the data you have to first transform your `Date` object into a string `Date.toISOString()`.
Because the `date-time` is sortable, you can do whatever query operations on that field and even use it as an index.
How to store schemaless data?
By design, RxDB requires that every collection has a schema. This means you cannot create a truly "schema-less" collection where top-level fields are unknown at schema creation time. RxDB must know about all fields of a document at the top level to perform validation, index creation, and other internal optimizations.
However, there is a way to store data of arbitrary structure at sub-fields. To do this, define a property with `type: "object"` in your schema. For example:
```ts
{
"version": 0,
"primaryKey": "id",
"type": "object",
"properties": {
"id": {
"type": "string",
"maxLength": 100
},
"myDynamicData": {
"type": "object"
// Here you can store any JSON data
// because it's an open object.
}
},
"required": ["id"]
}
```
Why does RxDB automatically set `additionalProperties: false` at the top level
RxDB automatically sets `additionalProperties: false` at the top level of a schema to ensure that all top-level fields are known in advance. This design choice offers several benefits:
- Prevents collisions with RxDocument class properties:
RxDB documents have built-in class methods (e.g., .toJSON, .save) at the top level. By forbidding unknown top-level properties, we avoid accidental naming collisions with these built-in methods.
- Avoids conflicts with user-defined ORM functions:
Developers can add custom [ORM methods](./orm.md) to RxDocuments. If top-level properties were unbounded, a property name could accidentally conflict with a method name, leading to unexpected behavior.
- Improves TypeScript typings:
If RxDB didn't know about all top-level fields, the document type would effectively become `any`. That means a simple typo like `myDocument.toJOSN()` would only be caught at runtime, not at build time. By disallowing unknown properties, TypeScript can provide strict typing and catch errors sooner.
Can't change the schema of a collection
When you make changes to the schema of a collection, you sometimes can get an error like
`Error: addCollections(): another instance created this collection with a different schema`.
This means you have created a collection before and added document-data to it.
When you now just change the schema, it is likely that the new schema does not match the saved documents inside of the collection.
This would cause strange bugs and would be hard to debug, so RxDB checks if your schema has changed and throws an error.
To change the schema in **production**-mode, do the following steps:
- Increase the `version` by 1
- Add the appropriate [migrationStrategies](https://pubkey.github.io/rxdb/migration-schema.html) so the saved data will be modified to match the new schema
Why does the top-level schema complain about a missing `_id` primary key field?
You encounter an error stating that the top-level schema is missing the `_id` primary key field during replication. RxDB requires every schema to explicitly define the primary key property. Other databases use an implicit `_id` field. You must add the `_id` property to your schema manually if your backend expects it. You declare `_id` as a string type and set it as the `primaryKey` in your schema definition.
In **development**-mode, the schema-change can be simplified by **one of these** strategies:
- Use the memory-storage so your db resets on restart and your schema is not saved permanently
- Call `removeRxDatabase('mydatabasename', RxStorage);` before creating a new RxDatabase-instance
- Add a timestamp as suffix to the database-name to create a new one each run like `name: 'heroesDB' + new Date().getTime()`
---
## Master Data - Create and Manage RxCollections
# RxCollection
A collection stores documents of the same type.
## Creating a Collection
To create one or more collections you need an [RxDatabase](./rx-database.md) object which has the `.addCollections()` method. Every collection needs a collection name and a valid [RxJsonSchema](./rx-schema.md). Other attributes are optional.
```js
const myCollections = await myDatabase.addCollections({
// key = collectionName
humans: {
schema: mySchema,
statics: {}, // (optional) ORM-functions for this collection
methods: {}, // (optional) ORM-functions for documents
attachments: {}, // (optional) ORM-functions for attachments
options: {}, // (optional) Custom parameters that might be used in plugins
migrationStrategies: {}, // (optional)
autoMigrate: true, // (optional) [default=true]
cacheReplacementPolicy: function(){}, // (optional) custom cache replacement policy
conflictHandler: function(){} // (optional) a custom conflict handler can be used
},
// you can create multiple collections at once
animals: {
// ...
}
});
```
### name
The name uniquely identifies the collection and should be used to refine the collection in the database. Two different collections in the same database can never have the same name. Collection names must match the following regex: `^[a-z][a-z0-9]*$`.
### schema
The schema defines how the documents of the collection are structured. RxDB uses a schema format, similar to [JSON schema](https://json-schema.org/). Read more about the RxDB schema format [here](./rx-schema.md).
### ORM-functions
With the parameters `statics`, `methods` and `attachments`, you can define ORM functions that are applied to each of these objects that belong to this collection. See [ORM/DRM](./orm.md).
### Migration
With the parameters `migrationStrategies` and `autoMigrate` you can specify how migration between different schema-versions should be done. [See Migration](./migration-schema.md).
## Get a collection from the database
To get an existing collection from the database, call the collection name directly on the database:
```javascript
// newly created collection
const collections = await db.addCollections({
heroes: {
schema: mySchema
}
});
const collection2 = db.heroes;
console.log(collections.heroes === collection2); //> true
```
## Functions
### Observe $
Calling this will return an [rxjs-Observable](https://rxjs.dev/guide/observable) which streams every change to data of this collection.
```js
myCollection.$.subscribe(changeEvent => console.dir(changeEvent));
// you can also observe single event-types with insert$ update$ remove$
myCollection.insert$.subscribe(changeEvent => console.dir(changeEvent));
myCollection.update$.subscribe(changeEvent => console.dir(changeEvent));
myCollection.remove$.subscribe(changeEvent => console.dir(changeEvent));
```
### insert()
Use this to insert new documents into the database. The collection will validate the schema and automatically encrypt any encrypted fields. Returns the new RxDocument.
```js
const doc = await myCollection.insert({
name: 'foo',
lastname: 'bar'
});
```
### insertIfNotExists()
The insertIfNotExists() method attempts to insert a new document into the collection only if a document with the same primary key does not already exist. This is useful for ensuring uniqueness without having to manually check for existing records before inserting or handling [conflicts](./transactions-conflicts-revisions.md).
Returns either the newly added [RxDocument](./rx-document.md) or the previous existing document.
```js
const doc = await myCollection.insertIfNotExists({
name: 'foo',
lastname: 'bar'
});
```
### bulkInsert()
When you have to insert many documents at once, use bulk insert. This is much faster than calling `.insert()` multiple times.
Returns an object with a `success` and `error` arrays.
```js
const result = await myCollection.bulkInsert([{
name: 'foo1',
lastname: 'bar1'
},
{
name: 'foo2',
lastname: 'bar2'
}]);
// > {
// success: [RxDocument, RxDocument],
// error: []
// }
```
:::note
`bulkInsert` will not fail on update conflicts and you cannot expect that on failure the other documents are not inserted. Also, the call to `bulkInsert()` will not throw if a single document errors because of validation errors. Instead it will return the error in the `.error` property of the returned object.
:::
### bulkRemove()
When you want to remove many documents at once, use bulk remove. Returns an object with a `success`- and `error`-array.
```js
const result = await myCollection.bulkRemove([
'primary1',
'primary2'
]);
// > {
// success: [RxDocument, RxDocument],
// error: []
// }
```
Instead of providing the document ids, you can also use the [RxDocument](./rx-document.md) instances. This can have better performance if your code knows them already at the moment of removing them:
```js
const result = await myCollection.bulkRemove([
myRxDocument1,
myRxDocument2,
/* ... */
]);
```
### upsert()
Inserts the document if it does not exist within the collection, otherwise it will overwrite it. Returns the new or overwritten RxDocument.
When the document already exists, any [inline attachments](./rx-attachment.md#inline-attachments-on-insert-and-upsert) in the upsert data are **merged** with existing attachments by default. Pass `{ deleteExistingAttachments: true }` as the second argument to replace all existing attachments instead.
```js
const doc = await myCollection.upsert({
name: 'foo',
lastname: 'bar2'
});
// with options
const doc2 = await myCollection.upsert(docData, { deleteExistingAttachments: true });
```
### bulkUpsert()
Same as `upsert()` but runs over multiple documents. Improves performance compared to running many `upsert()` calls.
Returns an `error` and a `success` array. Accepts an optional second argument for [upsert options](./rx-attachment.md#upsert-behavior-with-attachments).
```js
const docs = await myCollection.bulkUpsert([
{
name: 'foo',
lastname: 'bar2'
},
{
name: 'bar',
lastname: 'foo2'
}
]);
/**
* {
* success: [RxDocument, RxDocument]
* error: [],
* }
*/
```
### incrementalUpsert()
When you run many upsert operations on the same RxDocument in a very short timespan, you might get a `409 Conflict` error.
This means that you tried to run a `.upsert()` on the document, while the previous upsert operation was still running.
To prevent these types of errors, you can run incremental upsert operations.
The behavior is similar to [RxDocument.incrementalModify](./rx-document.md#incrementalModify).
```js
const docData = {
name: 'Bob', // primary
lastName: 'Kelso'
};
myCollection.upsert(docData);
myCollection.upsert(docData);
// -> throws because of parallel update to the same document
myCollection.incrementalUpsert(docData);
myCollection.incrementalUpsert(docData);
myCollection.incrementalUpsert(docData);
// wait until last upsert finished
await myCollection.incrementalUpsert(docData);
// -> works
```
### find()
To find documents in your collection, use this method. [See RxQuery.find()](./rx-query.md#find).
```js
// find all that are older than 18
const olderDocuments = await myCollection
.find()
.where('age')
.gt(18)
.exec(); // execute
```
### findOne()
This does basically what find() does, but it returns only a single document. You can pass a primary value to find a single document more easily.
To find documents in your collection, use this method. [See RxQuery.find()](./rx-query.md#findOne).
```js
// get document with name:foobar
myCollection.findOne({
selector: {
name: 'foo'
}
}).exec().then(doc => console.dir(doc));
// get document by primary, functionally identical to above query
myCollection.findOne('foo')
.exec().then(doc => console.dir(doc));
```
### findByIds()
Find many documents by their id (primary value). This has a way better performance than running multiple `findOne()` or a `find()` with a big `$or` selector.
Returns a `Map` where the primary key of the document is mapped to the document. Documents that do not exist or are deleted, will not be inside of the returned Map.
```js
const ids = [
'alice',
'bob',
/* ... */
];
const docsMap = await myCollection.findByIds(ids);
console.dir(docsMap); // Map(2)
```
:::note
The `Map` returned by `findByIds` is not guaranteed to return elements in the same order as the list of ids passed to it.
:::
### exportJSON()
Use this function to create a JSON export from every document in the collection.
Before `exportJSON()` and `importJSON()` can be used, you have to add the `json-dump` plugin.
```javascript
import { addRxPlugin } from 'rxdb';
import { RxDBJsonDumpPlugin } from 'rxdb/plugins/json-dump';
addRxPlugin(RxDBJsonDumpPlugin);
```
```js
myCollection.exportJSON()
.then(json => console.dir(json));
```
### importJSON()
To import the JSON dump into your collection, use this function.
```js
// import the dump to the database
myCollection.importJSON(json)
.then(() => console.log('done'));
```
Note that importing will fire events for each inserted document.
### remove()
Removes all known data of the collection and its previous versions.
This removes the documents, the schemas, and older schemaVersions.
```js
await myCollection.remove();
// collection is now removed and can be re-created
```
### close()
Removes the collection's object instance from the [RxDatabase](./rx-database.md). This is to free up memory and stop all observers and replications. It will not delete the collection's data. When you create the collection again with `database.addCollections()`, the newly added collection will still have all data.
```js
await myCollection.close();
```
### onClose / onRemove()
With these you can add a function that is run when the collection was closed or removed.
This works even across multiple browser tabs so you can detect when another tab removes the collection
and your application can behave accordingly.
```js
await myCollection.onClose(() => console.log('I am closed'));
await myCollection.onRemove(() => console.log('I am removed'));
```
### isRxCollection
Returns true if the given object is an instance of RxCollection. Returns false if not.
```js
const is = isRxCollection(myObj);
```
## FAQ
When I reload the browser window, will my collections still be in the database?
No, the javascript instance of the collections will not automatically load into the database on page reloads.
You have to call the `addCollections()` method each time you create your database. This will create the JavaScript object instance of the RxCollection so that you can use it in the RxDatabase. The persisted data will automatically be available in your RxCollection each time you create it.
How to remove the limit of 13 collections?
In the open-source version of RxDB, the amount of RxCollections that can exist in parallel is limited to `13`.
To remove this limit, you can purchase the [Premium Plugins](/premium/) and call the `setPremiumFlag()` function before creating a database:
```ts
import { setPremiumFlag } from 'rxdb-premium/plugins/shared';
setPremiumFlag();
```
---
## RxDocument
An RxDocument is an object which represents the data of a single JSON document stored in a [collection](./rx-collection.md). It can be compared to a single record in a relational database table. You get an `RxDocument` either as return on inserts/updates, or as result-set of [queries](./rx-query.md).
RxDB works on RxDocuments instead of plain JSON data to have more convenient operations on the documents. Also Documents that are fetched multiple times by different queries or operations are automatically de-duplicated by RxDB in memory.
## insert
To insert a document into a collection, you have to call the collection's `.insert()` function.
```js
await myCollection.insert({
name: 'foo',
lastname: 'bar'
});
```
## find
To find documents in a collection, you have to call the collection's `.find()` function. [See RxQuery](./rx-query.md).
```js
const docs = await myCollection.find().exec(); // <- find all documents
```
## Functions
### get()
This will get a single field of the document. If the field is encrypted, it will be automatically decrypted before returning.
```js
const name = myDocument.get('name'); // returns the name
// OR
const name = myDocument.name;
```
### get$()
This function returns an observable of the given path's value.
The current value of this path will be emitted each time the document changes.
```js
// get the live-updating value of 'name'
var isName;
myDocument.get$('name')
.subscribe(newName => {
isName = newName;
});
await myDocument.incrementalPatch({name: 'foobar2'});
console.dir(isName); // isName is now 'foobar2'
// OR
myDocument.name$
.subscribe(newName => {
isName = newName;
});
```
### proxy-get
All properties of an `RxDocument` are assigned as getters so you can also directly access values instead of using the get()-function.
```js
// Identical to myDocument.get('name');
var name = myDocument.name;
// Can also get nested values.
var nestedValue = myDocument.whatever.nestedfield;
// Also usable with observables:
myDocument.firstName$.subscribe(newName => console.log('name is: ' + newName));
// > 'name is: Stefe'
await myDocument.incrementalPatch({firstName: 'Steve'});
// > 'name is: Steve'
```
### update()
Updates the document based on the [Mongo update syntax](https://docs.mongodb.com/manual/reference/operator/update-field/), based on the [mingo library](https://github.com/kofrasa/mingo#updating-documents).
```js
/**
* If not done before, you have to add the update plugin.
*/
import { addRxPlugin } from 'rxdb';
import { RxDBUpdatePlugin } from 'rxdb/plugins/update';
addRxPlugin(RxDBUpdatePlugin);
await myDocument.update({
$inc: {
age: 1 // increases age by 1
},
$set: {
firstName: 'foobar' // sets firstName to foobar
}
});
```
### modify()
Updates a document's data based on a function that mutates the current data and returns the new value.
```js
const changeFunction = (oldData) => {
oldData.age = oldData.age + 1;
oldData.name = 'foooobarNew';
return oldData;
}
await myDocument.modify(changeFunction);
console.log(myDocument.name); // 'foooobarNew'
```
### patch()
Overwrites the given attributes in the document's data.
```js
await myDocument.patch({
name: 'Steve',
age: undefined // setting an attribute to undefined will remove it
});
console.log(myDocument.name); // 'Steve'
```
### Prevent conflicts with the incremental methods
Making a normal change to the non-latest version of an `RxDocument` will lead to a `409 CONFLICT` error because RxDB
uses [revision checks](./transactions-conflicts-revisions.md) instead of transactions.
To make a change to a document, no matter what the current state is, you can use the `incremental` methods:
```js
// update
await myDocument.incrementalUpdate({
$inc: {
age: 1 // increases age by 1
}
});
// modify
await myDocument.incrementalModify(docData => {
docData.age = docData.age + 1;
return docData;
});
// patch
await myDocument.incrementalPatch({
age: 100
});
// remove
await myDocument.incrementalRemove({
age: 100
});
```
### getLatest()
Returns the latest known state of the `RxDocument`.
```js
const myDocument = await myCollection.findOne('foobar').exec();
const docAfterEdit = await myDocument.incrementalPatch({
age: 10
});
const latestDoc = myDocument.getLatest();
console.log(docAfterEdit === latestDoc); // > true
```
### Observe $
Calling this will return an [RxJS Observable](https://rxjs.dev/guide/observable) which emits the current newest state of the RxDocument.
```js
// get all changeEvents
myDocument.$
.subscribe(currentRxDocument => console.dir(currentRxDocument));
```
### remove()
This removes the document from the collection. Notice that this will not purge the document from the store but set `_deleted:true` so that it will be no longer returned on queries.
To fully purge a document, use the [cleanup plugin](./cleanup.md).
```js
myDocument.remove();
```
### Remove and update in a single atomic operation
Sometimes you want to change a document's value and also remove it in the same operation. For example this can be useful when you use [replication](./replication.md) and want to set a `deletedAt` timestamp. Then you might have to ensure that setting this timestamp and deleting the document happens in the same atomic operation.
To do this the modifying operations of a document accept setting the `_deleted` field. For example:
```ts
// update() and remove()
await doc.update({
$set: {
deletedAt: new Date().getTime(),
_deleted: true
}
});
// modify() and remove()
await doc.modify(data => {
data.age = 1;
data._deleted = true;
return data;
});
```
### deleted$
Emits a boolean value, depending on whether the RxDocument is deleted or not.
```js
let lastState = null;
myDocument.deleted$.subscribe(state => lastState = state);
console.log(lastState);
// false
await myDocument.remove();
console.log(lastState);
// true
```
### get deleted
A getter to get the current value of `deleted$`.
```js
console.log(myDocument.deleted);
// false
await myDocument.remove();
console.log(myDocument.deleted);
// true
```
### toJSON()
Returns the document's data as plain JSON object. This will return an **immutable** object. To get something that can be modified, use `toMutableJSON()` instead.
```js
const json = myDocument.toJSON();
console.dir(json);
/* { passportId: 'h1rg9ugdd30o',
firstName: 'Carolina',
lastName: 'Gibson',
age: 33 ...
*/
```
You can also set `withMetaFields: true` to get additional meta fields like the revision, attachments or the deleted flag.
```js
const json = myDocument.toJSON(true);
console.dir(json);
/* { passportId: 'h1rg9ugdd30o',
firstName: 'Carolina',
lastName: 'Gibson',
_deleted: false,
_attachments: { ... },
_rev: '1-aklsdjfhaklsdjhf...'
*/
```
### toMutableJSON()
Same as `toJSON()` but returns a deep cloned object that can be mutated afterwards.
Remember that deep cloning is expensive and should only be done when necessary.
```js
const json = myDocument.toMutableJSON();
json.firstName = 'Alice'; // The returned document can be mutated
```
:::note All methods of RxDocument are bound to the instance
When you get a method from a `RxDocument`, the method is automatically bound to the document's instance. This means you do not have to use things like `myMethod.bind(myDocument)` like you would do in jsx.
:::
### isRxDocument
Returns true if the given object is an instance of RxDocument. Returns false if not.
```js
const is = isRxDocument(myObj);
```
## Document Lifetime and Immutability
**RxDocument instances are immutable.** Each instance represents a snapshot of the document at the time it was fetched or last written. Modifying a document does not update existing instances of it - it creates a new `RxDocument` instance with the updated data. The old instance retains its original data.
```js
const doc = await myCollection.findOne('foobar').exec();
console.log(doc.age); // 10
await doc.incrementalPatch({ age: 20 });
// The original instance still has the old data
console.log(doc.age); // 10
// Use getLatest() to get the updated state
console.log(doc.getLatest().age); // 20
```
**RxDB de-duplicates document instances.** When the same document is fetched multiple times without any writes in between, RxDB returns the same instance to save memory. Once a write occurs, subsequent fetches return a new instance reflecting the updated state.
**Calling non-incremental write methods on an outdated instance throws a `CONFLICT` error.** If you hold a reference to a document and another operation modifies that document in the meantime, calling `.patch()`, `.update()`, or `.modify()` on the outdated instance will fail with a conflict error. See [Transactions, Conflicts and Revisions](./transactions-conflicts-revisions.md) for details on how RxDB handles conflicts.
To avoid this, either:
- Use the [incremental methods](#prevent-conflicts-with-the-incremental-methods) (`incrementalPatch`, `incrementalModify`, `incrementalUpdate`) which always fetch the latest state before applying changes.
- Call `getLatest()` to get the current state before writing.
- Re-query the collection to get a fresh document.
**How long to keep a reference to an `RxDocument`.** Treat an `RxDocument` like plain JSON data - it is a snapshot valid at the time of retrieval. RxDB manages query result caching internally via [event-reduce](./rx-query.md), so you do not need to cache documents yourself. For components that display document data and need live updates, subscribe to the document's `$` observable instead of holding a static reference.
---
## RxQuery
To find documents inside of an [RxCollection](./rx-collection.md), RxDB uses the RxQuery interface that handles all query operations: it serves as the main interface for fetching documents, relies on a MongoDB-like [Mango Query Syntax](https://github.com/cloudant/mango), and provides three types of queries: [find()](#find), [findOne()](#findone) and [count()](#count). By caching and de-duplicating results, RxQuery ensures efficient in-memory handling, and when queries are observed or re-run, the [EventReduce algorithm](https://github.com/pubkey/event-reduce) speeds up updates for a fast real-time experience and queries that run more than once.
## find()
To create a basic `RxQuery`, call `.find()` on a collection and insert selectors. The result-set of normal queries is an array with documents.
```js
// find all that are older then 18
const query = myCollection
.find({
selector: {
age: {
$gt: 18
}
}
});
```
## findOne()
A findOne-query has only a single [RxDocument](./rx-document.md) or `null` as result-set.
```js
// find alice
const query = myCollection
.findOne({
selector: {
name: 'alice'
}
});
```
```js
// find the youngest one
const query = myCollection
.findOne({
selector: {},
sort: [
{age: 'asc'}
]
});
```
```js
// find one document by the primary key
const query = myCollection.findOne('foobar');
```
## exec()
Returns a `Promise` that resolves with the result-set of the query.
```js
const query = myCollection.find();
const results = await query.exec();
console.dir(results); // > [RxDocument,RxDocument,RxDocument..]
```
On `.findOne()` queries, you can call `.exec(true)` to ensure your document exists and to make TypeScript handling easier:
```ts
// docOrUndefined can be the type RxDocument or null which then has to be handled to be typesafe.
const docOrUndefined = await myCollection.findOne().exec();
// with .exec(true), it will throw if the document cannot be found and always have the type RxDocument
const doc = await myCollection.findOne().exec(true);
```
## Observe $
An `BehaviorSubject` [see](https://medium.com/@luukgruijs/understanding-rxjs-behaviorsubject-replaysubject-and-asyncsubject-8cc061f1cfc0) that always has the current result-set as value.
This is extremely helpful when used together with UIs that should always show the same state as what is written in the database.
```js
const query = myCollection.find();
const querySub = query.$.subscribe(results => {
console.log('got results: ' + results.length);
});
// > 'got results: 5' // BehaviorSubjects emit on subscription
await myCollection.insert({/* ... */}); // insert one
// > 'got results: 6' // $.subscribe() was called again with the new results
// stop watching this query
querySub.unsubscribe()
```
## update()
Runs an [update](./rx-document.md#update) on every RxDocument of the query-result.
```js
// to use the update() method, you need to add the update plugin.
import { RxDBUpdatePlugin } from 'rxdb/plugins/update';
addRxPlugin(RxDBUpdatePlugin);
const query = myCollection.find({
selector: {
age: {
$gt: 18
}
}
});
await query.update({
$inc: {
age: 1 // increases age of every found document by 1
}
});
```
## patch() / incrementalPatch()
Runs the [RxDocument.patch()](./rx-document.md#patch) function on every RxDocument of the query result.
```js
const query = myCollection.find({
selector: {
age: {
$gt: 18
}
}
});
await query.patch({
age: 12 // set the age of every found to 12
});
```
## modify() / incrementalModify()
Runs the [RxDocument.modify()](./rx-document.md#modify) function on every RxDocument of the query result.
```js
const query = myCollection.find({
selector: {
age: {
$gt: 18
}
}
});
await query.modify((docData) => {
docData.age = docData.age + 1; // increases age of every found document by 1
return docData;
});
```
## remove() / incrementalRemove()
Deletes all found documents. Returns a promise which resolves to the deleted documents.
```javascript
// All documents where the age is less than 18
const query = myCollection.find({
selector: {
age: {
$lt: 18
}
}
});
// Remove the documents from the collection
const removedDocs = await query.remove();
```
## doesDocumentDataMatch()
Returns `true` if the given document data matches the query.
```js
const documentData = {
id: 'foobar',
age: 19
};
myCollection.find({
selector: {
age: {
$gt: 18
}
}
}).doesDocumentDataMatch(documentData); // > true
myCollection.find({
selector: {
age: {
$gt: 20
}
}
}).doesDocumentDataMatch(documentData); // > false
```
## Query Builder Plugin
To use chained query methods, you can also use the `query-builder` plugin.
```ts
// add the query builder plugin
import { addRxPlugin } from 'rxdb';
import { RxDBQueryBuilderPlugin } from 'rxdb/plugins/query-builder';
addRxPlugin(RxDBQueryBuilderPlugin);
// now you can use chained query methods
const query = myCollection.find().where('age').gt(18);
const result = await query.exec();
```
## Query Examples
Here some examples to learn quickly how to write queries without reading the docs.
- [Pouch-find-docs](https://github.com/pouchdb/pouchdb/blob/master/packages/node_modules/pouchdb-find/README.md) - learn how to use mango-queries
- [mquery-docs](https://github.com/aheckmann/mquery/blob/master/README.md) - learn how to use chained-queries
```js
// directly pass search-object
myCollection.find({
selector: {
name: { $eq: 'foo' }
}
})
.exec().then(documents => console.dir(documents));
/*
* find by using sql equivalent '%like%' syntax
* This example will e.g. match 'foo' but also 'fifoo' or 'foofa' or 'fifoofa'
* Notice that in RxDB queries, a regex is represented as a $regex string with the $options parameter for flags.
* Using a RegExp instance is not allowed because they are not JSON.stringify()-able and also
* RegExp instances are mutable which could cause undefined behavior when the RegExp is mutated
* after the query was parsed.
*/
myCollection.find({
selector: {
name: { $regex: '.*foo.*' }
}
})
.exec().then(documents => console.dir(documents));
// find using a composite statement eg: $or
// This example checks where name is either foo or if name is not existent on the document
myCollection.find({
selector: { $or: [ { name: { $eq: 'foo' } }, { name: { $exists: false } }] }
})
.exec().then(documents => console.dir(documents));
// do a case insensitive search
// This example will match 'foo' or 'FOO' or 'FoO' etc...
myCollection.find({
selector: { name: { $regex: '^foo$', $options: 'i' } }
})
.exec().then(documents => console.dir(documents));
// chained queries
myCollection.find().where('name').eq('foo')
.exec().then(documents => console.dir(documents));
```
:::note RxDB will always append the primary key to the sort parameters
For several performance optimizations, like the [EventReduce algorithm](https://github.com/pubkey/event-reduce), RxDB expects all queries to return a deterministic sort order that does not depend on the insert order of the documents. To ensure a deterministic ordering, RxDB will always append the primary key as last sort parameter to all queries and to all indexes.
This works in contrast to most other databases where a query without sorting would return the documents in the order in which they had been inserted to the database.
:::
## Setting a specific index
By default, the query will be sent to the RxStorage, where a query planner will determine which one of the available indexes must be used.
But the query planner cannot know everything and sometimes will not pick the most optimal index.
To improve query performance, you can specify which index must be used, when running the query.
```ts
const query = myCollection
.findOne({
selector: {
age: {
$gt: 18
},
gender: {
$eq: 'm'
}
},
/**
* Because the developer knows that 50% of the documents are 'male',
* but only 20% are below age 18,
* it makes sense to enforce using the ['gender', 'age'] index to improve performance.
* This could not be known by the query planner which might have chosen ['age', 'gender'] instead.
*/
index: ['gender', 'age']
});
```
## Count
When you only need the amount of documents that match a query, but you do not need the document data itself, you can use a count query for **better performance**.
The performance difference compared to a normal query differs depending on which [RxStorage](./rx-storage.md) implementation is used.
```ts
const query = myCollection.count({
selector: {
age: {
$gt: 18
}
}
// 'limit' and 'skip' MUST NOT be set for count queries.
});
// get the count result once
const matchingAmount = await query.exec(); // > number
// observe the result
query.$.subscribe(amount => {
console.log('Currently has ' + amount + ' documents');
});
```
:::note
Count queries have a better performance than normal queries because they do not have to fetch the full document data out of the storage. Therefore it is **not** possible to run a `count()` query with a selector that requires fetching and comparing the document data. So if your query selector **does not** fully match an index of the schema, it is not allowed to run it. These queries would have no performance benefit compared to normal queries but have the tradeoff of not using the fetched document data for caching.
:::
```ts
/**
* The following will throw an error because
* the count operation cannot run on any specific index range
* because the $regex operator is used.
*/
const query = myCollection.count({
selector: {
age: {
$regex: 'foobar'
}
}
});
/**
* The following will throw an error because
* the count operation cannot run on any specific index range
* because there is no ['age' ,'otherNumber'] index
* defined in the schema.
*/
const query = myCollection.count({
selector: {
age: {
$gt: 20
},
otherNumber: {
$gt: 10
}
}
});
```
If you want to count these kinds of queries, you should do a normal query instead and use the length of the result set as counter. This has the same performance as running a non-fully-indexed count which has to fetch all document data from the database and run a query matcher.
```ts
// get count manually once
const resultSet = await myCollection.find({
selector: {
age: {
$regex: 'foobar'
}
}
}).exec();
const count = resultSet.length;
// observe count manually
const count$ = myCollection.find({
selector: {
age: {
$regex: 'foobar'
}
}
}).$.pipe(
map(result => result.length)
);
/**
* To allow non-fully-indexed count queries,
* you can also specify that by setting allowanceSlowCount: true
* when creating the database.
*/
const database = await createRxDatabase({
name: 'mydatabase',
allowSlowCount: true, // set this to true [default=false]
/* ... */
});
```
### `allowSlowCount`
To allow non-fully-indexed count queries, you can also specify that by setting `allowSlowCount: true` when creating the database.
Doing this is mostly not wanted, because it would run the counting on the storage without having the document stored in the RxDB document cache.
This is only recommended if the RxStorage is running remotely like in a WebWorker and you do not always want to send the document-data between the worker and the main thread. In this case you might only need the count-result instead to save performance.
## RxQuery instances are immutable
Because RxDB is a reactive database, we can do heavy performance-optimisation on query-results which change over time. To be able to do this, RxQueries have to be immutable.
This means, when you have a `RxQuery` and run a `.where()` on it, the original RxQuery object is not changed. Instead the where-function returns a new `RxQuery`-Object with the changed where-field. Keep this in mind if you create RxQueries and change them afterwards.
Example:
```javascript
const queryObject = myCollection.find().where('age').gt(18);
// Creates a new RxQuery object, does not modify previous one
queryObject.sort('name');
const results = await queryObject.exec();
console.dir(results); // result-documents are not sorted by name
const queryObjectSort = queryObject.sort('name');
const results = await queryObjectSort.exec();
console.dir(results); // result-documents are now sorted
```
### isRxQuery
Returns true if the given object is an instance of RxQuery. Returns false if not.
```js
const is = isRxQuery(myObj);
```
## Design Decisions
Like most other noSQL-Databases, RxDB uses the [mango-query-syntax](https://github.com/cloudant/mango) similar to MongoDB and others.
- We use the JSON-based Mango Query Syntax because:
- Mango Queries work better with TypeScript compared to SQL strings.
- Mango Queries are composable and easy to transform by code without joining SQL strings.
- Queries can be run very fast and efficient with only a minimal query planner to plan the best indexes and operations.
- NoSQL queries can be optimized with the [EventReduce](https://github.com/pubkey/event-reduce) algorithm to improve performance of observed and cached queries.
## FAQ
Can I specify which document fields are returned by an RxDB query?
No, RxDB does not support partial document retrieval. Because RxDB is a client-side database with limited memory, it caches and de-duplicates entire documents across multiple queries. Even if you only need a few fields, most storages must still fetch the entire JSON data, so subselecting fields would not significantly improve performance. Therefore, RxDB always returns full documents. If you only need certain fields, you can filter them out in your application code or consider storing just the necessary data in a separate collection.
Why doesn't RxDB support aggregations on queries?
RxDB runs entirely on the client side. Any "aggregation" or data processing you might do within RxDB would still happen in the same JavaScript environment as your application code. Therefore, there's no real performance advantage or difference between doing the aggregation in RxDB vs. doing it in your own code after fetching the data. As a result, RxDB doesn't provide built-in aggregation methods. Instead, just query the documents you need and perform any calculations directly in your app's code.
Why does RxDB not support cross-collection queries?
RxDB is a client-side database and does not provide built-in cross-collection queries or transactions. Instead, you can execute multiple queries in your JavaScript code and combine their results as needed. Because everything runs in the same environment, this approach offers the same performance you would get if cross-collection queries were built in - without the added complexity.
Why Doesn't RxDB Support Case-Insensitive Search?
RxDB relies on various storage engines as its backend, and these storage engines generally do not support case-insensitive search natively, like [IndexedDB](./rx-storage-indexeddb.md) or [FoundationDB](./rx-storage-foundationdb.md). This limitation arises from the design of these engines, which prioritize efficiency and flexibility for specific types of queries rather than universal features like case-insensitivity. Although RxDB does not offer built-in support for case-insensitive search, there are two common workarounds:
- **Store Data in a Meta-Field for Lowercase Search**: To enable case-insensitive search, you can store an additional field in your documents where the relevant text data is preprocessed and saved in lowercase.
```ts
const document = {
name: 'John Doe',
nameLowercase: 'john doe' // Meta-field
};
await myCollection.insert(document);
const query = myCollection.find({
selector: {
nameLowercase: { $eq: 'john doe' }
}
});
```
- **Use a Regex Query**: Regular expressions can perform case-insensitive searches. For example:
```ts
const query = myCollection.find({
selector: {
name: { $regex: '^john doe$', $options: 'i' } // Case-insensitive regex
}
});
```
However, this method has a significant downside: regex queries often cannot leverage indexes efficiently. As a result, they may be slower, especially for large datasets.
---
## βοΈ RxStorage Layer - Choose the Perfect RxDB Storage for Every Use Case
# RxStorage
RxDB is not a self-contained database. Instead the data is stored in an implementation of the [RxStorage interface](https://github.com/pubkey/rxdb/blob/master/src/types/rx-storage.interface.d.ts). This allows you to **switch out** the underlying data layer, depending on the JavaScript environment and performance requirements. For example you can use the SQLite storage for a capacitor app or you can use the LocalStorage RxStorage to store data in localstorage in a browser-based application. There are also storages for other JavaScript runtimes like Node.js, React-Native, NativeScript and more.
## Quick Recommendations
- In the Browser: Use the [LocalStorage](./rx-storage-localstorage.md) storage for simple setup and small build size. For bigger datasets, use either the [dexie.js storage](./rx-storage-dexie.md) (free) or the [IndexedDB RxStorage](./rx-storage-indexeddb.md) if you have [π premium access](/premium/) which is a bit faster and has a smaller build size.
- In [Electron](./electron-database.md) and [ReactNative](./react-native-database.md): Use the [SQLite RxStorage](./rx-storage-sqlite.md) if you have [π premium access](/premium/) or the [trial-SQLite RxStorage](./rx-storage-sqlite.md) for tryouts.
- In Capacitor: Use the [SQLite RxStorage](./rx-storage-sqlite.md) if you have [π premium access](/premium/), otherwise use the [localStorage](./rx-storage-localstorage.md) storage.
## Configuration Examples
The RxStorage layer of RxDB is very flexible. Here are some examples on how to configure more complex settings:
### Storing much data in a browser securely
Lets say you build a browser app that needs to store a big amount of data as securely as possible. Here we can use a combination of the storages (encryption, IndexedDB, compression, schema-checks) that increase security and reduce the stored data size.
We use the schema-validation on the top level to ensure schema-errors are clearly readable and do not contain [encrypted](./encryption.md)/[compressed](./key-compression.md) data. The encryption is used inside of the compression because encryption of compressed data is more efficient.
```ts
import { wrappedValidateAjvStorage } from 'rxdb/plugins/validate-ajv';
import { wrappedKeyCompressionStorage } from 'rxdb/plugins/key-compression';
import { wrappedKeyEncryptionCryptoJsStorage } from 'rxdb/plugins/encryption-crypto-js';
import { getRxStorageIndexedDB } from 'rxdb-premium/plugins/storage-indexeddb';
const myDatabase = await createRxDatabase({
storage: wrappedValidateAjvStorage({
storage: wrappedKeyCompressionStorage({
storage: wrappedKeyEncryptionCryptoJsStorage({
storage: getRxStorageIndexedDB()
})
})
})
});
```
### High query Load
Also we can utilize a combination of storages to create a database that is optimized to run complex queries on the data really fast. Here we use the sharding storage together with the worker storage. This allows to run queries in parallel multithreading instead of a single JavaScript process. Because the worker initialization can slow down the initial page load, we also use the [localstorage-meta-optimizer](./rx-storage-localstorage-meta-optimizer.md) to improve initialization time.
```ts
import { getRxStorageSharding } from 'rxdb-premium/plugins/storage-sharding';
import { getRxStorageWorker } from 'rxdb-premium/plugins/storage-worker';
import { getRxStorageIndexedDB } from 'rxdb-premium/plugins/storage-indexeddb';
import { getLocalstorageMetaOptimizerRxStorage } from 'rxdb-premium/plugins/storage-localstorage-meta-optimizer';
const myDatabase = await createRxDatabase({
storage: getLocalstorageMetaOptimizerRxStorage({
storage: getRxStorageSharding({
storage: getRxStorageWorker({
workerInput: 'path/to/worker.js',
storage: getRxStorageIndexedDB()
})
})
})
});
```
### Low Latency on Writes and Simple Reads
Here we create a storage configuration that is optimized to have a low latency on simple reads and writes. It uses the memory-mapped storage to fetch and store data in memory. For persistence the OPFS storage is used in the main thread which has lower latency for fetching big chunks of data when at initialization the data is loaded from disk into memory. We do not use workers because sending data from the main thread to workers and backwards would increase the latency.
```ts
import { getLocalstorageMetaOptimizerRxStorage } from 'rxdb-premium/plugins/storage-localstorage-meta-optimizer';
import { getMemoryMappedRxStorage } from 'rxdb-premium/plugins/storage-memory-mapped';
import { getRxStorageOPFSMainThread } from 'rxdb-premium/plugins/storage-worker';
const myDatabase = await createRxDatabase({
storage: getLocalstorageMetaOptimizerRxStorage({
storage: getMemoryMappedRxStorage({
storage: getRxStorageOPFSMainThread()
})
})
});
```
## All RxStorage Implementations List
### Memory
A storage that stores the data as plain data in the memory of the JavaScript process. Really fast and can be used in all environments. [Read more](./rx-storage-memory.md)
### LocalStorage
The localStorage based storage stores the data inside of a browsers [localStorage API](./articles/localstorage.md). It is the easiest to set up and has a small bundle size. **If you are new to RxDB, you should start with the LocalStorage RxStorage**. [Read more](./rx-storage-localstorage.md)
### π IndexedDB
The IndexedDB `RxStorage` is based on plain IndexedDB. For most use cases, this has the best performance together with the OPFS storage. [Read more](./rx-storage-indexeddb.md)
### π OPFS
The OPFS `RxStorage` is based on the File System Access API. This has the best performance of all other non-in-memory storage, when RxDB is used inside of a browser. [Read more](./rx-storage-opfs.md)
### π Filesystem Node
The Filesystem Node storage is best suited when you use RxDB in a Node.js process or with [electron.js](./electron.md). [Read more](./rx-storage-filesystem-node.md)
### Storage Wrapper Plugins
#### π Worker
The worker RxStorage is a wrapper around any other RxStorage which allows to run the storage in a WebWorker (in browsers) or a Worker Thread (in Node.js). By doing so, you can take CPU load from the main process and move it into the worker's process which can improve the perceived performance of your application. [Read more](./rx-storage-worker.md)
#### π SharedWorker
The SharedWorker RxStorage is a wrapper around any other RxStorage which allows to run the storage in a SharedWorker (only in browsers). By doing so, you can take CPU load from the main process and move it into the worker's process which can improve the perceived performance of your application. [Read more](./rx-storage-shared-worker.md)
#### Remote
The Remote RxStorage is made to use a remote storage and communicate with it over an asynchronous message channel. The remote part could be on another JavaScript process or even on a different host machine. Mostly used internally in other storages like Worker or Electron-ipc. [Read more](./rx-storage-remote.md)
#### π Sharding
On some `RxStorage` implementations (like IndexedDB), a huge performance improvement can be done by sharding the documents into multiple database instances. With the sharding plugin you can wrap any other `RxStorage` into a sharded storage. [Read more](./rx-storage-sharding.md)
#### π Memory Mapped
The memory-mapped [RxStorage](./rx-storage.md) is a wrapper around any other RxStorage. The wrapper creates an in-memory storage that is used for query and write operations. This memory instance stores its data in an underlying storage for persistence.
The main reason to use this is to improve query/write performance while still having the data stored on disk. [Read more](./rx-storage-memory-mapped.md)
#### π Localstorage Meta Optimizer
The [RxStorage](./rx-storage.md) Localstorage Meta Optimizer is a wrapper around any other RxStorage. The wrapper uses the original RxStorage for normal collection documents. But to optimize the initial page load time, it uses [localstorage](./articles/localstorage.md) to store the plain key-value metadata that RxDB needs to create databases and collections. This plugin can only be used in browsers. [Read more](./rx-storage-localstorage-meta-optimizer.md)
#### Electron IpcRenderer & IpcMain
To use RxDB in [electron](./electron-database.md), it is recommended to run the RxStorage in the main process and the RxDatabase in the renderer processes. With the rxdb electron plugin you can create a remote RxStorage and consume it from the renderer process. [Read more](./electron.md)
### Third Party based Storages
#### π SQLite
The SQLite storage has great performance when RxDB is used on **Node.js**, **Electron**, **React Native**, **Cordova** or **Capacitor**. [Read more](./rx-storage-sqlite.md)
#### Dexie.js
The Dexie.js based storage is based on the Dexie.js IndexedDB wrapper library. [Read more](./rx-storage-dexie.md)
#### MongoDB
To use RxDB on the server side, the MongoDB RxStorage provides a way of having a secure, scalable and performant storage based on the popular MongoDB NoSQL database. [Read more](./rx-storage-mongodb.md)
#### DenoKV
To use RxDB in Deno. The DenoKV RxStorage provides a way of having a secure, scalable and performant storage based on the Deno Key Value Store. [Read more](./rx-storage-denokv.md)
#### FoundationDB
To use RxDB on the server side, the FoundationDB RxStorage provides a way of having a secure, fault-tolerant and performant storage. [Read more](./rx-storage-foundationdb.md)
---
## RxDB LocalStorage - The Easiest Way to Persist Data in Your Web App
import {Steps} from '@site/src/components/steps';
# RxStorage LocalStorage
RxDB can persist data in various ways. One of the simplest methods is using the browserβs built-in [LocalStorage](./articles/localstorage.md). This storage engine allows you to store and retrieve [RxDB documents](./rx-document.md) directly from the browser without needing additional plugins or libraries.
> **Recommended Default for using RxDB in the Browser**
>
> We highly recommend using LocalStorage for a quick and easy RxDB setup, especially when you want a minimal project configuration. For professional projects, the [IndexedDB RxStorage](./rx-storage-indexeddb.md) is recommended in most cases.
## Key Benefits
1. **Simplicity**: No complicated configurations or external dependencies - LocalStorage is already built into the browser.
2. **Fast for small Datasets**: Writing and Reading small sets of data from localStorage is really fast as shown in [these benchmarks](./articles/localstorage-indexeddb-cookies-opfs-sqlite-wasm.md#performance-comparison).
4. **Ease of Setup**: Just import the plugin, import it, and pass `getRxStorageLocalstorage()` into `createRxDatabase()`. Thatβs it!
## Limitations
While LocalStorage is the easiest way to get started, it does come with some constraints:
1. **Limited Storage Capacity**: Browsers often limit LocalStorage to around [5 MB per domain](./articles/localstorage.md#understanding-the-limitations-of-local-storage), though exact limits vary.
2. **Synchronous Access**: LocalStorage operations block the main thread. This is usually fine for small amounts of data but can cause performance bottlenecks with heavier use.
Despite these limitations, LocalStorage remains a great default option for smaller projects, prototypes, or cases where you need the absolute simplest way to persist data in the browser.
## How to use the LocalStorage RxStorage with RxDB
### Import the Storage
```ts
import { createRxDatabase } from 'rxdb/plugins/core';
import { getRxStorageLocalstorage } from 'rxdb/plugins/storage-localstorage';
```
### Create a Database
```ts
const db = await createRxDatabase({
name: 'exampledb',
storage: getRxStorageLocalstorage()
});
```
### Add a Collection
```ts
await db.addCollections({
tasks: {
schema: {
title: 'tasks schema',
version: 0,
primaryKey: 'id',
type: 'object',
properties: {
id: { type: 'string' },
title: { type: 'string' },
done: { type: 'boolean' }
},
required: ['id', 'title', 'done']
}
}
});
```
### Insert a document
```ts
await db.tasks.insert({ id: 'task-01', title: 'Get started with RxDB', done: false });
```
### Query documents
```ts
const nonDoneTasks = await db.tasks.find({
selector: {
done: {
$eq: false
}
}
}).exec();
```
## Mocking the LocalStorage API for testing in Node.js
While the `localStorage` API only exists in browsers, you can use the LocalStorage based storage in [Node.js](./nodejs-database.md) by using the mock that comes with RxDB.
This is intended to be used in unit tests or other test suites:
```ts
import { createRxDatabase } from 'rxdb/plugins/core';
import {
getRxStorageLocalstorage,
getLocalStorageMock
} from 'rxdb/plugins/storage-localstorage';
const db = await createRxDatabase({
name: 'exampledb',
storage: getRxStorageLocalstorage({
localStorage: getLocalStorageMock()
})
});
```
---
## Instant Performance with IndexedDB RxStorage
# IndexedDB RxStorage
The IndexedDB [RxStorage](./rx-storage.md) is based on plain IndexedDB and can be used in browsers, [electron](./electron-database.md) or [hybrid apps](./articles/mobile-database.md).
Compared to other [browser based storages](./articles/browser-database.md), the IndexedDB storage has the smallest write- and read latency, the fastest initial page load
and the smallest build size. Only for big datasets (more than 10k documents), the [OPFS storage](./rx-storage-opfs.md) is better suited.
While the IndexedDB API itself can be very slow, the IndexedDB storage uses many tricks and performance optimizations, some of which are described [here](./slow-indexeddb.md). For example it uses custom index strings instead of the native IndexedDB indexes, batches cursors for faster bulk reads and many other improvements. The IndexedDB storage also operates on [Write-ahead logging](https://en.wikipedia.org/wiki/Write-ahead_logging) similar to SQLite, to improve write latency while still ensuring consistency on writes.
## IndexedDB performance comparison
Here is some performance comparison with other storages. Compared to the non-memory storages like [OPFS](./rx-storage-opfs.md) and [WASM SQLite](./rx-storage-sqlite.md), IndexedDB has the smallest build size and fastest write speed. Only OPFS is faster on queries over big datasets. See [performance comparison](./rx-storage-performance.md) page for a comparison with all storages.
## Using the IndexedDB RxStorage
To use the indexedDB storage you import it from the [RxDB Premium π](/premium/) npm module and use `getRxStorageIndexedDB()` when creating the [RxDatabase](./rx-database.md).
```ts
import {
createRxDatabase
} from 'rxdb';
import {
getRxStorageIndexedDB
} from 'rxdb-premium/plugins/storage-indexeddb';
const db = await createRxDatabase({
name: 'exampledb',
storage: getRxStorageIndexedDB({
/**
* For better performance, queries run with a batched cursor.
* You can change the batchSize to optimize the query time
* for specific queries.
* You should only change this value when you are also doing performance measurements.
* [default=300]
*/
batchSize: 300
})
});
```
## Overwrite/Polyfill the native IndexedDB
[Node.js](./nodejs-database.md) has no IndexedDB API. To still run the IndexedDB `RxStorage` in Node.js, for example to run unit tests, you have to polyfill it.
You can do that by using the [fake-indexeddb](https://github.com/dumbmatter/fakeIndexedDB) module and pass it to the `getRxStorageIndexedDB()` function.
```ts
import { createRxDatabase } from 'rxdb';
import { getRxStorageIndexedDB } from 'rxdb-premium/plugins/storage-indexeddb';
//> npm install fake-indexeddb --save
const fakeIndexedDB = require('fake-indexeddb');
const fakeIDBKeyRange = require('fake-indexeddb/lib/FDBKeyRange');
const db = await createRxDatabase({
name: 'exampledb',
storage: getRxStorageIndexedDB({
indexedDB: fakeIndexedDB,
IDBKeyRange: fakeIDBKeyRange
})
});
```
## Storage Buckets
The [Storage Buckets API](https://wicg.github.io/storage-buckets/) provides a way for sites to organize locally stored data into groupings called "storage buckets". This allows the user agent or sites to manage and delete buckets independently rather than applying the same treatment to all the data from a single origin. [Read More](https://developer.chrome.com/docs/web-platform/storage-buckets?hl=en)
To use different storage buckets with the RxDB IndexedDB Storage, you can use a function instead of a plain object when providing the `indexedDB` attribute:
```ts
import { createRxDatabase } from 'rxdb';
import { getRxStorageIndexedDB } from 'rxdb-premium/plugins/storage-indexeddb';
const db = await createRxDatabase({
name: 'exampledb',
storage: getRxStorageIndexedDB({
indexedDB: async(params) => {
const myStorageBucket = await navigator.storageBuckets.open('myApp-' + params.databaseName);
return myStorageBucket.indexedDB;
},
IDBKeyRange
})
});
```
## Limitations of the IndexedDB RxStorage
- It is part of the [RxDB Premium π](/premium/) plugin that must be purchased. If you just need a storage that works in the browser and you do not have to care about performance, you can use the [LocalStorage storage](./rx-storage-localstorage.md) instead.
- The IndexedDB storage requires support for [IndexedDB v2](https://caniuse.com/indexeddb2), it does not work on Internet Explorer.
---
## Supercharged OPFS Database with RxDB
# Origin Private File System (OPFS) Database with the RxDB OPFS-RxStorage
With the [RxDB](https://rxdb.info/) OPFS storage you can build a fully featured database on top of the [Origin Private File System](https://web.dev/opfs) (OPFS) browser API. Compared to other storage solutions, it has a way better performance.
## What is OPFS
The **Origin Private File System (OPFS)** is a native browser storage API that allows web applications to manage files in a private, sandboxed, **origin-specific virtual filesystem**. Unlike [IndexedDB](./rx-storage-indexeddb.md) and [LocalStorage](./articles/localstorage.md), which are optimized as object/key-value storage, OPFS provides more granular control for file operations, enabling byte-by-byte access, file streaming, and even low-level manipulations.
OPFS is ideal for applications requiring **high-performance** file operations (**3x-4x faster compared to IndexedDB**) inside of a client-side application, offering advantages like improved speed, more efficient use of resources, and enhanced security and privacy features.
### OPFS limitations
From the beginning of 2023, the Origin Private File System API is supported by [all modern browsers](https://caniuse.com/native-filesystem-api) like Safari, Chrome, Edge and Firefox. Only Internet Explorer is not supported and likely will never get support.
It is important to know that the most performant synchronous methods like [`read()`](https://developer.mozilla.org/en-US/docs/Web/API/FileSystemSyncAccessHandle/read) and [`write()`](https://developer.mozilla.org/en-US/docs/Web/API/FileSystemSyncAccessHandle/write) of the OPFS API are **only available inside of a [WebWorker](./rx-storage-worker.md)**.
They cannot be used in the main thread, an iFrame or even a [SharedWorker](./rx-storage-shared-worker.md).
The OPFS [`createSyncAccessHandle()`](https://developer.mozilla.org/en-US/docs/Web/API/FileSystemFileHandle/createSyncAccessHandle) method that gives you access to the synchronous methods is not exposed in the main thread, only in a Worker.
While there is no concrete **data size limit** defined by the API, browsers will refuse to store more [data at some point](./articles/indexeddb-max-storage-limit.md).
If no more data can be written, a `QuotaExceededError` is thrown which should be handled by the application, like showing an error message to the user.
## How the OPFS API works
The OPFS API is pretty straightforward to use. First you get the root filesystem. Then you can create files and directories on that. Notice that whenever you _synchronously_ write to, or read from a file, an `ArrayBuffer` must be used that contains the data. It is not possible to synchronously write plain strings or objects into the file. Therefore the `TextEncoder` and `TextDecoder` API must be used.
Also notice that some of the methods of `FileSystemSyncAccessHandle` [have been asynchronous](https://developer.chrome.com/blog/sync-methods-for-accesshandles) in the past, but are synchronous since Chromium 108. To make it less confusing, we just use `await` in front of them, so it will work in both cases.
```ts
// Access the root directory of the origin's private file system.
const root = await navigator.storage.getDirectory();
// Create a subdirectory.
const diaryDirectory = await root.getDirectoryHandle('subfolder', {
create: true,
});
// Create a new file named 'example.txt'.
const fileHandle = await diaryDirectory.getFileHandle('example.txt', {
create: true,
});
// Create a FileSystemSyncAccessHandle on the file.
const accessHandle = await fileHandle.createSyncAccessHandle();
// Write a sentence to the file.
let writeBuffer = new TextEncoder().encode('Hello from RxDB');
const writeSize = accessHandle.write(writeBuffer);
// Read file and transform data to string.
const readBuffer = new Uint8Array(writeSize);
const readSize = accessHandle.read(readBuffer, { at: 0 });
const contentAsString = new TextDecoder().decode(readBuffer);
// Write an exclamation mark to the end of the file.
writeBuffer = new TextEncoder().encode('!');
accessHandle.write(writeBuffer, { at: readSize });
// Truncate file to 10 bytes.
await accessHandle.truncate(10);
// Get the new size of the file.
const fileSize = await accessHandle.getSize();
// Persist changes to disk.
await accessHandle.flush();
// Always close FileSystemSyncAccessHandle if done, so others can open the file again.
await accessHandle.close();
```
A more detailed description of the OPFS API can be found [on MDN](https://developer.mozilla.org/en-US/docs/Web/API/File_System_API/Origin_private_file_system).
## OPFS performance
Because the Origin Private File System API provides low-level access to binary files, it is much faster compared to [IndexedDB](./slow-indexeddb.md) or [localStorage](./articles/localstorage.md). According to the [storage performance test](https://pubkey.github.io/client-side-databases/database-comparison/index.html), OPFS is up to 2x times faster on plain inserts when a new file is created on each write. Reads are even faster.
A good comparison about real world scenarios, are the [performance results](./rx-storage-performance.md) of the various RxDB storages. Here it shows that reads are up to 4x faster compared to IndexedDB, even with complex queries:
## Using OPFS as RxStorage in RxDB
The OPFS [RxStorage](./rx-storage.md) itself must run inside a WebWorker. Therefore we use the [Worker RxStorage](./rx-storage-worker.md) and let it point to the prebuild `opfs.worker.js` file that comes shipped with RxDB Premium π.
Notice that the OPFS RxStorage is part of the [RxDB Premium π](/premium/) plugin that must be purchased.
```ts
import {
createRxDatabase
} from 'rxdb';
import { getRxStorageWorker } from 'rxdb-premium/plugins/storage-worker';
const database = await createRxDatabase({
name: 'mydatabase',
storage: getRxStorageWorker(
{
/**
* This file must be statically served from a webserver.
* You might want to first copy it somewhere outside of
* your node_modules folder.
*/
workerInput: 'node_modules/rxdb-premium/dist/workers/opfs.worker.js'
}
)
});
```
## Using OPFS in the main thread instead of a worker
The `createSyncAccessHandle()` method from the OPFS File System Access API is only available inside of a WebWorker. Therefore you cannot use `getRxStorageOPFS()` in the main thread. Instead, RxDB provides `getRxStorageOPFSMainThread()`, which uses the asynchronous OPFS APIs (such as `FileSystemFileHandle.createWritable()`) under the hood. Because it cannot use the synchronous access handle, this main-thread variant is slightly slower for heavy write workloads than the worker-based storage.
Using OPFS from the main thread can still have benefits, because avoiding the worker bridge can reduce latency for some read and write patterns and may simplify your application architecture.
```ts
import { createRxDatabase } from 'rxdb';
import { getRxStorageOPFSMainThread } from 'rxdb-premium/plugins/storage-opfs';
const database = await createRxDatabase({
name: 'mydatabase',
storage: getRxStorageOPFSMainThread()
});
```
## Building a custom `worker.js`
When you want to run additional plugins like storage wrappers or replication **inside** of the worker, you have to build your own `worker.js` file. You can do that similar to other workers by calling `exposeWorkerRxStorage` like described in the [worker storage plugin](./rx-storage-worker.md).
```ts
// inside of the worker.js file
import { getRxStorageOPFS } from 'rxdb-premium/plugins/storage-opfs';
import { exposeWorkerRxStorage } from 'rxdb-premium/plugins/storage-worker';
const storage = getRxStorageOPFS();
exposeWorkerRxStorage({
storage
});
```
## Setting `usesRxDatabaseInWorker` when a RxDatabase is also used inside of the worker
When you use the OPFS inside of a worker, it will internally use strings to represent operation results. This has the benefit that transferring strings from the worker to the main thread, is way faster compared to complex json objects. The `getRxStorageWorker()` will automatically decode these strings on the main thread so that the data can be used by the RxDatabase.
But using a RxDatabase **inside** of your worker can make sense for example when you want to move the [replication](./replication.md) with a server. To enable this, you have to set `usesRxDatabaseInWorker` to `true`:
```ts
// inside of the worker.js file
import { getRxStorageOPFS } from 'rxdb-premium/plugins/storage-opfs';
const storage = getRxStorageOPFS({
usesRxDatabaseInWorker: true
});
```
If you forget to set this and still create and use a [RxDatabase](./rx-database.md) inside of the worker, you might get the error message "Uncaught (in promise) TypeError: Cannot read properties of undefined (reading 'length')".
## OPFS in Electron, React-Native or Capacitor.js
Origin Private File System is a browser API that is only accessible in browsers. Other JavaScript like React-Native or Node.js, do not support it.
**Electron** has two JavaScript contexts: the browser (chromium) context and the Node.js context. While you could use the OPFS API in the browser context, it is not recommended. Instead you should use the Filesystem API of Node.js and then only transfer the relevant data with the [ipcRenderer](https://www.electronjs.org/de/docs/latest/api/ipc-renderer). With RxDB that is pretty easy to configure:
- In the `main.js`, expose the [Node Filesystem](./rx-storage-filesystem-node.md) storage with the `exposeIpcMainRxStorage()` that comes with the [electron plugin](./electron.md)
- In the browser context, access the main storage with the `getRxStorageIpcRenderer()` method.
**React Native** (and Expo) does not have an OPFS API. You could use the ReactNative Filesystem to directly write data. But to get a fully featured database like RxDB it is easier to use the [SQLite RxStorage](./rx-storage-sqlite.md) which starts an SQLite database inside of the ReactNative app and uses that to do the database operations.
**Capacitor.js** is able to access the OPFS API.
## Difference between `File System Access API` and `Origin Private File System (OPFS)`
Often developers are confused with the differences between the `File System Access API` and the `Origin Private File System (OPFS)`.
- The `File System Access API` provides access to the files on the device file system, like the ones shown in the file explorer of the operating system. To use the File System API, the user has to actively select the files from a filepicker.
- `Origin Private File System (OPFS)` is a sub-part of the `File System Standard` and it only describes the things you can do with the filesystem root from `navigator.storage.getDirectory()`. OPFS writes to a **sandboxed** filesystem, not visible to the user. Therefore the user does not have to actively select or allow the data access.
## Learn more about OPFS:
- [WebKit: The File System API with Origin Private File System](https://webkit.org/blog/12257/the-file-system-access-api-with-origin-private-file-system/)
- [Browser Support](https://caniuse.com/native-filesystem-api)
- [Performance Test Tool](https://pubkey.github.io/client-side-databases/database-comparison/index.html)
---
## Lightning-Fast Memory Storage for RxDB
# Memory RxStorage
The Memory [RxStorage](./rx-storage.md) is based on plain in-memory arrays and objects. It can be used in all environments and is made for performance. By storing data directly in RAM, it eliminates disk I/O bottlenecks and operates faster than traditional disk-based databases.
You should use this storage when you need a fast database configuration, such as in unit tests, server-side rendering, or high-throughput data processing.
## How it achieves maximum speed
- **No Disk I/O**: Operations happen entirely in RAM. There is no waiting for disk reads or writes.
- **No Serialization Overhead**: Data remains as JavaScript objects and arrays. It skips the expensive JSON serialization and deserialization steps required by index-based or file-based storages.
- **Binary Search**: It uses pure JavaScript arrays and binary search algorithms on all database operations, ensuring fast queries and index traversals.
- **Small Build Size**: The plugin contains minimal code, keeping your bundle size small.
## Use Cases
### 1. Unit Testing and CI/CD
The Memory storage is the recommended storage for testing RxDB applications. It provides two major benefits: speed and isolation. Because it keeps data only in memory, each test run can start with a clean state without needing to clean up leftover filesystem states or deleting IndexedDB databases.
You can also simulate multi-tab behavior inside a single Node.js process by creating multiple `RxDatabase` instances with the same name and the `ignoreDuplicate: true` setting. They will share the memory state and communicate with each other naturally.
### 2. Server-Side Rendering (SSR)
When rendering React, Vue, or Angular applications on the server, you often need to fetch data, populate a database state, and render the UI. Using the Memory storage ensures your server handles these requests quickly without touching the file system, reducing latency and avoiding disk write locks.
### 3. Caching and Real-Time Processing
For applications handling thousands of events per second, such as real-time analytics dashboards or temporary chat state, the Memory storage acts as a fast data layer. You achieve instantaneous data access for aggregations and queries.
### 4. Memory-Mapped Performance Upgrades
RxDB provides a [Memory-Mapped RxStorage](./rx-storage-memory-mapped.md) which uses the Memory storage as a fast, primary layer and replicates data to a slower persistence storage in the background. This improves initial page load and query times while still keeping data safe on disk.
## Implementation
```ts
import {
createRxDatabase
} from 'rxdb';
import {
getRxStorageMemory
} from 'rxdb/plugins/storage-memory';
const db = await createRxDatabase({
name: 'exampledb',
storage: getRxStorageMemory()
});
```
### Constraints
- **No Persistence**: All data is lost when the JavaScript process exits or the browser tab is closed.
- **Memory Limits**: The dataset is constrained by the available RAM in the JavaScript runtime environment.
---
## Blazing-Fast Node Filesystem Storage
# Filesystem Node RxStorage
The Filesystem Node [RxStorage](./rx-storage.md) for RxDB is built on top of the [Node.js Filesystem API](https://nodejs.org/api/fs.html).
It stores data in plain JSON/txt files like any "normal" database does. It is a bit faster compared to the [SQLite storage](./rx-storage-sqlite.md) and its setup is less complex.
Using the same database folder in parallel with multiple Node.js processes is supported when you set `multiInstance: true` while creating the [RxDatabase](./rx-database.md).
### Pros
- Easier setup compared to [SQLite](./rx-storage-sqlite.md)
- [Fast](./rx-storage-performance.md)
### Cons
- It is part of the [RxDB Premium π](/premium/) plugin that must be purchased.
## Usage
```ts
import {
createRxDatabase
} from 'rxdb';
import {
getRxStorageFilesystemNode
} from 'rxdb-premium/plugins/storage-filesystem-node';
import path from 'path';
const myRxDatabase = await createRxDatabase({
name: 'exampledb',
storage: getRxStorageFilesystemNode({
basePath: path.join(__dirname, 'my-database-folder'),
/**
* Set inWorker=true if you use this RxStorage
* together with the WebWorker plugin.
*/
inWorker: false
})
});
/* ... */
```
---
## RxDB SQLite RxStorage for Hybrid Apps
import {Steps} from '@site/src/components/steps';
import {Tabs} from '@site/src/components/tabs';
# SQLite RxStorage
This [RxStorage](./rx-storage.md) is based on [SQLite](https://www.sqlite.org/index.html) and is made to work with **Node.js**, [Electron](./electron-database.md), [React Native](./react-native-database.md) and [Capacitor](./capacitor-database.md) or SQLite via webassembly in the browser. It can be used with different so called `sqliteBasics` adapters to account for the differences in the various SQLite bundles and libraries that exist.
SQLite is a natural fit for RxDB because most platforms - Android, iOS, Node.js, and beyond - already ship with a built-in SQLite engine, delivering robust performance and minimal setup overhead. Its proven reliability, having powered countless applications over the years, ensures a battle-tested foundation for local data. By placing RxDB on top of SQLite, you gain advanced features suited for building interactive, [offline-capable](./offline-first.md) UI apps: [real-time queries](./rx-query.md#observe), reactive state updates, [conflict handling](./transactions-conflicts-revisions.md), [data encryption](./encryption.md), and straightforward [schema management](./rx-schema.md). This combination offers a unified NoSQL-like experience without sacrificing the speed and broad availability that SQLite brings.
## Performance comparison with other storages
The SQLite storage is a bit slower compared to other Node.js based storages like the [Filesystem Storage](./rx-storage-filesystem-node.md) because wrapping SQLite has a bit of overhead and sending data from the JavaScript process to SQLite and backwards increases the latency. However for most hybrid apps the SQLite storage is the best option because it can leverage the SQLite version that comes already installed on the smartphone's OS (iOS and android). Also for desktop Electron apps it can be a viable solution because it is easy to ship SQLite together inside of the Electron bundle.
## Using the SQLite RxStorage
There are two versions of the SQLite storage available for RxDB:
- The **trial version** which comes directly shipped with RxDB Core. It contains an SQLite storage that allows you to try out RxDB on devices that support SQLite, like React Native or Electron. While the trial version does pass the full RxDB storage test-suite, it is not made for production. It is not using indexes, has no [attachment support](./rx-attachment.md), is limited to store 300 documents and fetches the whole storage state to run queries in memory. **Use it for evaluation and prototypes only!**
- The **[RxDB Premium π](/premium/) version** which contains the full production-ready SQLite storage. It contains a full load of performance optimizations and full query support. To use the SQLite storage you have to import `getRxStorageSQLite` from the [RxDB Premium π](/premium/) package and then add the correct `sqliteBasics` adapter depending on which sqlite module you want to use. This can then be used as storage when creating the [RxDatabase](./rx-database.md). In the following you can see some examples for some of the most common SQLite packages.
## Trial Version
```ts
// Import the Trial SQLite Storage
import {
getRxStorageSQLiteTrial,
getSQLiteBasicsNodeNative
} from 'rxdb/plugins/storage-sqlite';
// Create a Storage for it, here we use the nodejs-native SQLite module
// other SQLite modules can be used with a different sqliteBasics adapter
import { DatabaseSync } from 'node:sqlite';
const storage = getRxStorageSQLiteTrial({
sqliteBasics: getSQLiteBasicsNodeNative(DatabaseSync)
});
// Create a Database with the Storage
const myRxDatabase = await createRxDatabase({
name: 'exampledb',
storage: storage
});
```
## RxDB Premium π
```ts
// Import the SQLite Storage from the premium plugins.
import {
getRxStorageSQLite,
getSQLiteBasicsNodeNative
} from 'rxdb-premium/plugins/storage-sqlite';
// Create a Storage for it, here we use the nodejs-native SQLite module
// other SQLite modules can be used with a different sqliteBasics adapter
import { DatabaseSync } from 'node:sqlite';
const storage = getRxStorageSQLite({
sqliteBasics: getSQLiteBasicsNodeNative(DatabaseSync)
});
// Create a Database with the Storage
const myRxDatabase = await createRxDatabase({
name: 'exampledb',
storage: storage
});
```
In the following, all examples are shown with the premium SQLite storage. Still they work the same with the trial version.
## SQLiteBasics
Different SQLite libraries have different APIs to create and access the SQLite database. Therefore the library must be massaged to work with the RxDB SQlite storage. This is done in a so-called `SQLiteBasics` interface. RxDB directly ships with a wide range of these for various SQLite libraries that are commonly used. Also creating your own one is pretty simple, check the source code of the existing ones for that.
For example for the `sqlite3` npm library we have the `getSQLiteBasicsNode()` implementation. For `node:sqlite` we have the `getSQLiteBasicsNodeNative()` implementation and so on..
## Using the SQLite RxStorage with different SQLite libraries
### Usage with the **sqlite3 npm package**
```ts
import {
createRxDatabase
} from 'rxdb';
import {
getRxStorageSQLite,
getSQLiteBasicsNode
} from 'rxdb-premium/plugins/storage-sqlite';
/**
* In Node.js, we use the SQLite database
* from the 'sqlite' npm module.
* @link https://www.npmjs.com/package/sqlite3
*/
import sqlite3 from 'sqlite3';
const myRxDatabase = await createRxDatabase({
name: 'exampledb',
storage: getRxStorageSQLite({
/**
* Different runtimes have different interfaces to SQLite.
* For example in node.js we have a callback API,
* while in capacitor sqlite we have Promises.
* So we need a helper object that is capable of doing the basic
* sqlite operations.
*/
sqliteBasics: getSQLiteBasicsNode(sqlite3)
})
});
```
### Usage with the **node:sqlite** package
With Node.js version 22 and newer, you can use the "native" [sqlite module](https://nodejs.org/api/sqlite.html) that comes shipped with Node.js.
```ts
import { createRxDatabase } from 'rxdb';
import {
getRxStorageSQLite,
getSQLiteBasicsNodeNative
} from 'rxdb-premium/plugins/storage-sqlite';
import { DatabaseSync } from 'node:sqlite';
const myRxDatabase = await createRxDatabase({
name: 'exampledb',
storage: getRxStorageSQLite({
sqliteBasics: getSQLiteBasicsNodeNative(DatabaseSync)
})
});
```
### Usage with Webassembly in the Browser
In the browser you can use the [wa-sqlite](https://github.com/rhashimoto/wa-sqlite) package to run SQLite in Webassembly. The wa-sqlite module also allows using persistence with IndexedDB or OPFS. Notice that in general SQLite via Webassembly is slower compared to other storages like [IndexedDB](./rx-storage-indexeddb.md) or [OPFS](./rx-storage-opfs.md) because sending data from the main thread to wasm and backwards is slow in the browser. Have a look at the [performance comparison](./rx-storage-performance.md).
```ts
import {
createRxDatabase
} from 'rxdb';
import {
getRxStorageSQLite,
getSQLiteBasicsWasm
} from 'rxdb-premium/plugins/storage-sqlite';
/**
* In the Browser, we use the SQLite database
* from the 'wa-sqlite' npm module. This contains the SQLite library
* compiled to Webassembly
* @link https://www.npmjs.com/package/wa-sqlite
*/
import SQLiteESMFactory from 'wa-sqlite/dist/wa-sqlite-async.mjs';
import SQLite from 'wa-sqlite';
const sqliteModule = await SQLiteESMFactory();
const sqlite3 = SQLite.Factory(module);
const myRxDatabase = await createRxDatabase({
name: 'exampledb',
storage: getRxStorageSQLite({
sqliteBasics: getSQLiteBasicsWasm(sqlite3)
})
});
```
### Usage with **React Native**
1. Install the [react-native-quick-sqlite npm module](https://www.npmjs.com/package/react-native-quick-sqlite)
2. Import `getSQLiteBasicsQuickSQLite` from the SQLite plugin and use it to create a [RxDatabase](./rx-database.md):
```ts
import {
createRxDatabase
} from 'rxdb';
import {
getRxStorageSQLite,
getSQLiteBasicsQuickSQLite
} from 'rxdb-premium/plugins/storage-sqlite';
import { open } from 'react-native-quick-sqlite';
// create database
const myRxDatabase = await createRxDatabase({
name: 'exampledb',
multiInstance: false, // <- Set multiInstance to false when using RxDB in React Native
storage: getRxStorageSQLite({
sqliteBasics: getSQLiteBasicsQuickSQLite(open)
})
});
```
If `react-native-quick-sqlite` does not work for you, as alternative you can use the [react-native-sqlite-2](https://www.npmjs.com/package/react-native-sqlite-2) library instead:
```ts
import {
getRxStorageSQLite,
getSQLiteBasicsWebSQL
} from 'rxdb-premium/plugins/storage-sqlite';
import SQLite from 'react-native-sqlite-2';
const storage = getRxStorageSQLite({
sqliteBasics: getSQLiteBasicsWebSQL(SQLite.openDatabase)
});
```
### Usage with **Expo SQLite**
Notice that [expo-sqlite](https://www.npmjs.com/package/expo-sqlite) cannot be used on android (but it works on iOS) if you use Expo SDK version 50 or older. Please update to Version 50 or newer to use it.
In the latest expo SDK version, use the `getSQLiteBasicsExpoSQLiteAsync()` method:
```ts
import {
createRxDatabase
} from 'rxdb';
import {
getRxStorageSQLite,
getSQLiteBasicsExpoSQLiteAsync
} from 'rxdb-premium/plugins/storage-sqlite';
import * as SQLite from 'expo-sqlite';
const myRxDatabase = await createRxDatabase({
name: 'exampledb',
multiInstance: false,
storage: getRxStorageSQLite({
sqliteBasics: getSQLiteBasicsExpoSQLiteAsync(SQLite.openDatabaseAsync)
})
});
```
In older Expo SDK versions, you might have to use the non-async API:
```ts
import {
createRxDatabase
} from 'rxdb';
import {
getRxStorageSQLite,
getSQLiteBasicsExpoSQLite
} from 'rxdb-premium/plugins/storage-sqlite';
import { openDatabase } from 'expo-sqlite';
const myRxDatabase = await createRxDatabase({
name: 'exampledb',
multiInstance: false,
storage: getRxStorageSQLite({
sqliteBasics: getSQLiteBasicsExpoSQLite(openDatabase)
})
});
```
### Usage with **SQLite Capacitor**
1. Install the [sqlite capacitor npm module](https://github.com/capacitor-community/sqlite)
2. Add the iOS database location to your capacitor config
```json
{
"plugins": {
"CapacitorSQLite": {
"iosDatabaseLocation": "Library/CapacitorDatabase"
}
}
}
```
3. Use the function `getSQLiteBasicsCapacitor` to get the capacitor sqlite wrapper.
```ts
import {
createRxDatabase
} from 'rxdb';
import {
getRxStorageSQLite,
getSQLiteBasicsCapacitor
} from 'rxdb-premium/plugins/storage-sqlite';
/**
* Import SQLite from the capacitor plugin.
*/
import {
CapacitorSQLite,
SQLiteConnection
} from '@capacitor-community/sqlite';
import { Capacitor } from '@capacitor/core';
const sqlite = new SQLiteConnection(CapacitorSQLite);
const myRxDatabase = await createRxDatabase({
name: 'exampledb',
storage: getRxStorageSQLite({
/**
* Different runtimes have different interfaces to SQLite.
* For example in node.js we have a callback API,
* while in capacitor sqlite we have Promises.
* So we need a helper object that is capable of doing the basic
* sqlite operations.
*/
sqliteBasics: getSQLiteBasicsCapacitor(sqlite, Capacitor)
})
});
```
### Usage with Tauri SQLite
1. Add the [Tauri SQL plugin](https://tauri.app/plugin/sql/#setup) to your Tauri project.
2. Make sure to add `sqlite` as your database engine by running `cargo add tauri-plugin-sql --features sqlite` inside `src-tauri`.
3. Use the `getSQLiteBasicsTauri` function to get the Tauri SQLite wrapper.
```ts
import {
createRxDatabase
} from 'rxdb';
import {
getRxStorageSQLite,
getSQLiteBasicsTauri
} from 'rxdb/plugins/storage-sqlite';
import sqlite3 from '@tauri-apps/plugin-sql';
const myRxDatabase = await createRxDatabase({
name: 'exampledb',
storage: getRxStorageSQLite({
sqliteBasics: getSQLiteBasicsTauri(sqlite3)
})
});
```
## Database Connection
If you need to access the database connection for any reason you can use `getDatabaseConnection` to do so:
```ts
import { getDatabaseConnection } from 'rxdb-premium/plugins/storage-sqlite'
```
It has the following signature:
```ts
getDatabaseConnection(
sqliteBasics: SQLiteBasics,
databaseName: string
): Promise;
```
## Known Problems of SQLite in JavaScript apps
- Some JavaScript runtimes do not contain a `Buffer` API which is used by SQLite to store binary attachments data as `BLOB`. You can set `storeAttachmentsAsBase64String: true` if you want to store the attachments data as base64 string instead. This increases the database size but makes it work even without having a `Buffer`.
- The SQlite RxStorage works on SQLite libraries that use SQLite in version `3.38.0 (2022-02-22)` or newer, because it uses the [SQLite JSON](https://www.sqlite.org/json1.html) methods like `JSON_EXTRACT`. If you get an error like `[Error: no such function: JSON_EXTRACT (code 1 SQLITE_ERROR[1])`, you might have a too old version of SQLite.
- To debug all SQL operations, you can pass a log function to `getRxStorageSQLite()` like this. This does not work with the trial version:
```ts
const storage = getRxStorageSQLite({
sqliteBasics: getSQLiteBasicsCapacitor(sqlite, Capacitor),
// pass log function
log: console.log.bind(console)
});
```
- By default, all tables will be created with the `WITHOUT ROWID` flag. Some tools like drizzle do not support tables with that option. You can disable it by setting `withoutRowId: false` when calling `getRxStorageSQLite()`:
```ts
const storage = getRxStorageSQLite({
sqliteBasics: getSQLiteBasicsCapacitor(sqlite, Capacitor),
withoutRowId: false
});
```
## Related
- [React Native Databases](./react-native-database.md)
---
## RxDB Dexie.js Database - Fast, Reactive, Sync with Any Backend
import {Steps} from '@site/src/components/steps';
# RxStorage Dexie.js
To store the data inside of and [RxDB Database](./rx-database.md) in IndexedDB in the [browser](./articles/browser-database.md), you can use the [Dexie.js](https://github.com/dexie/Dexie.js) based [RxStorage](./rx-storage.md). Dexie.js is a minimal wrapper around IndexedDB and the Dexie.js RxStorage wraps that again to use it for an RxDB database in the browser. For side projects and prototypes that run in a browser, you should use the dexie RxStorage as a default.
## Dexie.js vs IndexedDB Storage
While Dexie.js [RxStorage](./rx-storage.md) can be used for free, most professional projects should switch to our **premium [IndexedDB RxStorage](./rx-storage-indexeddb.md) π** in production:
- It is faster and reduces build size by up to **36%**.
- It has a way [better performance](./rx-storage-performance.md) on reads and writes.
- It stores attachments data as binary instead of base64 which reduces used space by 33%.
- It does not use a [Batched Cursor](./slow-indexeddb.md#batched-cursor) or [custom indexes](./slow-indexeddb.md#custom-indexes) which makes queries slower compared to the [IndexedDB RxStorage](./rx-storage-indexeddb.md).
- It supports **non-required indexes** which is [not possible](https://github.com/pubkey/rxdb/pull/6643#issuecomment-2505310082) with Dexie.js.
- It runs in a **WAL-like mode** (similar to SQLite) for faster writes and improved responsiveness.
- It support the [Storage Buckets API](./rx-storage-indexeddb.md#storage-buckets)
## How to use Dexie.js as a Storage for RxDB
### Import the Dexie Storage
```ts
import { createRxDatabase } from 'rxdb/plugins/core';
import { getRxStorageDexie } from 'rxdb/plugins/storage-dexie';
```
### Create a Database
```ts
const db = await createRxDatabase({
name: 'exampledb',
storage: getRxStorageDexie()
});
```
## Overwrite/Polyfill the native IndexedDB API with an in-memory version
Node.js has no IndexedDB API. To still run the Dexie `RxStorage` in Node.js, for example to run unit tests, you have to polyfill it.
You can do that by using the [fake-indexeddb](https://github.com/dumbmatter/fakeIndexedDB) module and pass it to the `getRxStorageDexie()` function.
```ts
import { createRxDatabase } from 'rxdb/plugins/core';
import { getRxStorageDexie } from 'rxdb/plugins/storage-dexie';
//> npm install fake-indexeddb --save
const fakeIndexedDB = require('fake-indexeddb');
const fakeIDBKeyRange = require('fake-indexeddb/lib/FDBKeyRange');
const db = await createRxDatabase({
name: 'exampledb',
storage: getRxStorageDexie({
indexedDB: fakeIndexedDB,
IDBKeyRange: fakeIDBKeyRange
})
});
```
## Using Dexie Addons
Dexie.js has its own plugin system with [many plugins](https://dexie.org/docs/DerivedWork#known-addons) for encryption, replication or other use cases. With the Dexie.js `RxStorage` you can use the same plugins by passing them to the `getRxStorageDexie()` function.
```ts
const db = await createRxDatabase({
name: 'exampledb',
storage: getRxStorageDexie({
addons: [ /* Your Dexie.js plugins */ ]
})
});
```
## Sync Dexie.js with your Backend in RxDB
Having your local data in sync with a remote backend is a key feature of RxDB. Here are two approaches to achieve this when using the Dexie.js RxStorage:
* **Dexie Cloud** provides a **managed solution**: For quick setups, letting you rely on its Cloud backend and conflict resolution.
* [RxDB's replication](./replication.md): Offers **full control** over your backend, data flow, and [conflict handling](./transactions-conflicts-revisions.md).
Choose the approach that best suits your needs - whether you want to get started quickly with Dexie Cloud or require the adaptability and autonomy of RxDB's native replication.
### A. Use Dexie Cloud Sync
**Dexie Cloud** is an official SaaS solution provided by the Dexie team. It offers automatic synchronization, user management, and conflict resolution out of the box. The primary benefits are:
- **Automatic Sync**: Dexie Cloud keeps your local IndexedDB in sync with its cloud-based backend.
- **User Authentication**: Built-in user management (auth, roles, permissions).
- **Conflict Resolution**: Automated resolution logic on the server side.
#### Install the Dexie Cloud Addon
```bash
npm install dexie-cloud-addon
```
#### Import RxDB and dexie-cloud
```ts
import { createRxDatabase } from 'rxdb/plugins/core';
import { getRxStorageDexie } from 'rxdb/plugins/storage-dexie';
import dexieCloud from 'dexie-cloud-addon';
```
#### Create a Dexie based RxStorage with the Cloud Plugin
```ts
const storage = getRxStorageDexie({
addons: [dexieCloud],
/*
* Whenever a new dexie database instance is created,
* this method will be called.
*/
async onCreate(dexieDatabase, dexieDatabaseName) {
await dexieDatabase.cloud.configure({
databaseUrl: "https://.dexie.cloud",
requireAuth: true // optional
});
}
});
```
#### Create an RxDB Database
```ts
const db = await createRxDatabase({
name: 'mydb',
storage
});
```
### B. Use the RxDB Replication
For **full flexibility** over your backend or conflict resolution strategy, you can use one of **RxDB's many replication plugins** like
- [CouchDB Replication](./replication-couchdb.md) Plugin: Replicate with a CouchDB Server
- [GraphQL Replication](./replication-graphql.md) Plugin: Sync data with any GraphQL endpoint. Useful when you have a custom schema or you want to utilize GraphQL's powerful query features.
- [Custom Replication with REST APIs](./replication-http.md): Implement your own replication by building a pull/push handler that communicates with any RESTful backend.
Below is an example of replicating an RxDB collection with a CouchDB backend using RxDB's CouchDB replication plugin:
#### Import the RxDB with dexie and the CouchDB plugin
```ts
import { replicateCouchDB } from 'rxdb/plugins/replication-couchdb';
import { getRxStorageDexie } from 'rxdb/plugins/storage-dexie';
import { createRxDatabase } from 'rxdb/plugins/core';
```
#### Create an RxDB Database with the Dexie Storage
```ts
const db = await createRxDatabase({
name: 'mydb',
storage: getRxStorageDexie()
});
```
#### Add a Collection
```ts
await db.addCollections({
humans: {
schema: {
version: 0,
type: 'object',
primaryKey: 'id',
properties: {
id: { type: 'string', maxLength: 100 },
name: { type: 'string' },
age: { type: 'number' }
},
required: ['id', 'name']
}
}
});
```
#### Sync the Collection with a CouchDB Server
```ts
const replicationState = replicateCouchDB({
replicationIdentifier: 'my-couchdb-replication',
collection: db.humans,
// The URL to your CouchDB endpoint
url: 'http://example.com/db/humans'
});
```
## liveQuery - Realtime Queries
Dexie.js offers a feature called `liveQuery` which automatically updates query results as data changes, allowing you to react to these changes in real-time. However, because RxDB intrinsically provides [reactive queries](./rx-query.md#observe), you typically do **not** need to enable live queries through Dexie. Once you have created your database and collections with RxDB, any query you perform can be observed by subscribing to it, for example via `collection.find().$.subscribe(results => { /*... */ })`. This means RxDB takes care of listening for changes and automatically emitting new results - ensuring your UI stays in sync with the underlying data without requiring extra plugins or manual polling.
## Disabling the non-premium console log
We want to be transparent with our community, and you'll notice a console message when using the free Dexie.js based RxStorage implementation. This message serves to inform you about the availability of faster storage solutions within our [π Premium Plugins](/premium/). We understand that this might be a minor inconvenience, and we sincerely apologize for that. However, maintaining and improving RxDB requires substantial resources, and our premium users help us ensure its sustainability. If you find value in RxDB and wish to remove this message, we encourage you to explore our premium storage options, which are optimized for professional use and production environments. Thank you for your understanding and support.
If you already have premium access and want to use the Dexie.js [RxStorage](./rx-storage.md) without the log, you can call the `setPremiumFlag()` function to disable the log.
```js
import { setPremiumFlag } from 'rxdb-premium/plugins/shared';
setPremiumFlag();
```
## Performance comparison with other RxStorage plugins
The performance of the Dexie.js RxStorage is good enough for most use cases but other storages can have way better performance metrics:
---
## Unlock MongoDB Power with RxDB
import {Steps} from '@site/src/components/steps';
# MongoDB RxStorage
RxDB MongoDB RxStorage is an RxDB [RxStorage](./rx-storage.md) that allows you to use [MongoDB](https://www.mongodb.com/) as the underlying storage engine for your RxDB database. With this you can take advantage of MongoDB's features and scalability while benefiting from RxDB's real-time data synchronization capabilities.
The storage is made to work with any plain MongoDB Server, [MongoDB Replica Set](https://www.mongodb.com/docs/manual/tutorial/deploy-replica-set/), [Sharded MongoDB Cluster](https://www.mongodb.com/docs/manual/sharding/) or [Atlas Cloud Database](https://www.mongodb.com/atlas/database).
## Limitations of the MongoDB RxStorage
- Multiple Node.js servers using the same MongoDB database is currently not supported
- [RxAttachments](./rx-attachment.md) are currently not supported
- Doing non-RxDB writes on the MongoDB database is not supported. RxDB expects all writes to come from RxDB which update the required metadata. Doing non-RxDB writes can confuse the RxDatabase and lead to undefined behavior. But you can perform read-queries on the MongoDB storage from the outside at any time.
## Using the MongoDB RxStorage
### Install the mongodb package
```bash
npm install mongodb --save
```
### Setups the MongoDB RxStorage
To use the storage, you simply import the `getRxStorageMongoDB` method and use that when creating the [RxDatabase](./rx-database.md). The `connection` parameter contains the [MongoDB connection string](https://www.mongodb.com/docs/manual/reference/connection-string/).
```ts
import {
createRxDatabase
} from 'rxdb';
import {
getRxStorageMongoDB
} from 'rxdb/plugins/storage-mongodb';
const myRxDatabase = await createRxDatabase({
name: 'exampledb',
storage: getRxStorageMongoDB({
/**
* MongoDB connection string
* @link https://www.mongodb.com/docs/manual/reference/connection-string/
*/
connection: 'mongodb://localhost:27017,localhost:27018,localhost:27019'
})
});
```
---
## DenoKV RxStorage
# RxDB Database on top of Deno Key Value Store
With the DenoKV [RxStorage](./rx-storage.md) layer for [RxDB](https://rxdb.info), you can run a fully featured **NoSQL database** on top of the [DenoKV API](https://docs.deno.com/kv/manual).
This gives you the benefits and features of the RxDB JavaScript Database, combined with the global availability and distribution features of the DenoKV.
## What is DenoKV
[DenoKV](https://deno.com/kv) is a strongly consistent key-value storage, globally replicated for low-latency reads across 35 worldwide regions via [Deno Deploy](https://deno.com/deploy).
When you release your Deno application on Deno Deploy, it will start a instance on each of the [35 worldwide regions](https://docs.deno.com/deploy/manual/regions). This edge deployment guarantees minimal latency when serving requests to end users devices around the world. DenoKV is a shared storage which shares its state across all instances.
But, because DenoKV is "only" a **Key-Value storage**, it only supports basic CRUD operations on datasets and indexes. Complex features like queries, encryption, compression or client-server replication, are missing. Using RxDB on top of DenoKV fills this gap and makes it easy to build realtime [offline-first](./offline-first.md) application on top of Deno backend.
## Use cases
Using RxDB-DenoKV instead of plain DenoKV, can have a wide range of benefits depending on your use case.
- **Reduce vendor lock-in**: RxDB has a swappable [storage layer](./rx-storage.md) which allows you to swap out the underlying storage of your database. If you ever decide to move away from DenoDeploy or Deno at all, you do not have to refactor your whole application and instead just **swap the storage plugin**. For example if you decide migrate to Node.js, you can use the [FoundationDB RxStorage](./rx-storage-foundationdb.md) and store your data there. DenoKV is also implemented on top of FoundationDB so you can get similar performance. Alternatively RxDB supports a wide range of [storage plugins](./rx-storage.md) you can decide from.
- **Add reactiveness**: DenoKV is a plain request-response datastore. While it supports observation of single rows by id, it does not allow to observe row-ranges or events. This makes it hard to impossible to build realtime applications with it because polling would be the only way to watch ranges of key-value pairs. With RxDB on top of DenoKV, changes to the database are **shared between DenoDeploy instances** so when you **observe a [query](./rx-query.md)** you can be sure that it is always up to date, no matter which instance has changed the document. Internally RxDB uses the [Deno BroadcastChannel API](https://docs.deno.com/deploy/api/runtime-broadcast-channel) to share events between instances.
- **Reuse Client and Server Code**: When you use RxDB on the server and on the client side, many parts of your code can be reused on both sides which decreases development time significantly.
- **Replicate from DenoKV to a local RxDB state**: Instead of running all operations against the global DenoKV, you can run a [realtime-replication](./replication.md) between a DenoKV-RxDatabase and a [locally stored dataset](./rx-storage-filesystem-node.md) or maybe even an [in-memory](./rx-storage-memory.md) stored one. This improves **query performance** and can **reduce your Deno Deploy cloud costs** because less operations run against the DenoKV, they run only locally instead.
- **Replicate with other backends**: The RxDB [Sync Engine](./replication.md) is pretty simple and allows you to easily build a replication with any backend architecture. For example if you already have your data stored in a self-hosted MySQL server, you can use RxDB to do a realtime replication of that data into a DenoKV RxDatabase instance. RxDB also has many plugins for replication with backend/protocols like [GraphQL](./replication-graphql.md), [Websocket](./replication-websocket.md), [CouchDB](./replication-couchdb.md), [WebRTC](./replication-webrtc.md), [Firestore](./replication-firestore.md) and [NATS](./replication-nats.md).
## Using the DenoKV RxStorage
To use the DenoKV RxStorage with RxDB, you import the `getRxStorageDenoKV` function from the plugin and set it as storage when calling [createRxDatabase](./rx-database.md#creation)
```ts
import { createRxDatabase } from 'rxdb';
import { getRxStorageDenoKV } from 'rxdb/plugins/storage-denokv';
const myRxDatabase = await createRxDatabase({
name: 'exampledb',
storage: getRxStorageDenoKV({
/**
* Consistency level, either 'strong' or 'eventual'
* (Optional) default='strong'
*/
consistencyLevel: 'strong',
/**
* Path which is used in the first argument of Deno.openKv(settings.openKvPath)
* (Optional) default=''
*/
openKvPath: './foobar',
/**
* Some operations have to run in batches,
* you can test different batch sizes to improve performance.
* (Optional) default=100
*/
batchSize: 100
})
});
```
On top of that [RxDatabase](./rx-database.md) you can then create your collections and run operations. Follow the [quickstart](./quickstart.md) to learn more about how to use RxDB.
## Using non-DenoKV storages in Deno
When you use other storages than the DenoKV storage inside of a Deno app, make sure you set `multiInstance: false` when creating the database. Also you should only run one process per Deno-Deploy instance. This ensures your events are not mixed up by the [BroadcastChannel](https://docs.deno.com/deploy/api/runtime-broadcast-channel) across instances which would lead to wrong behavior.
```ts
// DenoKV based database
const db = await createRxDatabase({
name: 'denokvdatabase',
storage: getRxStorageDenoKV(),
/**
* Use multiInstance: true so that the Deno Broadcast Channel
* emits event across DenoDeploy instances
* (true is also the default, so you can skip this setting)
*/
multiInstance: true
});
// Non-DenoKV based database
const db = await createRxDatabase({
name: 'denokvdatabase',
storage: getRxStorageFilesystemNode(),
/**
* Use multiInstance: false so that it does not share events
* across instances because the stored data is anyway not shared
* between them.
*/
multiInstance: false
});
```
---
## RxDB on FoundationDB - Performance at Scale
# RxDB Database on top of FoundationDB
[FoundationDB](https://www.foundationdb.org/) is a distributed key-value store designed to handle large volumes of structured data across clusters of computers while maintaining high levels of performance, scalability, and fault tolerance. While FoundationDB itself only can store and query key-value pairs, it lacks more advanced features like complex queries, encryption and replication.
With the FoundationDB based [RxStorage](./rx-storage.md) of [RxDB](https://rxdb.info/) you can combine the benefits of FoundationDB while having a fully featured, high performance NoSQL database.
## Features of RxDB+FoundationDB
Using RxDB on top of FoundationDB, gives you many benefits compare to using the plain FoundationDB API:
- **Indexes**: In RxDB with a FoundationDB storage layer, indexes are used to optimize query performance, allowing for fast and efficient data retrieval even in large datasets. You can define single and compound indexes with the [RxDB schema](./rx-schema.md).
- **Schema Based Data Model**: Utilizing a [jsonschema](./rx-schema.md) based data model, the system offers a highly structured and versatile approach to organizing and [validating data](./schema-validation.md), ensuring consistency and clarity in database interactions.
- **Complex Queries**: The system supports complex [NoSQL queries](./rx-query.md), allowing for advanced data manipulation and retrieval, tailored to specific needs and intricate data relationships. For example you can do `$regex` or `$or` queries which is hardly possible with the plain key-value access of FoundationDB.
- **Observable Queries & Documents**: RxDB's observable queries and documents feature ensures real-time updates and synchronization, providing dynamic and responsive data interactions in applications.
- **Compression**: RxDB employs data [compression techniques](./key-compression.md) to reduce storage requirements and enhance transmission efficiency, making it more cost-effective and faster, especially for large volumes of data. You can compress the [NoSQL document](./key-compression.md) data, but also the [binary attachments](./rx-attachment.md#attachment-compression) data.
- **Attachments**: RxDB supports the storage and management of [attachments](./rx-attachment.md) which allowing for the seamless inclusion of binary data like images or documents alongside structured data within the database.
## Installation
- Install the [FoundationDB client cli](https://apple.github.io/foundationdb/getting-started-linux.html) which is used to communicate with the FoundationDB cluster.
- Install the [FoundationDB node bindings npm module](https://www.npmjs.com/package/foundationdb) via `npm install foundationdb`. This will install `v2.x.x`, which is only compatible with FoundationDB server and client `v7.3.x` (which is the only version currently maintained by the FoundationDB team). If you need to use an older version (e.g. `7.1.x` or `6.3.x`), you should run `npm install foundationdb@1.1.4` (though this might only work with `v6.3.x`).
- Due to an outstanding bug in node foundationdb, you will need to specify an `apiVersion` of `720` even though you are using `730`. When [this PR](https://github.com/josephg/node-foundationdb/pull/86) is merged, you will be able to use `730`.
## Usage
```typescript
import {
createRxDatabase
} from 'rxdb';
import {
getRxStorageFoundationDB
} from 'rxdb/plugins/storage-foundationdb';
const db = await createRxDatabase({
name: 'exampledb',
storage: getRxStorageFoundationDB({
/**
* Version of the API of the FoundationDB cluster..
* FoundationDB is backwards compatible across a wide range of versions,
* so you have to specify the api version.
* If in doubt, set it to 720.
*/
apiVersion: 720,
/**
* Path to the FoundationDB cluster file.
* (optional)
* If in doubt, leave this empty to use the default location.
*/
clusterFile: '/path/to/fdb.cluster',
/**
* Amount of documents to be fetched in batch requests.
* You can change this to improve performance depending on
* your database access patterns.
* (optional)
* [default=50]
*/
batchSize: 50
})
});
```
## Multi Instance
Because FoundationDB does not offer a [changestream](https://forums.foundationdb.org/t/streaming-data-out-of-foundationdb/683/2), it is not possible to use the same cluster from more than one Node.js process at the same time. For example you cannot spin up multiple servers with RxDB databases that all use the same cluster. There might be workarounds to create something like a FoundationDB changestream and you can make a Pull Request if you need that feature.
---
## Schema Validation
# Schema validation
RxDB has multiple validation implementations that can be used to ensure that your document data is always matching the provided JSON
schema of your [RxCollection](./rx-collection.md).
The schema validation is **not a plugin** but comes in as a wrapper around any other `RxStorage` and it will then validate all data that is written into that storage. This is required for multiple reasons:
- It allows us to run the validation inside of a [Worker RxStorage](./rx-storage-worker.md) instead of running it in the main JavaScript process.
- It allows us to configure which [RxDatabase](./rx-database.md) instance must use the validation and which does not. In production it often makes sense to validate user data, but you might not need the validation for data that is only replicated from the backend.
:::warning
Schema validation can be **CPU expensive** and increases your build size. You should always use a schema validation in development mode. For most use cases, you **should not** use a validation in production for better performance.
:::
When no validation is used, any document data can be saved but there might be **undefined behavior** when saving data that does not comply to the schema of a `RxCollection`.
RxDB has different implementations to validate data, each of them is based on a different [JSON Schema library](https://json-schema.org/tools). In this example we use the [LocalStorage RxStorage](./rx-storage-localstorage.md), but you can wrap the validation around **any other** [RxStorage](./rx-storage.md).
### validate-ajv
A validation-module that does the schema-validation. This one is using [ajv](https://github.com/epoberezkin/ajv) as validator which is a bit faster. Better compliant to the jsonschema-standard but also has a bigger build-size.
```javascript
import { wrappedValidateAjvStorage } from 'rxdb/plugins/validate-ajv';
import { getRxStorageLocalstorage } from 'rxdb/plugins/storage-localstorage';
// wrap the validation around the main RxStorage
const storage = wrappedValidateAjvStorage({
storage: getRxStorageLocalstorage()
});
const db = await createRxDatabase({
name: randomCouchString(10),
storage
});
```
### validate-z-schema
Both `is-my-json-valid` and `validate-ajv` use `eval()` to perform validation which might not be wanted when `'unsafe-eval'` is not allowed in Content Security Policies. This one is using [z-schema](https://github.com/zaggino/z-schema) as validator which doesn't use `eval`.
```javascript
import { wrappedValidateZSchemaStorage } from 'rxdb/plugins/validate-z-schema';
import { getRxStorageLocalstorage } from 'rxdb/plugins/storage-localstorage';
// wrap the validation around the main RxStorage
const storage = wrappedValidateZSchemaStorage({
storage: getRxStorageLocalstorage()
});
const db = await createRxDatabase({
name: randomCouchString(10),
storage
});
```
### validate-is-my-json-valid
**WARNING**: The `is-my-json-valid` validation is no longer supported until [this bug](https://github.com/mafintosh/is-my-json-valid/pull/192) is fixed.
The `validate-is-my-json-valid` plugin uses [is-my-json-valid](https://www.npmjs.com/package/is-my-json-valid) for schema validation.
```javascript
import { wrappedValidateIsMyJsonValidStorage } from 'rxdb/plugins/validate-is-my-json-valid';
import { getRxStorageLocalstorage } from 'rxdb/plugins/storage-localstorage';
// wrap the validation around the main RxStorage
const storage = wrappedValidateIsMyJsonValidStorage({
storage: getRxStorageLocalstorage()
});
const db = await createRxDatabase({
name: randomCouchString(10),
storage
});
```
## Custom Formats
The schema validators provide methods to add custom formats like a `email` format.
You have to add these formats **before** you create your database.
### Ajv Custom Format
```ts
import { getAjv } from 'rxdb/plugins/validate-ajv';
const ajv = getAjv();
ajv.addFormat('email', {
type: 'string',
validate: v => v.includes('@') // ensure email fields contain the @ symbol
});
```
### Z-Schema Custom Format
```ts
import { ZSchemaClass } from 'rxdb/plugins/validate-z-schema';
ZSchemaClass.registerFormat('email', function (v: string) {
return v.includes('@'); // ensure email fields contain the @ symbol
});
```
## Performance comparison of the validators
The RxDB team ran performance benchmarks using two storage options on an Ubuntu 24.04 machine with Chrome version `131.0.6778.85`. The testing machine has 32 core `13th Gen Intel(R) Core(TM) i9-13900HX` CPU.
IndexedDB Storage (based on the IndexedDB API in the browser):
| **IndexedDB Storage** | Time to First insert | Insert 3000 documents |
| ----------------- | :------------------: | --------------------: |
| no validator | 68 ms | 213 ms |
| ajv | 67 ms | 216 ms |
| z-schema | 71 ms | 230 ms |
Memory Storage: stores everything in memory for extremely fast reads and writes, with no persistence by default. Often used with the RxDB memory-mapped plugin that processes data in memory an later persists to disc in background:
| **Memory Storage** | Time to First insert | Insert 3000 documents |
| ------------------ | :------------------: | --------------------: |
| no validator | 1.15 ms | 0.8 ms |
| ajv | 3.05 ms | 2.7 ms |
| z-schema | 0.9 ms | 18 ms |
Including a validator library also increases your JavaScript bundle size. Here's how it breaks down (minified + gzip):
| **Build Size** (minified+gzip) | Build Size (IndexedDB) | Build Size (memory) |
| ------------------------------ | :----------------: | ------------------: |
| no validator | 73103 B | 39976 B |
| ajv | 106135 B | 72773 B |
| z-schema | 125186 B | 91882 B |
---
## Encryption
import {Steps} from '@site/src/components/steps';
# π Encrypted Local Storage with RxDB
The RxDB encryption plugin empowers developers to fortify their applications' data security. It seamlessly integrates with [RxDB](https://rxdb.info/), allowing for the secure storage and retrieval of documents by **encrypting them with a password**. With encryption and decryption processes handled internally, it ensures that sensitive data remains confidential, making it a valuable tool for building robust, privacy-conscious applications. The encryption works on all RxDB supported devices types like the **[browser](./articles/browser-database.md)**, **[ReactNative](./react-native-database.md)** or **[Node.js](./nodejs-database.md)**.
Encrypting client-side stored data in RxDB offers numerous advantages:
- **Enhanced Security**: In the unfortunate event of a user's device being stolen, the encrypted data remains safeguarded on the hard drive, inaccessible without the correct password.
- **Access Control**: You can retain control over stored data by revoking access at any time simply by withholding the password.
- **Tamper proof** Other applications on the device cannot read out the stored data when the password is only kept in the process-specific memory
## Querying encrypted data
RxDB handles the encryption and decryption of data internally. This means that when you work with a RxDocument, you can access the properties of the document just like you would with normal, unencrypted data. RxDB automatically decrypts the data for you when you retrieve it, making it transparent to your application code.
This means the encryption works with all [RxStorage](./rx-storage.md) like **[SQLite](./rx-storage-sqlite.md)**, **[IndexedDB](./rx-storage-indexeddb.md)**, **[OPFS](./rx-storage-opfs.md)** and so on.
However, there's a limitation when it comes to querying encrypted fields. **Encrypted fields cannot be used as operators in queries**. This means you cannot perform queries like "find all documents where the encrypted field equals a certain value." RxDB does not expose the encrypted data in a way that allows direct querying based on the encrypted content. To filter or search for documents based on the contents of encrypted fields, you would need to first decrypt the data and then perform the query, which might not be efficient or practical in some cases.
You could however use the [memory mapped](./rx-storage-memory-mapped.md) RxStorage to replicate the encrypted documents into a non-encrypted in-memory storage and then query them like normal.
## Password handling
RxDB does not define how you should store or retrieve the encryption password. It only requires you to provide the password on database creation which grants you flexibility in how you manage encryption passwords.
You could ask the user on app-start to insert the password, or you can retrieve the password from your backend on app start (or revoke access by no longer providing the password).
## Asymmetric encryption
The encryption plugin itself uses **symmetric encryption** with a password to guarantee best performance when reading and storing data.
It is not able to do **Asymmetric encryption** by itself. If you need Asymmetric encryption with a private/publicKey, it is recommended to encrypted the password itself with the asymmetric keys and store the encrypted password beside the other data. On app-start you can decrypt the password with the private key and use the decrypted password in the RxDB encryption plugin
## Using the RxDB Encryption Plugins
RxDB currently has two plugins for encryption:
- The free `encryption-crypto-js` plugin that is based on the `AES` algorithm of the [crypto-js](https://www.npmjs.com/package/crypto-js) library
- The [π premium](/premium/) `encryption-web-crypto` plugin that is based on the native [Web Crypto API](https://developer.mozilla.org/en-US/docs/Web/API/Web_Crypto_API) which makes it faster and more secure to use. Document inserts are about 10x faster compared to `crypto-js` and it has a smaller build size because it uses the browsers API instead of bundling an npm module.
An RxDB encryption plugin is a wrapper around any other [RxStorage](./rx-storage.md).
### Wrap your RxStorage with the encryption
```ts
import {
wrappedKeyEncryptionCryptoJsStorage
} from 'rxdb/plugins/encryption-crypto-js';
import { getRxStorageLocalstorage } from 'rxdb/plugins/storage-localstorage';
// wrap the normal storage with the encryption plugin
const encryptedStorage = wrappedKeyEncryptionCryptoJsStorage({
storage: getRxStorageLocalstorage()
});
```
### Create a RxDatabase with the wrapped storage
Also you have to set a **password** when creating the database. The format of the password depends on which encryption plugin is used.
```ts
import { createRxDatabase } from 'rxdb/plugins/core';
// create an encrypted database
const db = await createRxDatabase({
name: 'mydatabase',
storage: encryptedStorage,
password: 'sudoLetMeIn'
});
```
### Create an RxCollection with an encrypted property
To define a field as being encrypted, you have to add it to the `encrypted` fields list in the schema.
```ts
const schema = {
version: 0,
primaryKey: 'id',
type: 'object',
properties: {
id: {
type: 'string',
maxLength: 100
},
secret: {
type: 'string'
},
},
required: ['id'],
encrypted: ['secret']
};
await db.addCollections({
myDocuments: {
schema
}
})
```
## Using Web-Crypto API
For professionals, we have the `web-crypto` [π premium](/premium/) plugin which is faster and more secure:
```ts
import {
wrappedKeyEncryptionWebCryptoStorage,
createPassword
} from 'rxdb-premium/plugins/encryption-web-crypto';
import { getRxStorageIndexedDB } from 'rxdb-premium/plugins/storage-indexeddb';
// wrap the normal storage with the encryption plugin
const encryptedIndexedDbStorage = wrappedKeyEncryptionWebCryptoStorage({
storage: getRxStorageIndexedDB()
});
const myPasswordObject = {
// Algorithm can be oneOf: 'AES-CTR' | 'AES-CBC' | 'AES-GCM'
algorithm: 'AES-CTR',
password: 'myRandomPasswordWithMin8Length'
};
// create an encrypted database
const db = await createRxDatabase({
name: 'mydatabase',
storage: encryptedIndexedDbStorage,
password: myPasswordObject
});
/* ... */
```
## Changing the password
The password is set database specific and it is not possible to change the password of a database. Opening an existing database with a different password will throw an error. To change the password you can either:
- Use the [storage migration plugin](./migration-storage.md) to migrate the database state into a new database.
- Store a randomly created meta-password in a different RxDatabase as a value of a [local document](./rx-local-document.md). Encrypt the meta password with the actual user password and read it out before creating the actual database.
## Encrypted attachments
To store the [attachments](./rx-attachment.md) data encrypted, you have to set `encrypted: true` in the `attachments` property of the schema.
```ts
const mySchema = {
version: 0,
type: 'object',
properties: {
/* ... */
},
attachments: {
encrypted: true // if true, the attachment-data will be encrypted with the db-password
}
};
```
## Encryption and workers
If you are using [Worker RxStorage](./rx-storage-worker.md) or [SharedWorker RxStorage](./rx-storage-shared-worker.md) with encryption, it's recommended to run encryption inside of the worker. Encryption can be very cpu intensive and would take away CPU-power from the main thread which is the main reason to use workers.
You do not need to worry about setting the password inside of the worker. The password will be set when calling createRxDatabase from the main thread, and will be passed internally to the storage in the worker automatically.
## FAQ
What are some JavaScript libraries for client side field encryption?
RxDB provides robust plugins for client side field encryption directly within your javascript database. You encrypt sensitive document properties transparently before they save to local storage. The `encryption-crypto-js` plugin utilizes AES algorithms for dependable security. The premium `encryption-web-crypto` plugin employs native browser APIs to achieve superior performance. You maintain data confidentiality across Web, React Native, and Node.js environments.
---
## Key Compression
import {Steps} from '@site/src/components/steps';
# Key Compression
With the key compression plugin, documents will be stored in a compressed format which saves up to 40% disc space.
For compression the npm module [jsonschema-key-compression](https://github.com/pubkey/jsonschema-key-compression) is used.
It compresses json-data based on its json-schema while still having valid json. It works by compressing long attribute-names into smaller ones and backwards.
The compression and decompression happens internally, so when you work with a [RxDocument](./rx-document.md), you can access any property like normal.
## Enable key compression
The key compression plugin is a wrapper around any other [RxStorage](./rx-storage.md).
### Wrap your RxStorage with the key compression plugin
```ts
import { wrappedKeyCompressionStorage } from 'rxdb/plugins/key-compression';
import { getRxStorageLocalstorage } from 'rxdb/plugins/storage-localstorage';
const storageWithKeyCompression = wrappedKeyCompressionStorage({
storage: getRxStorageLocalstorage()
});
```
### Create an RxDatabase
```ts
import { createRxDatabase } from 'rxdb/plugins/core';
const db = await createRxDatabase({
name: 'mydatabase',
storage: storageWithKeyCompression
});
```
### Create a compressed RxCollection
```ts
const mySchema = {
keyCompression: true, // set this to true, to enable the keyCompression
version: 0,
primaryKey: 'id',
type: 'object',
properties: {
id: {
type: 'string',
maxLength: 100 // <- the primary key must have set maxLength
}
/* ... */
}
};
await db.addCollections({
docs: {
schema: mySchema
}
});
```
---
## RxDB Logger Plugin - Track & Optimize
# RxDB Logger Plugin
With the logger plugin you can log all operations to the [storage layer](./rx-storage.md) of your [RxDatabase](./rx-database.md).
This is useful to debug performance problems and for monitoring with Application Performance Monitoring (APM) tools like **Bugsnag**, **Datadog**, **Elastic**, **Sentry** and others.
Notice that the logger plugin is not part of the RxDB core, it is part of [RxDB Premium π](/premium/).
## Using the logger plugin
The logger is a wrapper that can be wrapped around any [RxStorage](./rx-storage.md). Once your storage is wrapped, you can create your database with the wrapped storage and the logging will automatically happen.
```ts
import {
wrappedLoggerStorage
} from 'rxdb-premium/plugins/logger';
import {
getRxStorageIndexedDB
} from 'rxdb-premium/plugins/storage-indexeddb';
// wrap a storage with the logger
const loggingStorage = wrappedLoggerStorage({
storage: getRxStorageIndexedDB({})
});
// create your database with the wrapped storage
const db = await createRxDatabase({
name: 'mydatabase',
storage: loggingStorage
});
// create collections etc...
```
## Specify what to be logged
By default, the plugin will log all operations and it will also run a `console.time()/console.timeEnd()` around each operation. You can specify what to log so that your logs are less noisy. For this you provide a settings object when calling `wrappedLoggerStorage()`.
```ts
const loggingStorage = wrappedLoggerStorage({
storage: getRxStorageIndexedDB({}),
settings: {
// can used to prefix all log strings, default=''
prefix: 'my-prefix',
/**
* Be default, all settings are true.
*/
// if true, it will log timings with console.time() and console.timeEnd()
times: true,
// if false, it will not log meta storage instances like used in replication
metaStorageInstances: true,
// operations
bulkWrite: true,
findDocumentsById: true,
query: true,
count: true,
info: true,
getAttachmentData: true,
getChangedDocumentsSince: true,
cleanup: true,
close: true,
remove: true
}
});
```
## Using custom logging functions
With the logger plugin you can also run custom log functions for all operations.
```ts
const loggingStorage = wrappedLoggerStorage({
storage: getRxStorageIndexedDB({}),
onOperationStart: (operationsName, logId, args) => void,
onOperationEnd: (operationsName, logId, args) => void,
onOperationError: (operationsName, logId, args, error) => void
});
```
---
## Remote RxStorage
The Remote [RxStorage](./rx-storage.md) is made to use a remote storage and communicate with it over an asynchronous message channel.
The remote part could be on another JavaScript process or even on a different host machine.
The remote storage plugin is used in many RxDB plugins like the [worker](./rx-storage-worker.md) or the [electron](./electron.md) plugin.
## Usage
The remote storage communicates over a message channel which has to implement the `messageChannelCreator` function which returns an object that has a `messages$` observable and a `send()` function on both sides and a `close()` function that closes the RemoteMessageChannel.
```ts
// on the client
import { getRxStorageRemote } from 'rxdb/plugins/storage-remote';
const storage = getRxStorageRemote({
identifier: 'my-id',
mode: 'storage',
messageChannelCreator: () => Promise.resolve({
messages$: new Subject(),
send(msg) {
// send to remote storage
}
})
});
const myDb = await createRxDatabase({
storage
});
// on the remote
import { getRxStorageLocalstorage } from 'rxdb/plugins/storage-localstorage';
import { exposeRxStorageRemote } from 'rxdb/plugins/storage-remote';
exposeRxStorageRemote({
storage: getRxStorageLocalstorage(),
messages$: new Subject(),
send(msg){
// send to other side
}
});
```
## Usage with a Websocket server
The remote storage plugin contains helper functions to create a remote storage over a WebSocket server.
This is often used in Node.js to give one microservice access to another services database **without** having to replicate the full database state.
```ts
// server.js
import { getRxStorageMemory } from 'rxdb/plugins/storage-memory';
import { startRxStorageRemoteWebsocketServer } from 'rxdb/plugins/storage-remote-websocket';
// either you can create the server based on a RxDatabase
const serverBasedOnDatabase = await startRxStorageRemoteWebsocketServer({
port: 8080,
database: myRxDatabase
});
// or you can create the server based on a pure RxStorage
const serverBasedOn = await startRxStorageRemoteWebsocketServer({
port: 8080,
storage: getRxStorageMemory()
});
```
```ts
// client.js
import { getRxStorageRemoteWebsocket } from 'rxdb/plugins/storage-remote-websocket';
const myDb = await createRxDatabase({
storage: getRxStorageRemoteWebsocket({
url: 'ws://example.com:8080'
})
});
```
## Sending custom messages
The remote storage can also be used to send custom messages to and from the remote instance.
On the remote you have to define a `customRequestHandler` like:
```ts
const serverBasedOnDatabase = await startRxStorageRemoteWebsocketServer({
port: 8080,
database: myRxDatabase,
async customRequestHandler(msg){
// here you can return any JSON object as an 'answer'
return {
foo: 'bar'
};
}
});
```
On the client instance you can then call the `customRequest()` method:
```ts
const storage = getRxStorageRemoteWebsocket({
url: 'ws://example.com:8080'
});
const answer = await storage.customRequest({ bar: 'foo' });
console.dir(answer); // > { foo: 'bar' }
```
---
## Turbocharge RxDB with Worker RxStorage
# Worker RxStorage
With the worker plugin, you can put the [RxStorage](./rx-storage.md) of your database inside of a WebWorker (in browsers) or a Worker Thread (in node.js). By doing so, you can take CPU load from the main process and move it into the worker's process which can improve the perceived performance of your application. Notice that for browsers, it is recommended to use the [SharedWorker](./rx-storage-shared-worker.md) instead to get a better performance.
:::note Premium
This plugin is part of [RxDB Premium π](/premium/). It is not part of the default RxDB module.
:::
## On the worker process
```ts
// worker.ts
import { exposeWorkerRxStorage } from 'rxdb-premium/plugins/storage-worker';
import { getRxStorageIndexedDB } from 'rxdb-premium/plugins/storage-indexeddb';
exposeWorkerRxStorage({
/**
* You can wrap any implementation of the RxStorage interface
* into a worker.
* Here we use the IndexedDB RxStorage.
*/
storage: getRxStorageIndexedDB()
});
```
## On the main process
```ts
import {
createRxDatabase
} from 'rxdb';
import { getRxStorageWorker } from 'rxdb-premium/plugins/storage-worker';
const database = await createRxDatabase({
name: 'mydatabase',
storage: getRxStorageWorker(
{
/**
* Contains any value that can be used as parameter
* to the Worker constructor of thread.js
* Most likely you want to put the path to the worker.js file in here.
*
* @link https://developer.mozilla.org/en-US/docs/Web/API/Worker/Worker
*/
workerInput: 'path/to/worker.js',
/**
* (Optional) options
* for the worker.
*/
workerOptions: {
type: 'module',
credentials: 'omit'
}
}
)
});
```
## Pre-build workers
The `worker.js` must be a self containing JavaScript file that contains all dependencies in a bundle.
To make it easier for you, RxDB ships with pre-bundles worker files that are ready to use.
You can find them in the folder `node_modules/rxdb-premium/dist/workers` after you have installed the [RxDB Premium π Plugin](/premium/). From there you can copy them to a location where it can be served from the webserver and then use their path to create the `RxDatabase`.
Any valid `worker.js` JavaScript file can be used both, for normal Workers and SharedWorkers.
```ts
import {
createRxDatabase
} from 'rxdb';
import { getRxStorageWorker } from 'rxdb-premium/plugins/storage-worker';
const database = await createRxDatabase({
name: 'mydatabase',
storage: getRxStorageWorker(
{
/**
* Path to where the copied file from node_modules/rxdb/dist/workers
* is reachable from the webserver.
*/
workerInput: '/indexeddb.worker.js'
}
)
});
```
## Building a custom worker
The easiest way to bundle a custom `worker.js` file is by using webpack. Here is the webpack-config that is also used for the prebuild workers:
```ts
// webpack.config.js
const path = require('path');
const TerserPlugin = require('terser-webpack-plugin');
const projectRootPath = path.resolve(
__dirname,
'../../' // path from webpack-config to the root folder of the repo
);
const babelConfig = require(path.join(projectRootPath, 'babel.config'));
const baseDir = './dist/workers/'; // output path
module.exports = {
target: 'webworker',
entry: {
'my-custom-worker': baseDir + 'my-custom-worker.js',
},
output: {
filename: '[name].js',
clean: true,
path: path.resolve(
projectRootPath,
'dist/workers'
),
},
mode: 'production',
module: {
rules: [
{
test: /\.tsx?$/,
exclude: /(node_modules)/,
use: {
loader: 'babel-loader',
options: babelConfig
}
}
],
},
resolve: {
extensions: ['.tsx', '.ts', '.js', '.mjs', '.mts']
},
optimization: {
moduleIds: 'deterministic',
minimize: true,
minimizer: [new TerserPlugin({
terserOptions: {
format: {
comments: false,
},
},
extractComments: false,
})],
}
};
```
## One worker per database
Each call to `getRxStorageWorker()` will create a different worker instance so that when you have more than one `RxDatabase`, each database will have its own JavaScript worker process.
To reuse the worker instance in more than one `RxDatabase`, you can store the output of `getRxStorageWorker()` into a variable and use that one. Reusing the worker can decrease the initial page load, but you might get slower database operations.
```ts
// Call getRxStorageWorker() exactly once
const workerStorage = getRxStorageWorker({
workerInput: 'path/to/worker.js'
});
// use the same storage for both databases.
const databaseOne = await createRxDatabase({
name: 'database-one',
storage: workerStorage
});
const databaseTwo = await createRxDatabase({
name: 'database-two',
storage: workerStorage
});
```
## Passing in a Worker instance
Instead of setting an url as `workerInput`, you can also specify a function that returns a new `Worker` instance when called.
```ts
getRxStorageWorker({
workerInput: () => new Worker('path/to/worker.js')
})
```
This can be helpful for environments where the worker is build dynamically by the bundler. For example in angular you would create a `my-custom.worker.ts` file that contains a custom build worker and then import it.
```ts
const storage = getRxStorageWorker({
workerInput: () => new Worker(new URL('./my-custom.worker', import.meta.url)),
});
```
```ts
//> my-custom.worker.ts
import { exposeWorkerRxStorage } from 'rxdb-premium/plugins/storage-worker';
import { getRxStorageIndexedDB } from 'rxdb-premium/plugins/storage-indexeddb';
exposeWorkerRxStorage({
storage: getRxStorageIndexedDB()
});
```
---
## Boost Performance with SharedWorker RxStorage
# SharedWorker RxStorage
The SharedWorker [RxStorage](./rx-storage.md) uses the [SharedWorker API](https://developer.mozilla.org/en-US/docs/Web/API/SharedWorker) to run the storage inside of a separate JavaScript process **in browsers**. Compared to a normal [WebWorker](./rx-storage-worker.md), the SharedWorker is created exactly once, even when there are multiple browser tabs opened. Because of having exactly one worker, multiple performance optimizations can be done because the storage itself does not have to handle multiple opened database connections.
:::note Premium
This plugin is part of [RxDB Premium π](/premium/). It is not part of the default RxDB module.
:::
## Usage
### On the SharedWorker process
In the worker process JavaScript file, you have to wrap the original RxStorage with `getRxStorageIndexedDB()`.
```ts
// shared-worker.ts
import { exposeWorkerRxStorage } from 'rxdb-premium/plugins/storage-worker';
import {
getRxStorageIndexedDB
} from 'rxdb-premium/plugins/storage-indexeddb';
exposeWorkerRxStorage({
/**
* You can wrap any implementation of the RxStorage interface
* into a worker.
* Here we use the IndexedDB RxStorage.
*/
storage: getRxStorageIndexedDB()
});
```
### On the main process
```ts
import {
createRxDatabase
} from 'rxdb';
import { getRxStorageSharedWorker } from 'rxdb-premium/plugins/storage-worker';
import { getRxStorageIndexedDB } from 'rxdb/plugins/storage-indexeddb';
const database = await createRxDatabase({
name: 'mydatabase',
storage: getRxStorageSharedWorker(
{
/**
* Contains any value that can be used as parameter
* to the SharedWorker constructor of thread.js
* Most likely you want to put the path to the shared-worker.js file in here.
*
* @link https://developer.mozilla.org/en-US/docs/Web/API/SharedWorker?retiredLocale=de
*/
workerInput: 'path/to/shared-worker.js',
/**
* (Optional) options
* for the worker.
*/
workerOptions: {
type: 'module',
credentials: 'omit',
extendedLifetime: true
}
}
)
});
```
## Pre-build workers
The `shared-worker.js` must be a self containing JavaScript file that contains all dependencies in a bundle.
To make it easier for you, RxDB ships with pre-bundles worker files that are ready to use.
You can find them in the folder `node_modules/rxdb-premium/dist/workers` after you have installed the [RxDB Premium π Plugin](/premium/). From there you can copy them to a location where it can be served from the webserver and then use their path to create the `RxDatabase`
Any valid `worker.js` JavaScript file can be used both, for normal Workers and SharedWorkers.
```ts
import {
createRxDatabase
} from 'rxdb';
import { getRxStorageSharedWorker } from 'rxdb-premium/plugins/storage-worker';
const database = await createRxDatabase({
name: 'mydatabase',
storage: getRxStorageSharedWorker(
{
/**
* Path to where the copied
* file from node_modules/rxdb-premium/dist/workers
* is reachable from the webserver.
*/
workerInput: '/indexeddb.shared-worker.js'
}
)
});
```
## Building a custom worker
To build a custom `worker.js` file, check out the webpack config at the [worker](./rx-storage-worker.md#building-a-custom-worker) documentation. Any worker file form the worker storage can also be used in a shared worker because `exposeWorkerRxStorage` detects where it runs and exposes the correct messaging endpoints.
## Passing in a SharedWorker instance
Instead of setting an url as `workerInput`, you can also specify a function that returns a new `SharedWorker` instance when called. This is mostly used when you have a custom worker file and dynamically import it.
This works equal to the [workerInput of the Worker Storage](./rx-storage-worker.md#passing-in-a-worker-instance)
## Set multiInstance: false
When you know that you only ever create your RxDatabase inside of the shared worker, you might want to set `multiInstance: false` to prevent sending change events across JavaScript realms and to improve performance. Do not set this when you also create the same storage on another realm, like when you have the same RxDatabase once inside the shared worker and once on the main thread.
## Replication with SharedWorker
When a SharedWorker RxStorage is used, it is recommended to run the replication **inside** of the worker. This is the best option for performance. You can do that by opening another [RxDatabase](./rx-database.md) inside of it and starting the replication there. If you are not concerned about performance, you can still start replication on the main thread instead. But you should never run replication on both the main thread **and** the worker.
```ts
// shared-worker.ts
import { exposeWorkerRxStorage } from 'rxdb-premium/plugins/storage-worker';
import {
getRxStorageIndexedDB
} from 'rxdb-premium/plugins/storage-indexeddb';
import {
createRxDatabase,
addRxPlugin
} from 'rxdb';
import {
RxDBReplicationGraphQLPlugin
} from 'rxdb/plugins/replication-graphql';
addRxPlugin(RxDBReplicationGraphQLPlugin);
const baseStorage = getRxStorageIndexedDB();
// first expose the RxStorage to the outside
exposeWorkerRxStorage({
storage: baseStorage
});
/**
* Then create a normal RxDatabase and RxCollections
* and start the replication.
*/
const database = await createRxDatabase({
name: 'mydatabase',
storage: baseStorage
});
await db.addCollections({
humans: {/* ... */}
});
const replicationState = db.humans.syncGraphQL({/* ... */});
```
### Limitations
- The SharedWorker API is [not available in some mobile browser](https://caniuse.com/sharedworkers)
### FAQ
Can I use this plugin with a Service Worker?
No. A Service Worker is not the same as a Shared Worker. While you can use RxDB inside of a ServiceWorker, you cannot use the ServiceWorker as a RxStorage that gets accessed by an outside RxDatabase instance.
---
## Blazing-Fast Memory Mapped RxStorage
# Memory Mapped RxStorage
The memory mapped [RxStorage](./rx-storage.md) is a wrapper around any other RxStorage. The wrapper creates an in-memory storage that is used for query and write operations. This memory instance is kept persistent with a given underlying storage.
## Pros
- Improves read/write performance because these operations run against the in-memory storage.
- Decreases initial page load because it loads all data in a single bulk request. It even detects if the database is used for the first time and then it does not have to await the creation of the persistent storage.
- Can store encrypted data on disc while still being able to run queries on the non-encrypted in-memory state.
## Cons
- It does not support attachments because storing big attachments data in-memory should not be done.
- When the JavaScript process is killed ungracefully like when the browser crashes or the power of the PC is terminated, it might happen that some memory writes are not persisted to the parent storage. This can be prevented with the `awaitWritePersistence` flag.
- The memory-mapped storage can only be used if all data fits into the memory of the JavaScript process. This is normally not a problem because a browser has much memory these days and plain JSON document data is not that big.
- Because it has to await an initial data loading from the parent storage into the memory, initial page load time can increase when much data is already stored. This is likely not a problem when you store less than `10k` documents.
- The `memory-mapped` storage is part of [RxDB Premium π](/premium/). It is not part of the default RxDB core module.
## Using the Memory-Mapped RxStorage
```ts
import {
getRxStorageIndexedDB
} from 'rxdb-premium/plugins/storage-indexeddb';
import {
getMemoryMappedRxStorage
} from 'rxdb-premium/plugins/storage-memory-mapped';
/**
* Here we use the IndexedDB RxStorage as persistence storage.
* Any other RxStorage can also be used.
*/
const parentStorage = getRxStorageIndexedDB();
// wrap the persistent storage with the memory-mapped storage.
const storage = getMemoryMappedRxStorage({
storage: parentStorage
});
// create the RxDatabase like you would do with any other RxStorage
const db = await createRxDatabase({
name: 'myDatabase',
storage,
});
/** ... **/
```
## Multi-Tab Support
By how the memory-mapped storage works, it is not possible to have the same storage open in multiple JavaScript processes. So when you use this in a browser application, you can not open multiple databases when the app is used in multiple browser tabs.
To solve this, use the [SharedWorker Plugin](./rx-storage-shared-worker.md) so that the memory-mapped storage runs inside of a SharedWorker exactly once and is then reused for all browser tabs.
If you have a single JavaScript process, like in a React Native app, you do not have to care about this and can just use the memory-mapped storage in the main process.
## Encryption of the persistent data
Normally RxDB is not capable of running queries on encrypted fields. But when you use the memory-mapped RxStorage, you can store the document data encrypted on disc, while being able to run queries on the not encrypted in-memory state. Make sure you use the encryption storage wrapper around the persistent storage, **NOT** around the memory-mapped storage as a whole.
```ts
import {
getRxStorageIndexedDB
} from 'rxdb-premium/plugins/storage-indexeddb';
import {
getMemoryMappedRxStorage
} from 'rxdb-premium/plugins/storage-memory-mapped';
import { wrappedKeyEncryptionWebCryptoStorage } from 'rxdb-premium/plugins/encryption-web-crypto';
const storage = getMemoryMappedRxStorage({
storage: wrappedKeyEncryptionWebCryptoStorage({
storage: getRxStorageIndexedDB()
})
});
const db = await createRxDatabase({
name: 'myDatabase',
storage,
});
/** ... **/
```
## Await Write Persistence
Running operations on the memory-mapped storage by default returns directly when the operation has run on the in-memory state and then persist changes in the background.
Sometimes you might want to ensure write operations is persisted, you can do this by setting `awaitWritePersistence: true`.
```ts
const storage = getMemoryMappedRxStorage({
awaitWritePersistence: true,
storage: getRxStorageIndexedDB()
});
```
## Block Size Limit
During cleanup, the memory-mapped storage will merge many small write-blocks into single big blocks for better initial load performance.
The `blockSizeLimit` defines the maximum of how many documents get stored in a single block. The default is `10000`.
```ts
const storage = getMemoryMappedRxStorage({
blockSizeLimit: 1000,
storage: getRxStorageIndexedDB()
});
```
## Migrating from other Storages
When you switch from a "normal" persistent storage (like [IndexedDB](./rx-storage-indexeddb.md) or [SQLite](./rx-storage-sqlite.md)) to the memory-mapped storage, you **must** migrate the data using the [Storage Migrator](./migration-storage.md).
You cannot simply switch the storage adapter on an existing database because the memory-mapped storage uses a different internal data structure.
To provide the fast initial page load and low write latency, the memory-mapped storage saves data in a "blockchain-like" structure. Writes are appended in blocks rather than modifying the state in place. These blocks are lazily cleaned up and processed later when the CPU is idle (see [Idle Functions](./rx-database.md#requestidlepromise)).
---
## Instant Performance with Memory Synced RxStorage
# Memory Synced RxStorage
The memory synced [RxStorage](./rx-storage.md) is a wrapper around any other RxStorage. The wrapper creates an in-memory storage that is used for query and write operations. This memory instance is replicated with the underlying storage for persistence.
The main reason to use this is to improve initial page load and query/write times. This is mostly useful in browser based applications.
## Pros
- Improves read/write performance because these operations run against the in-memory storage.
- Decreases initial page load because it load all data in a single bulk request. It even detects if the database is used for the first time and then it does not have to await the creation of the persistent storage.
## Cons
- It does not support attachments.
- When the JavaScript process is killed ungracefully like when the browser crashes or the power of the PC is terminated, it might happen that some memory writes are not persisted to the parent storage. This can be prevented with the `awaitWritePersistence` flag.
- This can only be used if all data fits into the memory of the JavaScript process. This is normally not a problem because a browser has much memory these days and plain json document data is not that big.
- Because it has to await an initial replication from the parent storage into the memory, initial page load time can increase when much data is already stored. This is likely not a problem when you store less than `10k` documents.
- The memory-synced storage itself does not support replication and migration. Instead you have to replicate the underlying parent storage.
- The `memory-synced` plugin is part of [RxDB Premium π](/premium/). It is not part of the default RxDB module.
:::note The memory-synced RxStorage was removed in RxDB version 16
The `memory-synced` was removed in RxDB version 16. Instead consider using the newer and better [memory-mapped RxStorage](./rx-storage-memory-mapped.md) which has better trade-offs and is easier to configure.
:::
## Usage
```ts
import {
getRxStorageIndexedDB
} from 'rxdb-premium/plugins/storage-indexeddb';
import {
getMemorySyncedRxStorage
} from 'rxdb-premium/plugins/storage-memory-synced';
/**
* Here we use the IndexedDB RxStorage as persistence storage.
* Any other RxStorage can also be used.
*/
const parentStorage = getRxStorageIndexedDB();
// wrap the persistent storage with the memory synced one.
const storage = getMemorySyncedRxStorage({
storage: parentStorage
});
// create the RxDatabase like you would do with any other RxStorage
const db = await createRxDatabase({
name: 'myDatabase,
storage,
});
/** ... **/
```
## Options
Some options can be provided to fine tune the performance and behavior.
```ts
import {
requestIdlePromise
} from 'rxdb';
const storage = getMemorySyncedRxStorage({
storage: parentStorage,
/**
* Defines how many document
* get replicated in a single batch.
* [default=50]
*
* (optional)
*/
batchSize: 50,
/**
* By default, the parent storage will be created without indexes for a faster page load.
* Indexes are not needed because the queries will anyway run on the memory storage.
* You can disable this behavior by setting keepIndexesOnParent to true.
* If you use the same parent storage for multiple RxDatabase instances where one is not
* a asynced-memory storage, you will get the error: 'schema not equal to existing storage'
* if you do not set keepIndexesOnParent to true.
*
* (optional)
*/
keepIndexesOnParent: true,
/**
* If set to true, all write operations will resolve AFTER the writes
* have been persisted from the memory to the parentStorage.
* This ensures writes are not lost even if the JavaScript process exits
* between memory writes and the persistence interval.
* default=false
*/
awaitWritePersistence: true,
/**
* After a write, await until the return value of this method resolves
* before replicating with the master storage.
*
* By returning requestIdlePromise() we can ensure that the CPU is idle
* and no other, more important operation is running. By doing so we can be sure
* that the replication does not slow down any rendering of the browser process.
*
* (optional)
*/
waitBeforePersist: () => requestIdlePromise();
});
```
## Replication and Migration with the memory-synced storage
The memory-synced storage itself does not support replication and migration. Instead you have to replicate the underlying parent storage.
For example when you use it on top of an [IndexedDB storage](./rx-storage-indexeddb.md), you have to run replication on that storage instead by creating a different [RxDatabase](./rx-database.md).
```js
const parentStorage = getRxStorageIndexedDB();
const memorySyncedStorage = getMemorySyncedRxStorage({
storage: parentStorage,
keepIndexesOnParent: true
});
const databaseName = 'mydata';
/**
* Create a parent database with the same name+collections
* and use it for replication and migration.
* The parent database must be created BEFORE the memory-synced database
* to ensure migration has already been run.
*/
const parentDatabase = await createRxDatabase({
name: databaseName,
storage: parentStorage
});
await parentDatabase.addCollections(/* ... */);
replicateRxCollection({
collection: parentDatabase.myCollection,
/* ... */
});
/**
* Create an equal memory-synced database with the same name+collections
* and use it for writes and queries.
*/
const memoryDatabase = await createRxDatabase({
name: databaseName,
storage: memorySyncedStorage
});
await memoryDatabase.addCollections(/* ... */);
```
---
## Sharding RxStorage π
# Sharding RxStorage
With the sharding plugin, you can improve the write and query times of **some** `RxStorage` implementations.
For example on [slow IndexedDB](./slow-indexeddb.md), a performance gain of **30-50% on reads**, and **25% on writes** can be achieved by using multiple IndexedDB Stores instead of putting all documents into the same store.
The sharding plugin works as a wrapper around any other `RxStorage`. The sharding plugin will automatically create multiple shards per storage instance and it will merge and split read and write calls to it.
:::note Premium
The sharding plugin is part of [RxDB Premium π](/premium/). It is not part of the default RxDB module.
:::
## Using the sharding plugin
```ts
import {
getRxStorageSharding
} from 'rxdb-premium/plugins/storage-sharding';
import { getRxStorageLocalstorage } from 'rxdb/plugins/storage-localstorage';
/**
* First wrap the original RxStorage with the sharding RxStorage.
*/
const shardedRxStorage = getRxStorageSharding({
/**
* Here we use the localStorage RxStorage,
* it is also possible to use any other RxStorage instead.
*/
storage: getRxStorageLocalstorage()
});
/**
* Add the sharding options to your schema.
* Changing these options will require a data migration.
*/
const mySchema = {
/* ... */
sharding: {
/**
* Amount of shards per RxStorage instance.
* Depending on your data size and query patterns, the optimal shard amount may differ.
* Do a performance test to optimize that value.
* 10 Shards is a good value to start with.
*
* IMPORTANT: Changing the value of shards is not possible on a already existing database state,
* you will lose access to your data.
*/
shards: 10,
/**
* Sharding mode,
* you can either shard by collection or by database.
* For most cases you should use 'collection' which will shard on the collection level.
* For example with the IndexedDB RxStorage, it will then create multiple stores per IndexedDB database
* and not multiple IndexedDB databases, which would be slower.
*/
mode: 'collection'
}
/* ... */
}
/**
* Create the RxDatabase with the wrapped RxStorage.
*/
const database = await createRxDatabase({
name: 'mydatabase',
storage: shardedRxStorage
});
```
---
## Fastest RxDB Starts - Localstorage Meta Optimizer
# RxStorage Localstorage Meta Optimizer
The [RxStorage](./rx-storage.md) Localstorage Meta Optimizer is a wrapper around any other RxStorage. The wrapper uses the original RxStorage for normal collection documents. But to optimize the initial page load time, it uses [localstorage](https://developer.mozilla.org/en-US/docs/Web/API/Window/localStorage?retiredLocale=de) to store the plain key-value metadata that RxDB needs to create databases and collections. This plugin can only be used in browsers.
Depending on your database usage and the collection amount, this can save about 200 milliseconds on the initial pageload. It is recommended to use this when you create more than 4 [RxCollections](./rx-collection.md).
:::note Premium
This plugin is part of [RxDB Premium π](/premium/). It is not part of the default RxDB module.
:::
## Usage
The meta optimizer gets wrapped around any other RxStorage. It will then automatically detect if an RxDB internal storage instance is created, and replace that with a [localstorage](./articles/localstorage.md) based instance.
```ts
import {
getLocalstorageMetaOptimizerRxStorage
} from 'rxdb-premium/plugins/storage-localstorage-meta-optimizer';
import { getRxStorageIndexedDB } from 'rxdb-premium/plugins/storage-indexeddb';
/**
* First wrap the original RxStorage with the optimizer.
*/
const optimizedRxStorage = getLocalstorageMetaOptimizerRxStorage({
/**
* Here we use the IndexedDB RxStorage,
* it is also possible to use any other RxStorage instead.
*/
storage: getRxStorageIndexedDB()
});
/**
* Create the RxDatabase with the wrapped RxStorage.
*/
const database = await createRxDatabase({
name: 'mydatabase',
storage: optimizedRxStorage
});
```
---
## Seamless Electron Storage with RxDB
# Electron Plugin
## RxStorage Electron IpcRenderer & IpcMain
To use RxDB in [electron](./electron-database.md), it is recommended to run the RxStorage in the main process and the RxDatabase in the renderer processes. With the rxdb electron plugin you can create a [remote RxStorage](./rx-storage-remote.md) and consume it from the renderer process.
To do this in a convenient way, the RxDB electron plugin provides the helper functions `exposeIpcMainRxStorage` and `getRxStorageIpcRenderer`.
Similar to the [Worker RxStorage](./rx-storage-worker.md), these wrap any other [RxStorage](./rx-storage.md) once in the main process and once in each renderer process. In the renderer you can then use the storage to create a [RxDatabase](./rx-database.md) which communicates with the storage of the main process to store and query data.
:::note
`nodeIntegration` must be enabled in [Electron](https://www.electronjs.org/docs/latest/api/browser-window#new-browserwindowoptions).
:::
```ts
// main.js
const { exposeIpcMainRxStorage } = require('rxdb/plugins/electron');
const { getRxStorageMemory } = require('rxdb/plugins/storage-memory');
app.on('ready', async function () {
exposeIpcMainRxStorage({
key: 'main-storage',
storage: getRxStorageMemory(),
ipcMain: electron.ipcMain
});
});
```
```ts
// renderer.js
const { getRxStorageIpcRenderer } = require('rxdb/plugins/electron');
const { getRxStorageMemory } = require('rxdb/plugins/storage-memory');
const db = await createRxDatabase({
name,
storage: getRxStorageIpcRenderer({
key: 'main-storage',
ipcRenderer: electron.ipcRenderer
})
});
/* ... */
```
## Related
- [Comparison of Electron Databases](./electron-database.md)
---
## βοΈ RxDB realtime Sync Engine for Local-First Apps
# RxDB's realtime Sync Engine for Local-First Apps
The RxDB Sync Engine provides the ability to sync the database state in **realtime** between the clients and the server.
The backend server does not have to be an RxDB instance; you can build a replication with **any infrastructure**.
For example you can replicate with a [custom GraphQL endpoint](./replication-graphql.md) or an [HTTP server](./replication-http.md) on top of a PostgreSQL or MongoDB database.
The replication is made to support the [Local-First](./articles/local-first-future.md) paradigm, so that when the client goes [offline](./offline-first.md), the RxDB [database](./rx-database.md) can still read and write [locally](./articles/local-database.md) and will continue the replication when the client goes online again.
## Design Decisions of the Sync Engine
In contrast to other (server-side) database replication protocols, the RxDB Sync Engine was designed with these goals in mind:
- **Easy to Understand**: The sync engine works in a simple "git-like" way that is easy to understand for an average developer. You only have to understand how three simple endpoints work.
- **Complex Parts are in RxDB, not in the Backend**: The complex parts of the Sync Engine, like [conflict handling](./transactions-conflicts-revisions.md) or offline-online switches, are implemented inside of RxDB itself. This makes creating a compatible backend very easy.
- **Compatible with any Backend**: Because the complex parts are in RxDB, the backend can be "dumb" which makes the protocol compatible to almost every backend. No matter if you use PostgreSQL, MongoDB or anything else.
- **Performance is optimized for Client Devices and Browsers**: By grouping updates and fetches into batches, it is faster to transfer and easier to compress. Client devices and browsers can also process this data faster, for example running `JSON.parse()` on a chunk of data is faster than calling it once per row. Same goes for how client side storage like [IndexedDB](./rx-storage-indexeddb.md) or [OPFS](./rx-storage-opfs.md) works where writing data in bulks is faster.
- **Offline-First Support**: By incorporating conflict handling at the client side, the protocol fully supports [offline-first apps](./offline-first.md). Users can continue making changes while offline, and those updates will sync seamlessly once a connection is reestablished - all without risking data loss or having undefined behavior.
- **Multi-Tab Support**: When RxDB is used in a browser and multiple tabs of the same application are opened, only exactly one runs the replication at any given time. This reduces client- and backend resources.
## The Sync Engine on the document level
On the RxDocument level, the replication works like git, where the fork/client contains all new writes and must be merged with the master/server before it can push its new state to the master/server.
```
A---B-----------D master/server state
\ /
B---C---D fork/client state
```
- The client pulls the latest state `B` from the master.
- The client does some changes `C+D`.
- The client pushes these changes to the master by sending the latest known master state `B` and the new client state `D` of the document.
- If the master state is equal to the latest master `B` state of the client, the new client state `D` is set as the latest master state.
- If the master also had changes and so the latest master change is different than the one that the client assumes, we have a conflict that has to be resolved on the client.
## The Sync Engine on the transfer level
When document states are transferred, all handlers use batches of documents for better performance.
The server **must** implement the following methods to be compatible with the replication:
- **pullHandler** Get the last checkpoint (or null) as input. Returns all documents that have been written **after** the given checkpoint. Also returns the checkpoint of the latest written returned document.
- **pushHandler** a method that can be called by the client to send client-side writes to the master. It gets an array with the `assumedMasterState` and the `newForkState` of each document write as input. It must return an array that contains the master document states of all conflicts. If there are no conflicts, it must return an empty array.
- **pullStream** an observable that emits batches of all master writes and the latest checkpoint of the write batches.
```
+--------+ +--------+
| | pullHandler() | |
| |---------------------> | |
| | | |
| | | |
| Client | pushHandler() | Server |
| |---------------------> | |
| | | |
| | pullStream$ | |
| | <-------------------------| |
+--------+ +--------+
```
The replication runs in two **different modes**:
### Checkpoint iteration
On first initial replication, or when the client comes online again, a checkpoint based iteration is used to catch up with the server state.
A checkpoint is a subset of the fields of the last pulled document. When the checkpoint is sent to the backend via `pullHandler()`, the backend must be able to respond with all documents that have been written **after** the given checkpoint.
For example if your documents contain an `id` and an `updatedAt` field, these two can be used as checkpoint.
When the checkpoint iteration reaches the last checkpoint, where the backend returns an empty array because there are no newer documents, the replication will automatically switch to the `event observation` mode.
### Event observation
While the client is connected to the backend, the events from the backend are observed via `pullStream$` and persisted to the client.
If your backend for any reason is not able to provide a full `pullStream$` that contains all events and the checkpoint, you can instead only emit `RESYNC` events that tell RxDB that anything unknown has changed on the server and it should run the pull replication via [checkpoint iteration](#checkpoint-iteration).
When the client goes offline and online again, it might happen that the `pullStream$` has missed out some events. Therefore the `pullStream$` should also emit a `RESYNC` event each time the client reconnects, so that the client can become in sync with the backend via the [checkpoint iteration](#checkpoint-iteration) mode.
## Data layout on the server
To use the replication you first have to ensure that:
- **documents are deterministically sortable by their last write time**
*deterministic* means that even if two documents have the same *last write time*, they have a predictable sort order.
This is most often ensured by using the *primaryKey* as second sort parameter as part of the checkpoint.
- **documents are never deleted, instead the `_deleted` field is set to `true`.**
This is needed so that the deletion state of a document exists in the database and can be replicated to other instances. If your backend uses a different field to mark deleted documents, you have to transform the data in the push/pull handlers or with the modifiers.
For example if your documents look like this:
```ts
const docData = {
"id": "foobar",
"name": "Alice",
"lastName": "Wilson",
/**
* Contains the last write timestamp
* so all document writes can be sorted by that value
* when they are fetched from the remote instance.
*/
"updatedAt": 1564483474,
/**
* Instead of physically deleting documents,
* a deleted document gets replicated.
*/
"_deleted": false
}
```
Then your data is always sortable by `updatedAt`. This ensures that when RxDB fetches 'new' changes via `pullHandler()`, it can send the latest `updatedAt+id` checkpoint to the remote endpoint and then receive all newer documents.
By default, the field is `_deleted`. If your remote endpoint uses a different field to mark deleted documents, you can set the `deletedField` in the replication options which will automatically map the field on all pull and push requests.
## Conflict handling
When multiple clients (or the server) modify the same document at the same time (or when they are offline), it can happen that a conflict arises during the replication.
```
A---B1---C1---X master/server state
\ /
B1---C2 fork/client state
```
In the case above, the client would tell the master to move the document state from `B1` to `C2` by calling `pushHandler()`. But because the actual master state is `C1` and not `B1`, the master would reject the write by sending back the actual master state `C1`.
**RxDB resolves all conflicts on the client** so it would call the conflict handler of the [RxCollection](./rx-collection.md) and create a new document state `D` that can then be written to the master.
```
A---B1---C1---X---D master/server state
\ / \ /
B1---C2---D fork/client state
```
The default conflict handler will always drop the fork state and use the master state. This ensures that clients that are offline for a very long time, do not accidentally overwrite other peoples changes when they go online again.
You can specify a custom conflict handler by setting the property `conflictHandler` when calling `addCollection()`.
Learn how to create a [custom conflict handler](./transactions-conflicts-revisions.md#custom-conflict-handler).
## replicateRxCollection()
You can start the replication of a single `RxCollection` by calling `replicateRxCollection()` like in the following:
```ts
import { replicateRxCollection } from 'rxdb/plugins/replication';
import {
lastOfArray
} from 'rxdb';
const replicationState = await replicateRxCollection({
collection: myRxCollection,
/**
* An id for the replication to identify it
* and so that RxDB is able to resume the replication on app reload.
* If you replicate with a remote server, it is recommended to put the
* server url into the replicationIdentifier.
*/
replicationIdentifier: 'my-rest-replication-to-https://example.com/api/sync',
/**
* By default it will do an ongoing realtime replication.
* By settings live: false the replication will run once until the local state
* is in sync with the remote state, then it will cancel itself.
* (optional), default is true.
*/
live: true,
/**
* Time in milliseconds after when a failed backend request
* has to be retried.
* This time will be skipped if a offline->online switch is detected
* via navigator.onLine
* (optional), default is 5 seconds.
*/
retryTime: 5 * 1000,
/**
* When multiInstance is true, like when you use RxDB in multiple browser tabs,
* the replication should always run in only one of the open browser tabs.
* If waitForLeadership is true, it will wait until the current instance is leader.
* If waitForLeadership is false, it will start replicating, even if it is not leader.
* [default=true]
*/
waitForLeadership: true,
/**
* If this is set to false,
* the replication will not start automatically
* but will wait for replicationState.start() being called.
* (optional), default is true
*/
autoStart: true,
/**
* Custom deleted field, the boolean property of the document data that
* marks a document as being deleted.
* If your backend uses a different field name than '_deleted', set the field name here.
* RxDB will still store the documents internally with '_deleted', setting this field
* only maps the data on the data layer.
*
* If a custom deleted field contains a non-boolean value, the deleted state
* of the documents depends on if the value is truthy or not. So instead of providing a boolean deleted value, you could also work with using a 'deletedAt' timestamp instead.
*
* [default='_deleted']
*/
deletedField: 'deleted',
/**
* Optional,
* only needed when you want to replicate local changes to the remote instance.
*/
push: {
/**
* Push handler
*/
async handler(docs) {
/**
* Push the local documents to a remote REST server.
*/
const rawResponse = await fetch('https://example.com/api/sync/push', {
method: 'POST',
headers: {
'Accept': 'application/json',
'Content-Type': 'application/json'
},
body: JSON.stringify({ docs })
});
/**
* Contains an array with all conflicts that appeared during this push.
* If there were no conflicts, return an empty array.
*/
const response = await rawResponse.json();
return response;
},
/**
* Batch size, optional
* Defines how many documents will be given to the push handler at once.
*/
batchSize: 5,
/**
* Modifies all documents before they are given to the push handler.
* Can be used to swap out a custom deleted flag instead of the '_deleted' field.
* If the push modifier return null, the document will be skipped and not sent to the remote.
* Notice that the modifier can be called multiple times and should not contain any side effects.
* (optional)
*/
modifier: d => d,
/**
* When a local write happens, the replication will normally start pushing immediately.
* By providing a function here that returns a promise, the replication waits for that
* promise to resolve before starting the next upstream persist cycle.
* This lets you batch writes from multiple rapid inserts into a single push call,
* or defer pushing until the CPU is idle (e.g. via requestIdleCallback).
* NOTE: The longer you wait, the higher the risk of losing writes if the replication
* closes unexpectedly.
* (optional)
*/
waitBeforePersist: () => new Promise(resolve => requestIdleCallback(resolve))
},
/**
* Optional,
* only needed when you want to replicate remote changes to the local state.
*/
pull: {
/**
* Pull handler
*/
async handler(lastCheckpoint, batchSize) {
const minTimestamp = lastCheckpoint ? lastCheckpoint.updatedAt : 0;
/**
* In this example we replicate with a remote REST server
*/
const response = await fetch(
`https://example.com/api/sync/?minUpdatedAt=${minTimestamp}&limit=${batchSize}`
);
const documentsFromRemote = await response.json();
return {
/**
* Contains the pulled documents from the remote.
* Not that if documentsFromRemote.length < batchSize,
* then RxDB assumes that there are no more un-replicated documents
* on the backend, so the replication will switch to 'Event observation' mode.
*/
documents: documentsFromRemote,
/**
* The last checkpoint of the returned documents.
* On the next call to the pull handler,
* this checkpoint will be passed as 'lastCheckpoint'
*/
checkpoint: documentsFromRemote.length === 0 ? lastCheckpoint : {
id: lastOfArray(documentsFromRemote).id,
updatedAt: lastOfArray(documentsFromRemote).updatedAt
}
};
},
batchSize: 10,
/**
* Modifies all documents after they have been pulled
* but before they are used by RxDB.
* Notice that the modifier can be called multiple times and should not contain any side effects.
* (optional)
*/
modifier: d => d,
/**
* Stream of the backend document writes.
* See below.
* You only need a stream$ when you have set live=true
*/
stream$: pullStream$.asObservable()
},
});
/**
* Creating the pull stream for realtime replication.
* Here we use a websocket but any other way of sending data to the client can be used,
* like long polling or server-sent events.
*/
const pullStream$ = new Subject>();
let firstOpen = true;
function connectSocket() {
const socket = new WebSocket('wss://example.com/api/sync/stream');
/**
* When the backend sends a new batch of documents+checkpoint,
* emit it into the stream$.
*
* event.data must look like this
* {
* documents: [
* {
* id: 'foobar',
* _deleted: false,
* updatedAt: 1234
* }
* ],
* checkpoint: {
* id: 'foobar',
* updatedAt: 1234
* }
* }
*/
socket.onmessage = event => pullStream$.next(event.data);
/**
* Automatically reconnect the socket on close and error.
*/
socket.onclose = () => connectSocket();
socket.onerror = () => socket.close();
socket.onopen = () => {
if(firstOpen) {
firstOpen = false;
} else {
/**
* When the client is offline and goes online again,
* it might have missed out events that happened on the server.
* So we have to emit a RESYNC so that the replication goes
* into 'Checkpoint iteration' mode until the client is in sync
* and then it will go back into 'Event observation' mode again.
*/
pullStream$.next('RESYNC');
}
}
}
```
## Multi Tab support
For better performance, the replication runs only in one instance when RxDB is used in multiple browser tabs or Node.js processes.
By setting `waitForLeadership: false` you can enforce that each tab runs its own replication cycles.
If used in a multi instance setting, so when at database creation `multiInstance: false` was not set,
you need to import the [leader election plugin](./leader-election.md) so that RxDB can know how many instances exist and which browser tab should run the replication.
## Error handling
When sending a document to the remote fails for any reason, RxDB will send it again in a later point in time.
This happens for **all** errors. The document write could have already reached the remote instance and be processed, while only the answering fails.
The remote instance must be designed to handle this properly and to not crash on duplicate data transmissions.
Depending on your use case, it might be ok to just write the duplicate document data again.
But for a more resilient error handling you could compare the last write timestamps or add a unique write id field to the document. This field can then be used to detect duplicates and ignore re-sent data.
Also the replication has an `.error$` stream that emits all [RxError](./errors.md) objects that arise during replication.
Notice that these errors contain an inner `.parameters.errors` field that contains the original error. Also they contain a `.parameters.direction` field that indicates if the error was thrown during `pull` or `push`. You can use these to properly handle errors. For example when the client is outdated, the server might respond with a `426 Upgrade Required` error code that can then be used to force a page reload.
```ts
replicationState.error$.subscribe((error) => {
if(
error.parameters.errors &&
error.parameters.errors[0] &&
error.parameters.errors[0].code === 426
) {
// client is outdated -> enforce a page reload
location.reload();
}
});
```
## Security
Be aware that client side clocks can never be trusted. When you have a client-backend replication, the backend should overwrite the `updatedAt` timestamp or use another field, when it receives the change from the client.
## RxReplicationState
The function `replicateRxCollection()` returns a `RxReplicationState` that can be used to manage and observe the replication.
### Observable
To observe the replication, the `RxReplicationState` has some `Observable` properties:
```ts
// emits each document that was received from the remote
myRxReplicationState.received$.subscribe(doc => console.dir(doc));
// emits each document that was sent to the remote
myRxReplicationState.sent$.subscribe(doc => console.dir(doc));
// emits all errors that happen when running the push- & pull-handlers.
myRxReplicationState.error$.subscribe(error => console.dir(error));
// emits true when the replication was canceled, false when not.
myRxReplicationState.canceled$.subscribe(bool => console.dir(bool));
// emits true when a replication cycle is running, false when not.
myRxReplicationState.active$.subscribe(bool => console.dir(bool));
```
### awaitInitialReplication()
With `awaitInitialReplication()` you can await the initial replication that is done when a full replication cycle was successfully finished for the first time. The returned promise will never resolve if you cancel the replication before the initial replication can be done.
```ts
await myRxReplicationState.awaitInitialReplication();
```
### awaitInSync()
Returns a `Promise` that resolves when:
- `awaitInitialReplication()` has emitted.
- All local data is replicated with the remote.
- No replication cycle is running or in retry-state.
:::warning
When `multiInstance: true` and `waitForLeadership: true` and another tab is already running the replication, `awaitInSync()` will not resolve until the other tab is closed and the replication starts in this tab.
```ts
await myRxReplicationState.awaitInSync();
```
:::
:::warning
#### `awaitInitialReplication()` and `awaitInSync()` should not be used to block the application
A common mistake in RxDB usage is when developers want to block the app usage until the application is in sync.
Often they just `await` the promise of `awaitInitialReplication()` or `awaitInSync()` and show a loading spinner until they resolve. This is dangerous and should not be done because:
- When `multiInstance: true` and `waitForLeadership: true (default)` and another tab is already running the replication, `awaitInitialReplication()` will not resolve until the other tab is closed and the replication starts in this tab.
- Your app can no longer be started when the device is offline because there `awaitInitialReplication()` will never resolve and the app cannot be used.
Instead you should store the last in-sync time in a [local document](./rx-local-document.md) and observe its value on all instances.
For example if you want to block clients from using the app if they have not been in sync for the last 24 hours, you could use this code:
```ts
// update last-in-sync-flag each time replication is in sync
await myCollection.insertLocal('last-in-sync', { time: 0 }).catch(); // ensure flag exists
myReplicationState.active$.pipe(
mergeMap(async() => {
await myReplicationState.awaitInSync();
await myCollection.upsertLocal('last-in-sync', { time: Date.now() })
})
);
// observe the flag and toggle loading spinner
await showLoadingSpinner();
const oneDay = 1000 * 60 * 60 * 24;
await firstValueFrom(
myCollection.getLocal$('last-in-sync').pipe(
filter(d => d.get('time') > (Date.now() - oneDay))
)
);
await hideLoadingSpinner();
```
:::
### reSync()
Triggers a `RESYNC` cycle where the replication goes into [checkpoint iteration](#checkpoint-iteration) until the client is in sync with the backend. Used in unit tests or when no proper `pull.stream$` can be implemented so that the client only knows that something has been changed but not what.
```ts
myRxReplicationState.reSync();
```
If your backend is not capable of sending events to the client at all, you could run `reSync()` in an interval so that the client will automatically fetch server changes after some time at least.
```ts
// trigger RESYNC each 10 seconds.
setInterval(() => myRxReplicationState.reSync(), 10 * 1000);
```
### cancel()
Cancels the replication. Returns a promise that resolves when everything has been cleaned up.
```ts
await myRxReplicationState.cancel();
```
### pause()
Pauses a running replication. The replication can later be resumed with `RxReplicationState.start()`.
```ts
await myRxReplicationState.pause();
await myRxReplicationState.start(); // restart
```
### remove()
Cancels the replication and deletes the metadata of the replication state. This can be used to restart the replication "from scratch". Calling `.remove()` will only delete the replication metadata, it will NOT delete the documents from the collection of the replication.
```ts
await myRxReplicationState.remove();
```
### isStopped()
Returns `true` if the replication is stopped. This can be if a non-live replication is finished or a replication got canceled.
```js
replicationState.isStopped(); // true/false
```
### isPaused()
Returns `true` if the replication is paused.
```js
replicationState.isPaused(); // true/false
```
### Setting a custom initialCheckpoint
By default, the push replication will start from the beginning of time and push all documents from there to the remote.
By setting a custom `push.initialCheckpoint`, you can tell the replication to only push writes that are newer than the given checkpoint.
```ts
// store the latest checkpoint of a collection
let lastLocalCheckpoint: any;
myCollection.checkpoint$.subscribe(checkpoint => lastLocalCheckpoint = checkpoint);
// start the replication but only push documents that are newer than the lastLocalCheckpoint
const replicationState = replicateRxCollection({
collection: myCollection,
replicationIdentifier: 'my-custom-replication-with-init-checkpoint',
/* ... */
push: {
handler: /* ... */,
initialCheckpoint: lastLocalCheckpoint
}
});
```
The same can be done for the other direction by setting a `pull.initialCheckpoint`. Notice that here we need the remote checkpoint from the backend instead of the one from the RxDB storage.
```ts
// get the last pull checkpoint from the server
const lastRemoteCheckpoint = await (await fetch('http://example.com/pull-checkpoint')).json();
// start the replication but only pull documents that are newer than the lastRemoteCheckpoint
const replicationState = replicateRxCollection({
collection: myCollection,
replicationIdentifier: 'my-custom-replication-with-init-checkpoint',
/* ... */
pull: {
handler: /* ... */,
initialCheckpoint: lastRemoteCheckpoint
}
});
```
### toggleOnDocumentVisible
Ensures replication continues running when the document is `visible`. This helps avoid situations where the leader-elected tab becomes stale or is hibernated by the browser to save battery.
When the tab becomes hidden, replication is automatically paused; when the tab becomes visible again (or the instance becomes leader), replication resumes.
**Default:** `true`
```ts
const replicationState = replicateRxCollection({
toggleOnDocumentVisible: true,
/* ... */
});
```
## Attachment replication
Attachment replication is supported in the RxDB Sync Engine itself. However not all replication plugins support it.
If you start the replication with a collection which has [enabled RxAttachments](./rx-attachment.md) attachment data will be added to all push- and write data.
The pushed documents will contain an `_attachments` object which contains:
- The attachment meta data (id, length, digest) of all non-attachments
- The full attachment data of all attachments that have been updated/added from the client.
- Deleted attachments are spared out in the pushed document.
With this data, the backend can decide onto which attachments must be deleted, added or overwritten.
Accordingly, the pulled document must contain the same data, if the backend has a new document state with updated attachments.
## Pull-Only Replication
With the replication protocol it is possible to do pull only replications where data is pulled from a backend but not pushed from the client. RxDB implements some performance optimizations for these like not storing server metadata on pull only streams.
## Partial Sync with RxDB
RxDB supports partial sync patterns where you dynamically manage multiple replication states for different data scopes. This keeps local storage lean and reduces network overhead. Learn more on the dedicated [Partial Sync](./partial-sync.md) page.
## FAQ
I have infinite loops in my replication, how to debug?
When you have infinite loops in your replication or random re-runs of http requests after some time, the reason is likely that your pull-handler
is crashing. To debug this, add a log to the error$ handler to debug it. `myRxReplicationState.error$.subscribe(err => console.log('error$', err))`.
---
## HTTP Replication
import {Steps} from '@site/src/components/steps';
import {Tabs} from '@site/src/components/tabs';
# HTTP Replication from a custom server to RxDB clients
While RxDB has a range of backend-specific replication plugins (like [GraphQL](./replication-graphql.md) or [Firestore](./replication-firestore.md)), the replication is built in a way to make it very easy to replicate data from a custom server to RxDB clients.
Using **HTTP** as a transport protocol makes it simple to create a compatible backend on top of your existing infrastructure. For events that must be sent from the server to the client, we can use [Server-Sent Events](https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events).
In this tutorial we will implement a HTTP replication between an RxDB client and a [MongoDB](./rx-storage-mongodb.md) express server. You can adapt this for any other backend database technology like PostgreSQL or even a non-Node.js server like go or java.
To create a compatible server for replication, we will start a server and implement the correct HTTP routes and replication handlers. We need a push-handler, a pull-handler and for the ongoing changes `pull.stream` we use **Server-Sent Events**.
## Setup
### Start the Replication on the RxDB Client
RxDB does not have a specific HTTP-replication plugin because the [replication primitives plugin](./replication.md) is simple enough to start a HTTP replication on top of it.
We import the `replicateRxCollection` function and start the replication from there for a single [RxCollection](./rx-collection.md).
```ts
// > client.ts
import { replicateRxCollection } from 'rxdb/plugins/replication';
const replicationState = await replicateRxCollection({
collection: myRxCollection,
replicationIdentifier: 'my-http-replication',
push: { /* add settings from below */ },
pull: { /* add settings from below */ }
});
```
### Start a Node.js process with Express and MongoDB
On the server side, we start an express server that has a MongoDB connection and serves the HTTP requests of the client.
```ts
// > server.ts
import { MongoClient } from 'mongodb';
import express from 'express';
const mongoClient = new MongoClient('mongodb://localhost:27017/');
const mongoConnection = await mongoClient.connect();
const mongoDatabase = mongoConnection.db('myDatabase');
const mongoCollection = await mongoDatabase.collection('myDocs');
const app = express();
app.use(express.json());
/* ... add routes from below */
app.listen(80, () => {
console.log(`Example app listening on port 80`)
});
```
### Implement the Pull Endpoint
As first HTTP Endpoint, we need to implement the pull handler. This is used by the RxDB replication to fetch all documents writes that happened after a given `checkpoint`.
The `checkpoint` format is not determined by RxDB, instead the server can use any type of changepoint that can be used to iterate across document writes. Here we will just use a unix timestamp `updatedAt` and a string `id` which is the most common used format.
When the pull endpoint is called, the server responds with an array of document data based on the given checkpoint and a new checkpoint.
Also the server has to respect the batchSize so that RxDB knows when there are no more new documents and the server returns a non-full array.
```ts
// > server.ts
import { lastOfArray } from 'rxdb/plugins/core';
app.get('/pull', (req, res) => {
const id = req.query.id;
const updatedAt = parseFloat(req.query.updatedAt);
const documents = await mongoCollection.find({
$or: [
/**
* Notice that we have to compare the updatedAt AND the id field
* because the updateAt field is not unique and when two documents
* have the same updateAt, we can still "sort" them by their id.
*/
{
updatedAt: { $gt: updatedAt }
},
{
updatedAt: { $eq: updatedAt },
id: { $gt: id }
}
]
})
.sort({updatedAt: 1, id: 1})
.limit(parseInt(req.query.batchSize, 10)).toArray();
const newCheckpoint = documents.length === 0 ? { id, updatedAt } : {
id: lastOfArray(documents).id,
updatedAt: lastOfArray(documents).updatedAt
};
res.setHeader('Content-Type', 'application/json');
res.end(JSON.stringify({ documents, checkpoint: newCheckpoint }));
});
```
### Implement the Pull Handler
On the client we add the `pull.handler` to the replication setting. The handler requests the correct server url and fetches the documents.
```ts
// > client.ts
const replicationState = await replicateRxCollection({
/* ... */
pull: {
async handler(checkpointOrNull, batchSize){
const updatedAt = checkpointOrNull ? checkpointOrNull.updatedAt : 0;
const id = checkpointOrNull ? checkpointOrNull.id : '';
const response = await fetch(
`https://localhost/pull?updatedAt=${updatedAt}&id=${id}&limit=${batchSize}`
);
const data = await response.json();
return {
documents: data.documents,
checkpoint: data.checkpoint
};
}
}
/* ... */
});
```
### Implement the Push Endpoint
To send client side writes to the server, we have to implement the `push.handler`. It gets an array of change rows as input and has to return only the conflicting documents that have not been written to the server. Each change row contains a `newDocumentState` and an optional `assumedMasterState`.
For [conflict detection](./transactions-conflicts-revisions.md), on the server we first have to detect if the `assumedMasterState` is correct for each row. If yes, we have to write the new document state to the database, otherwise we have to return the "real" master state in the conflict array.
The server also creates an `event` that is emitted to the `pullStream$` which is later used in the [pull.stream$](#pullstream-for-ongoing-changes).
```ts
// > server.ts
import { lastOfArray } from 'rxdb/plugins/core';
import { Subject } from 'rxjs';
// used in the pull.stream$ below
let lastEventId = 0;
const pullStream$ = new Subject();
app.get('/push', (req, res) => {
const changeRows = req.body;
const conflicts = [];
const event = {
id: lastEventId++,
documents: [],
checkpoint: null
};
for(const changeRow of changeRows){
const realMasterState = mongoCollection.findOne({id: changeRow.newDocumentState.id});
if(
realMasterState && !changeRow.assumedMasterState ||
(
realMasterState && changeRow.assumedMasterState &&
/*
* For simplicity we detect conflicts on the server by only compare the updateAt value.
* In reality you might want to do a more complex check or do a deep-equal comparison.
*/
realMasterState.updatedAt !== changeRow.assumedMasterState.updatedAt
)
) {
// we have a conflict
conflicts.push(realMasterState);
} else {
// no conflict -> write the document
mongoCollection.updateOne(
{id: changeRow.newDocumentState.id},
changeRow.newDocumentState
);
event.documents.push(changeRow.newDocumentState);
event.checkpoint = { id: changeRow.newDocumentState.id, updatedAt: changeRow.newDocumentState.updatedAt };
}
}
if(event.documents.length > 0){
myPullStream$.next(event);
}
res.setHeader('Content-Type', 'application/json');
res.end(JSON.stringify(conflicts));
});
```
:::note
For simplicity in this tutorial, we do not use transactions. In reality you should run the full push function inside of a MongoDB transaction to ensure that no other process can mix up the document state while the writes are processed. Also you should call batch operations on MongoDB instead of running the operations for each change row.
:::
### Implement the Push Handler
With the push endpoint in place, we can add a `push.handler` to the replication settings on the client.
```ts
// > client.ts
const replicationState = await replicateRxCollection({
/* ... */
push: {
async handler(changeRows){
const rawResponse = await fetch('https://localhost/push', {
method: 'POST',
headers: {
'Accept': 'application/json',
'Content-Type': 'application/json'
},
body: JSON.stringify(changeRows)
});
const conflictsArray = await rawResponse.json();
return conflictsArray;
}
}
/* ... */
});
```
### Implement the pullStream$ Endpoint
While the normal pull handler is used when the replication is in [iteration mode](./replication.md#checkpoint-iteration), we also need a stream of ongoing changes when the replication is in [event observation mode](./replication.md#event-observation). This brings the realtime replication to RxDB where changes on the server or on a client will directly get propagated to the other instances.
On the server we have to implement the `pullStream` route and emit the events. We use the `pullStream$` observable from [above](#push-from-the-client-to-the-server) to fetch all ongoing events and respond them to the client. Here we use Server-Sent-Events (SSE) which is the most commonly used way to stream data from the server to the client. Other method also exist like [WebSockets or Long-Polling](./articles/websockets-sse-polling-webrtc-webtransport.md).
```ts
// > server.ts
app.get('/pullStream', (req, res) => {
res.writeHead(200, {
'Content-Type': 'text/event-stream',
'Connection': 'keep-alive',
'Cache-Control': 'no-cache'
});
const subscription = pullStream$.subscribe(event => {
res.write('data: ' + JSON.stringify(event) + '\n\n');
});
req.on('close', () => subscription.unsubscribe());
});
```
:::note
How to build the `pullStream$` Observable is not part of this tutorial. This heavily depends on your backend and infrastructure. Likely you have to observe the MongoDB event stream.
:::
### Implement the pullStream$ Handler
From the client we can observe this endpoint and create a `pull.stream$` observable that emits all events that are sent from the server to the client.
The client connects to an url and receives server-sent-events that contain all ongoing writes.
```ts
// > client.ts
import { Subject } from 'rxjs';
const myPullStream$ = new Subject();
const eventSource = new EventSource(
'http://localhost/pullStream',
{ withCredentials: true }
);
eventSource.onmessage = event => {
const eventData = JSON.parse(event.data);
myPullStream$.next({
documents: eventData.documents,
checkpoint: eventData.checkpoint
});
};
const replicationState = await replicateRxCollection({
/* ... */
pull: {
/* ... */
stream$: myPullStream$.asObservable()
}
/* ... */
});
```
### pullStream$ RESYNC flag
In case the client loses the connection, the EventSource will automatically reconnect but there might have been some changes that have been missed out in the meantime. The replication has to be informed that it might have missed events by emitting a `RESYNC` flag from the `pull.stream$`.
The replication will then catch up by switching to the [iteration mode](./replication.md#checkpoint-iteration) until it is in sync with the server again.
```ts
// > client.ts
eventSource.onerror = () => myPullStream$.next('RESYNC');
```
The purpose of the `RESYNC` flag is to tell the client that "something might have changed" and then the client can react on that information without having to run operations in an interval.
If your backend is not capable of emitting the actual documents and checkpoint in the pull stream, you could just map all events to the `RESYNC` flag. This would make the replication work with a slight performance drawback:
```ts
// > client.ts
import { Subject } from 'rxjs';
const myPullStream$ = new Subject();
const eventSource = new EventSource(
'http://localhost/pullStream',
{ withCredentials: true }
);
eventSource.onmessage = () => myPullStream$.next('RESYNC');
const replicationState = await replicateRxCollection({
pull: {
stream$: myPullStream$.asObservable()
}
});
```
## Missing implementation details
In this tutorial we only covered the basics of doing a HTTP replication between RxDB clients and a server. We did not cover the following aspects of the implementation:
- Authentication: To authenticate the client on the server, you might want to send authentication headers with the HTTP requests
- Skip events on the `pull.stream$` for the client that caused the changes to improve performance.
- Version upgrades: You should add a version-flag to the endpoint urls. If you then update the version of your endpoints in any way, your old endpoints should emit a `Code 426` to outdated clients so that they can update their client version.
---
## RxDB Server Replication
The *Server Replication Plugin* connects to the replication endpoint of an [RxDB Server Replication Endpoint](./rx-server.md#replication-endpoint) and replicates data between the client and the server.
## Usage
The replication server plugin is imported from the `rxdb-server` npm package. Then you start the replication with a given collection and endpoint url by calling `replicateServer()`.
```ts
import { replicateServer } from 'rxdb-server/plugins/replication-server';
const replicationState = await replicateServer({
collection: usersCollection,
replicationIdentifier: 'my-server-replication',
url: 'http://localhost:80/users/0', // endpoint url with the servers collection schema version at the end
headers: {
Authorization: 'Bearer S0VLU0UhI...'
},
push: {},
pull: {},
live: true
});
```
## outdatedClient$
When you update your schema at the server and run a migration, you end up with a different replication url that has a new schema version number at the end.
Your clients might still be running an old version of your application that will no longer be compatible with the endpoint. Therefore when the client tries to call a server endpoint with an outdated schema version, the `outdatedClient$` observable emits to tell your client that the application must be updated. With that event you can tell the client to update the application.
On browser application you might want to just reload the page on that event:
```ts
replicationState.outdatedClient$.subscribe(() => {
location.reload();
});
```
## unauthorized$
When you clients auth data is not valid (or no longer valid), the server will no longer accept any requests from you client and inform the client that the auth headers must be updated.
The `unauthorized$` observable will emit and expects you to update the headers accordingly so that following requests will be accepted again.
```ts
replicationState.unauthorized$.subscribe(() => {
replicationState.setHeaders({
Authorization: 'Bearer S0VLU0UhI...'
});
});
```
## forbidden$
When you client behaves wrong in any case, like update non-allowed values or changing documents that it is not allowed to,
the server will drop the connection and the replication state will emit on the `forbidden$` observable.
It will also automatically stop the replication so that your client does not accidentally DOS attack the server.
```ts
replicationState.forbidden$.subscribe(() => {
console.log('Client is behaving wrong');
});
```
## Custom EventSource implementation
For the server send events, the [eventsource](https://github.com/EventSource/eventsource) npm package is used instead of the native `EventSource` API. We need this because the native browser API does not support sending headers with the request which is required by the server to parse the auth data.
If the eventsource package does not work for you, you can set an own implementation when creating the replication.
```ts
const replicationState = await replicateServer({
/* ... */
eventSource: MyEventSourceConstructor
/* ... */
});
```
---
## GraphQL Replication
# Replication with GraphQL
The GraphQL replication provides handlers for GraphQL to run [replication](./replication.md) with GraphQL as the transport layer.
The GraphQL replication is mostly used when you already have a backend that exposes a GraphQL API that can be adjusted to serve as a replication endpoint. If you do not already have a GraphQL endpoint, using the [HTTP replication](./replication-http.md) is an easier solution.
:::note
To play around, check out the full example of the RxDB [GraphQL replication with server and client](https://github.com/pubkey/rxdb/tree/master/examples/graphql)
:::
## Usage
Before you use the GraphQL replication, make sure you've learned how the [RxDB replication](./replication.md) works.
### Creating a compatible GraphQL Server
At the server-side, there must exist an endpoint which returns newer rows when the last `checkpoint` is used as input. For example lets say you create a `Query` `pullHuman` which returns a list of document writes that happened after the given checkpoint.
For the push-replication, you also need a `Mutation` `pushHuman` which lets RxDB update data of documents by sending the previous document state and the new client document state.
Also for being able to stream all ongoing events, we need a `Subscription` called `streamHuman`.
```graphql
input HumanInput {
id: ID!,
name: String!,
lastName: String!,
updatedAt: Float!,
deleted: Boolean!
}
type Human {
id: ID!,
name: String!,
lastName: String!,
updatedAt: Float!,
deleted: Boolean!
}
input Checkpoint {
id: String!,
updatedAt: Float!
}
type HumanPullBulk {
documents: [Human]!
checkpoint: Checkpoint
}
type Query {
pullHuman(checkpoint: Checkpoint, limit: Int!): HumanPullBulk!
}
input HumanInputPushRow {
assumedMasterState: HeroInputPushRowT0AssumedMasterStateT0
newDocumentState: HeroInputPushRowT0NewDocumentStateT0!
}
type Mutation {
# Returns a list of all conflicts
# If no document write caused a conflict, return an empty list.
pushHuman(rows: [HumanInputPushRow!]): [Human]
}
# headers are used to authenticate the subscriptions
# over websockets.
input Headers {
AUTH_TOKEN: String!;
}
type Subscription {
streamHuman(headers: Headers): HumanPullBulk!
}
```
The GraphQL resolver for the `pullHuman` would then look like:
```js
const rootValue = {
pullHuman: args => {
const minId = args.checkpoint ? args.checkpoint.id : '';
const minUpdatedAt = args.checkpoint ? args.checkpoint.updatedAt : 0;
// sorted by updatedAt first and the id as second
const sortedDocuments = documents.sort((a, b) => {
if (a.updatedAt > b.updatedAt) return 1;
if (a.updatedAt < b.updatedAt) return -1;
if (a.updatedAt === b.updatedAt) {
if (a.id > b.id) return 1;
if (a.id < b.id) return -1;
else return 0;
}
});
// only return documents newer than the input document
const filterForMinUpdatedAtAndId = sortedDocuments.filter(doc => {
if (doc.updatedAt < minUpdatedAt) return false;
if (doc.updatedAt > minUpdatedAt) return true;
if (doc.updatedAt === minUpdatedAt) {
// if updatedAt is equal, compare by id
if (doc.id > minId) return true;
else return false;
}
});
// only return some documents in one batch
const limitedDocs = filterForMinUpdatedAtAndId.slice(0, args.limit);
// use the last document for the checkpoint
const lastDoc = limitedDocs[limitedDocs.length - 1];
const retCheckpoint = lastDoc ? {
id: lastDoc.id,
updatedAt: lastDoc.updatedAt
} : args.checkpoint;
return {
documents: limitedDocs,
checkpoint: retCheckpoint
};
}
};
```
For examples for the other resolvers, consult the [GraphQL Example Project](https://github.com/pubkey/rxdb/blob/master/examples/graphql/server/index.js).
### RxDB Client
#### Pull replication
For the pull-replication, you first need a `pullQueryBuilder`. This is a function that gets the last replication `checkpoint` and a `limit` as input and returns an object with a GraphQL-query and its variables (or a promise that resolves to the same object). RxDB will use the query builder to construct what is later sent to the GraphQL endpoint.
```js
const pullQueryBuilder = (checkpoint, limit) => {
/**
* The first pull does not have a checkpoint
* so we fill it up with defaults
*/
if (!checkpoint) {
checkpoint = {
id: '',
updatedAt: 0
};
}
const query = `query PullHuman($checkpoint: CheckpointInput, $limit: Int!) {
pullHuman(checkpoint: $checkpoint, limit: $limit) {
documents {
id
name
age
updatedAt
deleted
}
checkpoint {
id
updatedAt
}
}
}`;
return {
query,
operationName: 'PullHuman',
variables: {
checkpoint,
limit
}
};
};
```
With the queryBuilder, you can then setup the pull-replication.
```js
import { replicateGraphQL } from 'rxdb/plugins/replication-graphql';
const replicationState = replicateGraphQL(
{
collection: myRxCollection,
// urls to the GraphQL endpoints
url: {
http: 'http://example.com/graphql'
},
pull: {
queryBuilder: pullQueryBuilder, // the queryBuilder from above
modifier: doc => doc, // (optional) modifies all pulled documents before they are handled by RxDB
dataPath: undefined, // (optional) specifies the object path to access the document(s). Otherwise, the first result of the response data is used.
/**
* Amount of documents that the remote will send in one request.
* If the response contains less than [batchSize] documents,
* RxDB will assume there are no more changes on the backend
* that are not replicated.
* This value is the same as the limit in the pullHuman() schema.
* [default=100]
*/
batchSize: 50
},
// headers which will be used in http requests against the server.
headers: {
Authorization: 'Bearer abcde...'
},
/**
* Options that have been inherited from the RxReplication
*/
deletedField: 'deleted',
live: true,
retryTime: 1000 * 5,
waitForLeadership: true,
autoStart: true,
}
);
```
#### Push replication
For the push-replication, you also need a `queryBuilder`. Here, the builder receives a changed document as input which has to be sent to the server. It also returns a GraphQL-Query and its data.
```js
const pushQueryBuilder = rows => {
const query = `
mutation PushHuman($writeRows: [HumanInputPushRow!]) {
pushHuman(writeRows: $writeRows) {
id
name
age
updatedAt
deleted
}
}
`;
const variables = {
writeRows: rows
};
return {
query,
operationName: 'PushHuman',
variables
};
};
```
With the queryBuilder, you can then setup the push-replication.
```js
const replicationState = replicateGraphQL(
{
collection: myRxCollection,
// urls to the GraphQL endpoints
url: {
http: 'http://example.com/graphql'
},
push: {
queryBuilder: pushQueryBuilder, // the queryBuilder from above
/**
* batchSize (optional)
* Amount of document that will be pushed to the server in a single request.
*/
batchSize: 5,
/**
* modifier (optional)
* Modifies all pushed documents before they are sent to the GraphQL endpoint.
* Returning null will skip the document.
*/
modifier: doc => doc
},
headers: {
Authorization: 'Bearer abcde...'
},
pull: {
/* ... */
},
/* ... */
}
);
```
#### Pull Stream
To create a **realtime** replication, you need to create a pull stream that pulls ongoing writes from the server.
The pull stream gets the `headers` of the `RxReplicationState` as input, so that it can be authenticated on the backend.
```js
const pullStreamQueryBuilder = (headers) => {
const query = `subscription onStream($headers: Headers) {
streamHero(headers: $headers) {
documents {
id,
name,
age,
updatedAt,
deleted
},
checkpoint {
id
updatedAt
}
}
}`;
return {
query,
variables: {
headers
}
};
};
```
With the `pullStreamQueryBuilder` you can then start a realtime replication.
```js
const replicationState = replicateGraphQL(
{
collection: myRxCollection,
// urls to the GraphQL endpoints
url: {
http: 'http://example.com/graphql',
ws: 'ws://example.com/subscriptions' // <- The websocket has to use a different url.
},
push: {
batchSize: 100,
queryBuilder: pushQueryBuilder
},
headers: {
Authorization: 'Bearer abcde...'
},
pull: {
batchSize: 100,
queryBuilder: pullQueryBuilder,
streamQueryBuilder: pullStreamQueryBuilder,
includeWsHeaders: false, // Includes headers as connection parameter to Websocket.
// Websocket options that can be passed as a parameter to initialize the subscription
// Can be applied anything from the graphql-ws ClientOptions - https://the-guild.dev/graphql/ws/docs/interfaces/client.ClientOptions
// Except these parameters: 'url', 'shouldRetry', 'webSocketImpl' - locked for internal usage
// Note: if you provide connectionParams as a wsOption, make sure it returns any necessary headers (e.g. authorization)
// because providing your own connectionParams prevents headers from being included automatically
wsOptions: {
retryAttempts: 10,
}
},
deletedField: 'deleted'
}
);
```
:::note
If it is not possible to create a websocket server on your backend, you can use any other method to pull out the ongoing events from the backend and then you can send them into `RxReplicationState.emitEvent()`.
:::
### Transforming null to undefined in optional fields
GraphQL fills up non-existent optional values with `null` while RxDB required them to be `undefined`.
Therefore, if your schema contains optional properties, you have to transform the pulled data to switch out `null` to `undefined`
```js
const replicationState: RxGraphQLReplicationState = replicateGraphQL(
{
collection: myRxCollection,
url: {/* ... */},
headers: {/* ... */},
push: {/* ... */},
pull: {
queryBuilder: pullQueryBuilder,
modifier: (doc => {
// We have to remove optional non-existent field values
// they are set as null by GraphQL but should be undefined
Object.entries(doc).forEach(([k, v]) => {
if (v === null) {
delete doc[k];
}
});
return doc;
})
},
/* ... */
}
);
```
### pull.responseModifier
With the `pull.responseModifier` you can modify the whole response from the GraphQL endpoint **before** it is processed by RxDB.
For example if your endpoint is not capable of returning a valid checkpoint, but instead only returns the plain document array, you can use the `responseModifier` to aggregate the checkpoint from the returned documents.
```ts
import {
} from 'rxdb';
const replicationState: RxGraphQLReplicationState = replicateGraphQL(
{
collection: myRxCollection,
url: {/* ... */},
headers: {/* ... */},
push: {/* ... */},
pull: {
responseModifier: async function(
plainResponse, // the exact response that was returned from the server
origin, // either 'handler' if plainResponse came from the pull.handler, or 'stream' if it came from the pull.stream
requestCheckpoint // if origin==='handler', the requestCheckpoint contains the checkpoint that was sent to the backend
) {
/**
* In this example we aggregate the checkpoint from the documents array
* that was returned from the graphql endpoint.
*/
const docs = plainResponse;
return {
documents: docs,
checkpoint: docs.length === 0 ? requestCheckpoint : {
name: lastOfArray(docs).name,
updatedAt: lastOfArray(docs).updatedAt
}
};
}
},
/* ... */
}
);
```
### push.responseModifier
It's also possible to modify the response of a push mutation. For example if your server returns more than just the conflicting docs:
```graphql
type PushResponse {
conflicts: [Human]
conflictMessages: [ReplicationConflictMessage]
}
type Mutation {
# Returns a PushResponse type that contains the conflicts along with other information
pushHuman(rows: [HumanInputPushRow!]): PushResponse!
}
```
```ts
import {} from "rxdb";
const replicationState: RxGraphQLReplicationState = replicateGraphQL(
{
collection: myRxCollection,
url: {/* ... */},
headers: {/* ... */},
push: {
responseModifier: async function (plainResponse) {
/**
* In this example we aggregate the conflicting documents from a response object
*/
return plainResponse.conflicts;
},
},
pull: {/* ... */},
/* ... */
}
);
```
#### Helper Functions
RxDB provides the helper functions `graphQLSchemaFromRxSchema()`, `pullQueryBuilderFromRxSchema()`, `pullStreamBuilderFromRxSchema()` and `pushQueryBuilderFromRxSchema()` that can be used to generate handlers and schemas from the [RxJsonSchema](./rx-schema.md). To learn how to use them, please inspect the [GraphQL Example](https://github.com/pubkey/rxdb/tree/master/examples/graphql).
### RxGraphQLReplicationState
When you call `myCollection.syncGraphQL()` it returns a `RxGraphQLReplicationState` which can be used to subscribe to events, for debugging or other functions. It extends the [RxReplicationState](./replication.md) with some GraphQL specific methods.
#### .setHeaders()
Changes the headers for the replication after it has been set up.
```js
replicationState.setHeaders({
Authorization: `...`
});
```
#### Sending Cookies
The underlying fetch framework uses a `same-origin` policy for credentials by default. That means, cookies and session data is only shared if you backend and frontend run on the same domain and port. Pass the credential parameter to `include` cookies in requests to servers from different origins via:
```js
replicationState.setCredentials('include');
```
or directly pass it in the `replicateGraphQL` function:
```js
replicateGraphQL(
{
collection: myRxCollection,
/* ... */
credentials: 'include',
/* ... */
}
);
```
See [the fetch spec](https://fetch.spec.whatwg.org/#concept-request-credentials-mode) for more information about available options.
:::note
To play around, check out the full example of the RxDB [GraphQL replication with server and client](https://github.com/pubkey/rxdb/tree/master/examples/graphql)
:::
---
## Websocket Replication
With the websocket replication plugin, you can spawn a websocket server from a RxDB database in Node.js and replicate with it.
:::note
The websocket replication plugin does not have any concept for authentication or permission handling. It is designed to create an easy **server-to-server** replication. It is **not** made for client-server replication. Make a pull request if you need that feature.
:::
## Starting the Websocket Server
```ts
import { createRxDatabase } from 'rxdb';
import {
startWebsocketServer
} from 'rxdb/plugins/replication-websocket';
// create a RxDatabase like normal
const myDatabase = await createRxDatabase({/* ... */});
// start a websocket server
const serverState = await startWebsocketServer({
database: myDatabase,
port: 1337,
path: '/socket'
});
// stop the server
await serverState.close();
```
## Connect to the Websocket Server
The replication has to be started once for each collection that you want to replicate.
```ts
import {
replicateWithWebsocketServer
} from 'rxdb/plugins/replication-websocket';
// start the replication
const replicationState = await replicateWithWebsocketServer({
/**
* To make the replication work,
* the client collection name must be equal
* to the server collection name.
*/
collection: myRxCollection,
url: 'ws://localhost:1337/socket'
});
// stop the replication
await replicationState.cancel();
```
## Customize
We use the [ws](https://www.npmjs.com/package/ws) npm library, so you can use all optional configuration provided by it.
This is especially important to improve performance by opting in of some optional settings.
---
## RxDB's CouchDB Replication Plugin
# Replication with CouchDB
A plugin to replicate between a [RxCollection](./rx-collection.md) and a CouchDB server.
This plugin uses the RxDB [Sync Engine](./replication.md) to replicate with a CouchDB endpoint. This plugin **does NOT** use the official [CouchDB replication protocol](https://docs.couchdb.org/en/stable/replication/protocol.html) because the CouchDB protocol was optimized for server-to-server replication and is not suitable for fast client side applications, mostly because it has to run many HTTP-requests (at least one per document) and also it has to store the whole revision tree of the documents at the client. This makes initial replication and querying very slow.
Because the way RxDB handles revisions and documents is very similar to CouchDB, using the RxDB replication with a CouchDB endpoint is pretty straightforward.
## Pros
- Faster initial replication.
- Works with any [RxStorage](./rx-storage.md), not just [PouchDB](./rx-storage-pouchdb.md).
- Easier conflict handling because conflicts are handled during replication and not afterwards.
- Does not have to store all document revisions on the client, only stores the newest version.
## Cons
- Does not support the replication of [attachments](./rx-attachment.md).
- Like all CouchDB replication plugins, this one is also limited to replicating 6 collections in parallel. [Read this for workarounds](./replication-couchdb.md#limitations)
## Usage
Start the replication via `replicateCouchDB()`.
```ts
import { replicateCouchDB } from 'rxdb/plugins/replication-couchdb';
const replicationState = replicateCouchDB(
{
replicationIdentifier: 'my-couchdb-replication',
collection: myRxCollection,
// url to the CouchDB endpoint (required)
url: 'http://example.com/db/humans',
/**
* true for live replication,
* false for a one-time replication.
* [default=true]
*/
live: true,
/**
* A custom fetch() method can be provided
* to add authentication or credentials.
* Can be swapped out dynamically
* by running 'replicationState.fetch = newFetchMethod;'.
* (optional)
*/
fetch: myCustomFetchMethod,
pull: {
/**
* Amount of documents to be fetched in one HTTP request
* (optional)
*/
batchSize: 60,
/**
* Custom modifier to mutate pulled documents
* before storing them in RxDB.
* (optional)
*/
modifier: docData => {/* ... */},
/**
* Heartbeat time in milliseconds
* for the long polling of the changestream.
* @link https://docs.couchdb.org/en/3.2.2-docs/api/database/changes.html
* (optional, default=60000)
*/
heartbeat: 60000
},
push: {
/**
* How many local changes to process at once.
* (optional)
*/
batchSize: 60,
/**
* Custom modifier to mutate documents
* before sending them to the CouchDB endpoint.
* (optional)
*/
modifier: docData => {/* ... */}
}
}
);
```
When you call `replicateCouchDB()` it returns a `RxCouchDBReplicationState` which can be used to subscribe to events, for debugging or other functions. It extends the [RxReplicationState](./replication.md) so any other method that can be used there can also be used on the CouchDB replication state.
## Conflict handling
When conflicts appear during replication, the `conflictHandler` of the `RxCollection` is used, equal to the other replication plugins. Read more about conflict handling [here](./replication.md#conflict-handling).
## Auth example
Lets say for authentication you need to add a [bearer token](https://swagger.io/docs/specification/authentication/bearer-authentication/) as HTTP header to each request. You can achieve that by crafting a custom `fetch()` method that adds the header field.
```ts
const myCustomFetch = (url, options) => {
// flat clone the given options to not mutate the input
const optionsWithAuth = Object.assign({}, options);
// ensure the headers property exists
if(!optionsWithAuth.headers) {
optionsWithAuth.headers = {};
}
// add bearer token to headers
optionsWithAuth.headers['Authorization'] ='Basic S0VLU0UhIExFQ0...';
// call the original fetch function with our custom options.
return fetch(
url,
optionsWithAuth
);
};
const replicationState = replicateCouchDB(
{
replicationIdentifier: 'my-couchdb-replication',
collection: myRxCollection,
url: 'http://example.com/db/humans',
/**
* Add the custom fetch function here.
*/
fetch: myCustomFetch,
pull: {},
push: {}
}
);
```
Also when your bearer token changes over time, you can set a new custom `fetch` method while the replication is running:
```ts
replicationState.fetch = newCustomFetchMethod;
```
Also there is a helper method `getFetchWithCouchDBAuthorization()` to create a fetch handler with authorization:
```ts
import {
replicateCouchDB,
getFetchWithCouchDBAuthorization
} from 'rxdb/plugins/replication-couchdb';
const replicationState = replicateCouchDB(
{
replicationIdentifier: 'my-couchdb-replication',
collection: myRxCollection,
url: 'http://example.com/db/humans',
/**
* Add the custom fetch function here.
*/
fetch: getFetchWithCouchDBAuthorization('myUsername', 'myPassword'),
pull: {},
push: {}
}
);
```
## Limitations
Since CouchDB only allows synchronization through HTTP/1.1 long polling requests there is a limitation of 6 active synchronization connections before the browser prevents sending any further request. This limitation is at the level of browser per tab per domain (some browser, especially older ones, might have a different limit, [see here](https://docs.pushtechnology.com/cloud/latest/manual/html/designguide/solution/support/connection_limitations.html)).
Since this limitation is at the **browser** level there are several solutions:
- Use only a single database for all entities and set a "type" field for each of the documents
- Create multiple subdomains for CouchDB and use a max of 6 active synchronizations (or less) for each
- Use a proxy (ex: HAProxy) between the browser and CouchDB and configure it to use HTTP/2.0, since HTTP/2.0 multiplexes requests.
If you use nginx in front of your CouchDB, you can use these settings to enable http2-proxying to prevent the connection limit problem:
```
server {
http2 on;
location /db {
rewrite /db/(.*) /$1 break;
proxy_pass http://172.0.0.1:5984;
proxy_redirect off;
proxy_buffering off;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded;
proxy_set_header Connection "keep_alive";
}
}
```
## Known problems
### Database missing
In contrast to PouchDB, this plugin **does NOT** automatically create missing CouchDB databases.
If your CouchDB server does not have a database yet, you have to create it by yourself by running a `PUT` request to the database `name` url:
```ts
// create a 'humans' CouchDB database on the server
const remoteDatabaseName = 'humans';
await fetch(
'http://example.com/db/' + remoteDatabaseName,
{
method: 'PUT'
}
);
```
## React Native
React Native does not have a global `fetch` method. You have to import fetch method with the [cross-fetch](https://www.npmjs.com/package/cross-fetch) package:
```ts
import crossFetch from 'cross-fetch';
const replicationState = replicateCouchDB(
{
replicationIdentifier: 'my-couchdb-replication',
collection: myRxCollection,
url: 'http://example.com/db/humans',
fetch: crossFetch,
pull: {},
push: {}
}
);
```
---
## WebRTC P2P Replication with RxDB - Sync Browsers and Devices
import {Steps} from '@site/src/components/steps';
# P2P WebRTC Replication with RxDB - Sync Data between Browsers and Devices in JavaScript
WebRTC P2P data connections are revolutionizing real-time web and mobile development by **eliminating central servers** in scenarios where clients can communicate directly. With the **RxDB** [Sync Engine](./replication.md), you can sync your local database state across multiple browsers or devices via **WebRTC P2P (Peer-to-Peer)** connections, ensuring scalable, secure, and **low-latency** data flows without traditional server bottlenecks.
## What is WebRTC?
[WebRTC](https://developer.mozilla.org/en-US/docs/Web/API/WebRTC_API) stands for Web [Real-Time](./articles/realtime-database.md) Communication. It is an open standard that enables browsers and native apps to exchange audio, video, or **arbitrary data** directly between peers, bypassing a central server after the initial connection is established. WebRTC uses NAT traversal techniques like [ICE](https://developer.liveswitch.io/liveswitch-server/guides/what-are-stun-turn-and-ice.html) (Interactive Connectivity Establishment) to punch through firewalls and establish direct links. This peer-to-peer nature drastically reduces latency while maintaining **high security** and **end-to-end encryption** capabilities.
For a deeper look at comparing WebRTC with **WebSockets** and **WebTransport**, you can read our [comprehensive overview](./articles/websockets-sse-polling-webrtc-webtransport.md). While WebSockets or WebTransport often work in client-server contexts, WebRTC offers direct peer-to-peer connections ideal for fully decentralized data flows.
## Benefits of P2P Sync with WebRTC Compared to Client-Server Architecture
1. **Reduced Latency** - By skipping a central server hop, data travels directly from one client to another, minimizing round-trip times and improving responsiveness.
2. **Scalability** - New peers can join without overloading a central infrastructure. The sync overhead increases linearly with the number of connections rather than requiring a massive server cluster.
3. **Privacy & Ownership** - Data stays within the userβs devices, avoiding risks tied to storing data on third-party servers. This design aligns well with [local-first](./articles/local-first-future.md) or "[zero-latency](./articles/zero-latency-local-first.md)" apps.
4. **Resilience** - In some scenarios, if the central server is unreachable, P2P connections remain operational (assuming a functioning signaling path). Apps can still replicate data among local networks like when they are in the same Wifi or LAN.
5. **Cost Savings** - Reducing the reliance on a high-bandwidth server can cut hosting and bandwidth expenses, particularly in high-traffic or IoT-style use cases.
## Peer-to-Peer (P2P) WebRTC Replication with the RxDB JavaScript Database
Traditionally, real-time data synchronization depends on **centralized servers** to manage and distribute updates. In contrast, RxDBβs WebRTC P2P replication allows data to flow **directly** among clients, removing the server as a data store. This approach is **live** and **fully decentralized**, requiring only a [signaling server](#signaling-server) for initial discovery:
- **No master-slave** concept - each peer hosts its own local RxDB.
- Clients ([browsers](./articles/browser-database.md), devices) connect to each other via WebRTC data channels.
- The [RxDB replication protocol](./replication.md) then handles pushing/pulling document changes across peers.
Because RxDB is a NoSQL database and the replication protocol is straightforward, setting up robust P2P sync is far **easier** than orchestrating a complex client-server database architecture.
## Using RxDB with the WebRTC Replication Plugin
Before you use this plugin, make sure that you understand how [WebRTC works](https://developer.mozilla.org/en-US/docs/Web/API/WebRTC_API). Here we build a todo-app that replicates todo-entries between clients:
You can find a fully build example of this at the [RxDB Quickstart Repository](https://github.com/pubkey/rxdb-quickstart) which you can also [try out online](https://pubkey.github.io/rxdb-quickstart/).
First you create the [database](./rx-database.md) and then you can configure the replication:
### Create the Database and Collection
Here we create a database with the [localstorage](./rx-storage-localstorage.md) based storage that stores data inside of the [LocalStorage API](./articles/localstorage.md) in a browser. RxDB has a wide [range of storages](./rx-storage.md) for other JavaScript runtimes.
```ts
import { createRxDatabase } from 'rxdb/plugins/core';
import { getRxStorageLocalstorage } from 'rxdb/plugins/storage-localstorage';
const db = await createRxDatabase({
name: 'myTodoDB',
storage: getRxStorageLocalstorage()
});
await db.addCollections({
todos: {
schema: {
title: 'todo schema',
version: 0,
type: 'object',
primaryKey: 'id',
properties: {
id: { type: 'string', maxLength: 100 },
title: { type: 'string' },
done: { type: 'boolean', default: false },
created: { type: 'string', format: 'date-time' }
},
required: ['id', 'title', 'done']
}
}
});
// insert an example document
await db.todos.insert({
id: 'todo-1',
title: 'P2P demo task',
done: false,
created: new Date().toISOString()
});
```
### Import the WebRTC replication plugin
```ts
import {
replicateWebRTC,
getConnectionHandlerSimplePeer
} from 'rxdb/plugins/replication-webrtc';
```
### Start the P2P replication
To start the replication you have to call `replicateWebRTC` on the [collection](./rx-collection.md).
As options you have to provide a `topic` and a connection handler function that implements the `P2PConnectionHandlerCreator` interface. As default you should start with the `getConnectionHandlerSimplePeer` method which uses the [simple-peer](https://github.com/feross/simple-peer) library and comes shipped with RxDB.
```ts
const replicationPool = await replicateWebRTC(
{
// Start the replication for a single collection
collection: db.todos,
// The topic is like a 'room-name'. All clients with the same topic
// will replicate with each other. In most cases you want to use
// a different topic string per user. Also you should prefix the topic with
// a unique identifier for your app, to ensure you do not let your users connect
// with other apps that also use the RxDB P2P Replication.
topic: 'my-users-pool',
/**
* You need a collection handler to be able to create WebRTC connections.
* Here we use the simple peer handler which uses the 'simple-peer' npm library.
* To learn how to create a custom connection handler, read the source code,
* it is pretty simple.
*/
connectionHandlerCreator: getConnectionHandlerSimplePeer({
// Set the signaling server url.
// You can use the server provided by RxDB for tryouts,
// but in production you should use your own server instead.
signalingServerUrl: 'wss://signaling.rxdb.info/',
// only in Node.js, we need the wrtc library
// because Node.js does not contain the WebRTC API.
wrtc: require('node-datachannel/polyfill'),
// only in Node.js, we need the WebSocket library
// because Node.js does not contain the WebSocket API.
webSocketConstructor: require('ws').WebSocket
}),
pull: {},
push: {}
}
);
```
Notice that in difference to the other [replication plugins](./replication.md), the WebRTC replication returns a `replicationPool` instead of a single `RxReplicationState`. The `replicationPool` contains all replication states of the connected peers in the P2P network.
### Observe Errors
To ensure we log out potential errors, observe the `error$` observable of the pool.
```ts
replicationPool.error$.subscribe(err => console.error('WebRTC Error:', err));
```
### Stop the Replication
You can also dynamically stop the replication.
```ts
replicationPool.cancel();
```
## Live replications
The WebRTC replication is **always live** because there can not be a one-time sync when it is always possible to have new Peers that join the connection pool. Therefore you cannot set the `live: false` option like in the other replication plugins.
## Signaling Server
For P2P replication to work with the RxDB WebRTC Replication Plugin, a [signaling server](https://developer.mozilla.org/en-US/docs/Web/API/WebRTC_API/Signaling_and_video_calling) is required. The signaling server helps peers discover each other and establish connections.
RxDB ships with a default signaling server that can be used with the simple-peer connection handler. This server is made for demonstration purposes and tryouts. It is not reliable and might be offline at any time.
In production you must always use your own signaling server instead!
Creating a basic signaling server is straightforward. The provided example uses 'socket.io' for WebSocket communication. However, in production, you'd want to create a more robust signaling server with authentication and additional logic to suit your application's needs.
Here is a quick example implementation of a signaling server that can be used with the connection handler from `getConnectionHandlerSimplePeer()`:
```ts
import {
startSignalingServerSimplePeer
} from 'rxdb/plugins/replication-webrtc';
const serverState = await startSignalingServerSimplePeer({
port: 8080 // <- port
});
```
For custom signaling servers with more complex logic, you can check the [source code of the default one](https://github.com/pubkey/rxdb/blob/master/src/plugins/replication-webrtc/signaling-server.ts).
## Peer Validation
By default the replication will replicate with every peer the signaling server tells them about.
You can prevent invalid peers from replication by passing a custom `isPeerValid()` function that either returns `true` on valid peers and `false` on invalid peers.
```ts
const replicationPool = await replicateWebRTC(
{
/* ... */
isPeerValid: async (peer) => {
return true;
}
pull: {},
push: {}
/* ... */
}
);
```
## Conflict detection in WebRTC replication
RxDB's conflict handling works by detecting and resolving conflicts that may arise when multiple clients in a decentralized database system attempt to modify the same data concurrently.
A **custom conflict handler** can be set up, which is a plain JavaScript function. The conflict handler is run on each replicated document write and resolves the conflict if required. [Find out more about RxDB conflict handling here](https://rxdb.info/transactions-conflicts-revisions.html)
## Known problems
### SimplePeer requires to have `process.nextTick()`
In the browser you might not have a process variable or process.nextTick() method. But the [simple peer](https://github.com/feross/simple-peer) uses that so you have to polyfill it.
In webpack you can use the `process/browser` package to polyfill it:
```js
const plugins = [
/* ... */
new webpack.ProvidePlugin({
process: 'process/browser',
})
/* ... */
];
```
In angular or other libraries you can add the polyfill manually:
```js
window.process = {
nextTick: (fn, ...args) => setTimeout(() => fn(...args)),
};
```
### Polyfill the WebSocket and WebRTC API in Node.js
While all modern browsers support the WebRTC and WebSocket APIs, they is missing in Node.js which will throw the error `No WebRTC support: Specify opts.wrtc option in this environment`. Therefore you have to polyfill it with a compatible WebRTC and WebSocket polyfill. It is recommended to use the [node-datachannel package](https://github.com/murat-dogan/node-datachannel/tree/master/src/polyfill) for WebRTC which **does not** come with RxDB but has to be installed before via `npm install node-datachannel --save`.
For the Websocket API use the `ws` package that is included into RxDB.
```ts
import nodeDatachannelPolyfill from 'node-datachannel/polyfill';
import { WebSocket } from 'ws';
const replicationPool = await replicateWebRTC(
{
/* ... */
connectionHandlerCreator: getConnectionHandlerSimplePeer({
signalingServerUrl: 'wss://example.com:8080',
wrtc: nodeDatachannelPolyfill,
webSocketConstructor: WebSocket
}),
pull: {},
push: {}
/* ... */
}
);
```
## Storing replicated data encrypted on client device
Storing replicated data encrypted on client devices using the RxDB Encryption Plugin is a pivotal step towards bolstering **data security** and **user privacy**.
The WebRTC replication plugin seamlessly integrates with the [RxDB encryption plugins](./encryption.md), providing a robust solution for encrypting sensitive information before it's stored locally. By doing so, it ensures that even if unauthorized access to the device occurs, the data remains protected and unintelligible without the encryption key (or password). This approach is particularly vital in scenarios where user-generated content or confidential data is replicated across devices, as it empowers users with control over their own data while adhering to stringent security standards. [Read more about the encryption plugins here](./encryption.md).
## FAQ
Which distributed database services offer peer discovery and sync plugins?
RxDB offers comprehensive peer discovery and sync plugins for distributed applications. The WebRTC replication plugin facilitates direct peer-to-peer data synchronization. A signaling server handles initial peer discovery and connection establishment. You connect browsers and mobile apps without a central database server. The sync engine automatically replicates local changes across all discovered peers.
## Follow Up
- **Check out the [RxDB Quickstart](./quickstart.md)** to see how to set up your first RxDB database.
- **Explore advanced features** like [Custom Conflict Handling](./transactions-conflicts-revisions.md) or [Offline-First Performance](./rx-storage-performance.md).
- **Try an example** at [RxDB Quickstart GitHub](https://github.com/pubkey/rxdb-quickstart) to see a working P2P Sync setup.
- **Join the RxDB Community** on [GitHub](/code/) or [Discord](/chat/) if you have questions or want to share your P2P WebRTC experiences.
---
## Smooth Firestore Sync for Offline Apps
import {Steps} from '@site/src/components/steps';
# Replication with Firestore from Firebase
With the `replication-firestore` plugin you can do a two-way realtime replication
between your client side [RxDB](./) Database and a [Cloud Firestore](https://firebase.google.com/docs/firestore) database that is hosted on the Firebase platform. It will use the [RxDB Sync Engine](./replication.md) to manage the replication streams, error- and conflict handling.
Replicating your Firestore state to RxDB can bring multiple benefits compared to using the Firestore directly:
- It can reduce your cloud fees because your queries run against the local state of the documents without touching a server and writes can be batched up locally and send to the backend in bulks. This is mostly the case for read heavy applications.
- You can run complex [NoSQL queries](./why-nosql.md) on your documents because you are not bound to the [Firestore Query](https://firebase.google.com/docs/firestore/query-data/queries) handling. You can also use local indexes, [compression](./key-compression.md) and [encryption](./encryption.md) and do things like fulltext search, fully locally.
- Your application can be truly [Offline-First](./offline-first.md) because your data is stored in a client side database. In contrast Firestore by itself only provides options to support [offline also](https://cloud.google.com/firestore/docs/manage-data/enable-offline) which more works like a cache and requires the user to be online at application start to run authentication.
- It reduces the vendor lock in because you can switch out the backend server afterwards without having to rebuild big parts of the application. RxDB supports replication plugins with multiple technologies and it is even easy to set up with your [custom backend](./replication.md).
- You can use sophisticated [conflict resolution strategies](./replication.md#conflict-handling) so you are not bound to the Firestore [last-write-wins](https://stackoverflow.com/a/47781502/3443137) strategy which is not suitable for many applications.
- The initial load time of your application can be decreased because it will do an incremental replication on restarts.
## Usage
### Install the firebase package
```bash
npm install firebase
```
### Initialize your Firestore Database
```ts
import * as firebase from 'firebase/app';
import {
getFirestore,
collection
} from 'firebase/firestore';
const projectId = 'my-project-id';
const app = firebase.initializeApp({
projectId,
databaseURL: 'http://localhost:8080?ns=' + projectId,
/* ... */
});
const firestoreDatabase = getFirestore(app);
const firestoreCollection = collection(firestoreDatabase, 'my-collection-name');
```
### Start the Replication
Start the replication by calling `replicateFirestore()` on your [RxCollection](./rx-collection.md).
```ts
const replicationState = replicateFirestore({
replicationIdentifier: `https://firestore.googleapis.com/${projectId}`,
collection: myRxCollection,
firestore: {
projectId,
database: firestoreDatabase,
collection: firestoreCollection
},
/**
* (required) Enable push and pull replication with firestore by
* providing an object with optional filter
* for each type of replication desired.
* [default=disabled]
*/
pull: {},
push: {},
/**
* Either do a live or a one-time replication
* [default=true]
*/
live: true,
/**
* (optional) likely you should just use the default.
*
* In firestore it is not possible to read out
* the internally used write timestamp of a document.
* Even if we could read it out, it is not indexed which
* is required for fetch 'changes-since-x'.
* So instead we have to rely on a custom user defined field
* that contains the server time
* which is set by firestore via serverTimestamp()
* Notice that the serverTimestampField MUST NOT be
* part of the collections RxJsonSchema!
* [default='serverTimestamp']
*/
serverTimestampField: 'serverTimestamp'
});
```
To observe and cancel the replication, you can use any other methods from the [ReplicationState](./replication.md) like `error$`, `cancel()` and `awaitInitialReplication()`.
## Handling deletes
RxDB requires you to never [fully delete documents](./replication.md#data-layout-on-the-server). This is needed to be able to replicate the deletion state of a document to other instances. The firestore replication will set a boolean `_deleted` field to all documents to indicate the deletion state. You can change this by setting a different `deletedField` in the sync options.
## Do not set `enableIndexedDbPersistence()`
Firestore has the `enableIndexedDbPersistence()` feature which caches document states locally to [IndexedDB](./rx-storage-indexeddb.md). This is not needed when you replicate your Firestore with RxDB because RxDB itself will store the data locally already.
## Using the replication with an already existing Firestore Database State
If you have not used RxDB before and you already have documents inside of your Firestore database, you have
to manually set the `_deleted` field to `false` and the `serverTimestamp` to all existing documents.
```ts
import {
getDocs,
query,
where,
serverTimestamp
} from 'firebase/firestore';
const allDocsResult = await getDocs(query(firestoreCollection));
allDocsResult.forEach(doc => {
doc.update({
_deleted: false,
serverTimestamp: serverTimestamp()
})
});
```
Also notice that if you do writes from non-RxDB applications, you have to keep these fields in sync. It is recommended to use the [Firestore triggers](https://firebase.google.com/docs/functions/firestore-events) to ensure that.
## Filtered Replication
You might need to replicate only a subset of your collection, either to or from Firestore. You can achieve this using `push.filter` and `pull.filter` options.
```ts
const replicationState = replicateFirestore(
{
collection: myRxCollection,
firestore: {
projectId,
database: firestoreDatabase,
collection: firestoreCollection
},
pull: {
filter: [
where('ownerId', '==', userId)
]
},
push: {
filter: (item) => item.syncEnabled === true
}
}
);
```
Keep in mind that you can not use inequality operators `(<, <=, !=, not-in, >, or >=)` in `pull.filter` since that would cause a conflict with ordering by `serverTimestamp`.
---
## MongoDB Realtime Sync Engine for Local-First Apps
import {Tabs} from '@site/src/components/tabs';
import {Steps} from '@site/src/components/steps';
import {VideoBox} from '@site/src/components/video-box';
import {RxdbMongoDiagramPlain} from '@site/src/components/mongodb-sync';
# MongoDB Replication Plugin for RxDB - Real-Time, Offline-First Sync
The [MongoDB](https://www.mongodb.com/) Replication Plugin for RxDB delivers seamless, two-way synchronization between [MongoDB](./rx-storage-mongodb.md) and RxDB, enabling [real-time](./articles/realtime-database.md) updates and [offline-first](./offline-first.md) functionality for your applications. Built on **MongoDB Change Streams**, it supports both Atlas and self-hosted deployments, ensuring your data stays consistent across every device and service.
Behind the scenes, the plugin is powered by the RxDB [Sync Engine](./replication.md), which manages the complexities of real-world data replication for you. It automatically handles [conflict detection and resolution](./transactions-conflicts-revisions.md), maintains precise checkpoints for incremental updates, and gracefully manages transitions between offline and online states. This means you don't need to manually implement retry logic, reconcile divergent changes, or worry about data loss during connectivity drops, the Sync Engine ensures consistency and reliability in every sync cycle.
## Key Features
- **Two-way replication** between MongoDB and RxDB collections
- **Offline-first support** with automatic incremental re-sync
- **Incremental updates** via MongoDB Change Streams
- **Conflict resolution** handled by the RxDB Sync Engine
- **Atlas and self-hosted support** for replica sets and sharded clusters
## Architecture Overview
The plugin operates in a three-tier architecture: Clients connect to [RxServer](./rx-server.md), which in turn connects to MongoDB. RxServer streams changes from MongoDB to connected clients and pushes client-side updates back to MongoDB.
For the client side, RxServer exposes a [replication endpoint](./rx-server.md#replication-endpoint) over WebSocket or HTTP, which your RxDB-powered applications can consume.
The following diagram illustrates the flow of updates between clients, RxServer, and MongoDB in a live synchronization setup:
:::note
The MongoDB Replication Plugin is optimized for Node.js environments (e.g., when RxDB runs within RxServer or other backend services). Direct connections from browsers or mobile apps to MongoDB are not supported because MongoDB does not use HTTP as its wire protocol and requires a driver-level connection to a replica set or sharded cluster.
:::
## Setting up the Client-RxServer-MongoDB Sync
### Install the Client Dependencies
In your JavaScript project, install the RxDB libraries and the MongoDB node.js driver:
```npm install rxdb rxdb-server mongodb --save```
### Set up a MongoDB Server
As first step, you need access to a running MongoDB Server. This can be done by either running a server locally or using the Atlas Cloud. Notice that we need to have a [replica set](https://www.mongodb.com/docs/manual/tutorial/deploy-replica-set/) because only on these, the MongoDB changestream can be used.
### Shell
If you have installed MongoDB locally, you can start the server with this command:
```mongod --replSet rs0 --bind_ip_all```
### Docker
If you have docker installed, you can start a container that runs the MongoDB server:
```docker run -p 27017:27017 -p 27018:27018 -p 27019:27019 --rm --name rxdb-mongodb mongo:8.0.4 mongod --replSet rs0 --bind_ip_all```
### MongoDB Atlas
Learn here how to create a MongoDB atlas account and how to start a MongoDB cluster that runs in the cloud:
After this step you should have a valid connection string that points to a running MongoDB Server like `mongodb://localhost:27017/`.
### Create a MongoDB Database and Collection
On your MongoDB server, make sure to create a database and a collection.
```ts
//> server.ts
import { MongoClient } from 'mongodb';
const mongoClient = new MongoClient('mongodb://localhost:27017/?directConnection=true');
const mongoDatabase = mongoClient.db('my-database');
await mongoDatabase.createCollection('my-collection', {
changeStreamPreAndPostImages: { enabled: true }
});
```
:::note
To observe document deletions on the changestream, `changeStreamPreAndPostImages` must be enabled. This is not required if you have an insert/update-only collection where no documents are deleted ever.
:::
### Create a RxDB Database and Collection
Now we create an RxDB [database](./rx-database.md) and a [collection](./rx-collection.md). In this example the [memory storage](./rx-storage-memory.md), in production you would use a [persistent storage](./rx-storage.md) instead.
```ts
//> server.ts
import { createRxDatabase, addRxPlugin } from 'rxdb';
import { getRxStorageMemory } from 'rxdb/plugins/storage-memory';
// Create server-side RxDB instance
const db = await createRxDatabase({
name: 'serverdb',
storage: getRxStorageMemory()
});
// Add your collection schema
await db.addCollections({
humans: {
schema: {
version: 0,
primaryKey: 'passportId',
type: 'object',
properties: {
passportId: { type: 'string', maxLength: 100 },
firstName: { type: 'string' },
lastName: { type: 'string' }
},
required: ['passportId', 'firstName', 'lastName']
}
}
});
```
### Sync the Collection with the MongoDB Server
Now we can start a [replication](./replication.md) that does a two-way replication between the RxDB Collection and the MongoDB Collection.
```ts
//> server.ts
import { replicateMongoDB } from 'rxdb/plugins/replication-mongodb';
const replicationState = replicateMongoDB({
mongodb: {
collectionName: 'my-collection',
connection: 'mongodb://localhost:27017',
databaseName: 'my-database'
},
collection: db.humans,
replicationIdentifier: 'humans-mongodb-sync',
pull: { batchSize: 50 },
push: { batchSize: 50 },
live: true
});
```
:::note You can do many things with the replication state
The `RxMongoDBReplicationState` which is returned from `replicateMongoDB()` allows you to run all functionality of the normal [RxReplicationState](./replication.md) like observing errors or doing start/stop operations.
:::
### Start a RxServer
Now that we have a RxDatabase and Collection that is replicated with MongoDB, we can spawn a [RxServer](./rx-server.md) on top of it. This server can then be used by client devices to connect.
```ts
//> server.ts
import { createRxServer } from 'rxdb-server/plugins/server';
import { RxServerAdapterExpress } from 'rxdb-server/plugins/adapter-express';
const server = await createRxServer({
database: db,
adapter: RxServerAdapterExpress,
port: 8080,
cors: '*'
});
const endpoint = server.addReplicationEndpoint({
name: 'humans',
collection: db.humans
});
console.log('Replication endpoint:', `http://localhost:8080/${endpoint.urlPath}`);
// do not forget to start the server!
await server.start();
```
### Sync a Client with the RxServer
On the client-side we create the exact same RxDatabase and collection and then replicate it with the replication endpoint of the RxServer.
```ts
//> client.ts
import { createRxDatabase } from 'rxdb';
import { getRxStorageDexie } from 'rxdb/plugins/storage-dexie';
import { replicateServer } from 'rxdb-server/plugins/replication-server';
const db = await createRxDatabase({
name: 'mydb-client',
storage: getRxStorageDexie()
});
await db.addCollections({
humans: {
schema: {
version: 0,
primaryKey: 'passportId',
type: 'object',
properties: {
passportId: { type: 'string', maxLength: 100 },
firstName: { type: 'string' },
lastName: { type: 'string' }
},
required: ['passportId', 'firstName', 'lastName']
}
}
});
// Start replication to the RxServer endpoint printed by the server:
// e.g. http://localhost:8080/humans/0
const replicationState = replicateServer({
replicationIdentifier: 'humans-rxserver',
collection: db.humans,
url: 'http://localhost:8080/humans/0',
live: true,
pull: { batchSize: 50 },
push: { batchSize: 50 }
});
```
## Follow Up
- Try it out with the [RxDB-MongoDB example repository](https://github.com/pubkey/rxdb-mongodb-sync-example)
- Read [From Local to Global: Scalable Edge Apps with RxDB + MongoDB](https://www.mongodb.com/company/blog/innovation/from-local-global-scalable-edge-apps-rxdb)
- [Replication API Reference](./replication.md)
- [RxServer Documentation](./rx-server.md)
- Join our [Discord Forum](./chat) for questions and feedback
---
## Supabase Replication Plugin for RxDB - Real-Time, Offline-First Sync
import {Tabs} from '@site/src/components/tabs';
import {Steps} from '@site/src/components/steps';
import {VideoBox} from '@site/src/components/video-box';
import {RxdbMongoDiagramPlain} from '@site/src/components/mongodb-sync';
# Supabase Replication Plugin for RxDB - Real-Time, Offline-First Sync
The **Supabase Replication Plugin** for RxDB delivers seamless, two-way synchronization between your RxDB collections and a Supabase (Postgres) table. It uses **PostgREST** for pull/push and **Supabase Realtime** (logical replication) to stream live updates, so your data stays consistent across devices with first-class [local-first](./articles/local-first-future.md), offline-ready support.
Under the hood, the plugin is powered by the RxDB [Sync Engine](./replication.md). It handles checkpointed incremental pulls, robust retry logic, and [conflict detection/resolution](./transactions-conflicts-revisions.md) for you. You focus on featuresβRxDB takes care of sync.
## Key Features of the RxDB-Supabase Plugin
- **Cloud Only Backend**: No self-hosted server required. Client devices directly sync with the Supabase Servers.
- **Two-way replication** between Supabase tables and RxDB [collections](./rx-collection.md)
- **Offline-first** with resumable, incremental sync
- **Live updates** via Supabase Realtime channels
- **Conflict resolution** handled by the [RxDB Sync Engine](./replication.md)
- **Works in browsers and Node.js** with `@supabase/supabase-js`
## Architecture Overview
Clients connect **directly to Supabase** using the official JS client. The plugin:
- **Pulls** documents over PostgREST using a checkpoint `(modified, id)` and deterministic ordering.
- **Pushes** inserts/updates using optimistic concurrency guards.
- **Streams** new changes using Supabase Realtime so live replication stays up to date.
:::note
Because Supabase exposes Postgres over **HTTP/WebSocket**, you can safely replicate from browsers and mobile apps. Protect your data with **Row Level Security (RLS)** policies; use the **anon** key on clients and the **service role** key only on trusted servers.
:::
## Setting up RxDB β Supabase Sync
### Install Dependencies
```bash
npm install rxdb @supabase/supabase-js
```
### Create a Supabase Project & Table
In your supabase project, create a new table. Ensure that:
- The primary key must have the type text (Primary keys must always be strings in RxDB)
- You have an modified field which stores the last modification timestamp of a row (default is `_modified`)
- You have a boolean field which stores if a row should is "deleted". You should not hard-delete rows in Supabase, because clients would miss the deletion if they haven't been online at the deletion time. Instead, use a deleted `boolean` to mark rows as deleted. This way all clients can still pull the deletion, and RxDB will hide the complexity on the client side.
- Enable the realtime observation of writes to the table.
Here is an example for a "human" table:
```sql
create extension if not exists moddatetime schema extensions;
create table "public"."humans" (
"passportId" text primary key,
"firstName" text not null,
"lastName" text not null,
"age" integer,
"_deleted" boolean DEFAULT false NOT NULL,
"_modified" timestamp with time zone DEFAULT now() NOT NULL
);
-- auto-update the _modified timestamp
CREATE TRIGGER update_modified_datetime BEFORE UPDATE ON public.humans FOR EACH ROW
EXECUTE FUNCTION extensions.moddatetime('_modified');
-- add a table to the publication so we can subscribe to changes
alter publication supabase_realtime add table "public"."humans";
```
### Create an RxDB Database & Collection
Create a normal RxDB database, then add a collection whose **schema mirrors your Supabase table**. The **primary key must match** (same column name and type), and fields should be **top-level simple types** (string/number/boolean). You donβt need to model server internals: the plugin maps the serverβs \_deleted flag to doc.\_deleted automatically, and \_modified is optional in your schema (the plugin strips it on push and will include it on pull only if you define it). For browsers use a persistent storage like Localstorage or IndexedDB. For tests you can use the [in-memory storage](./rx-storage-memory.md).
```ts
// client
import { createRxDatabase } from 'rxdb/plugins/core';
import { getRxStorageLocalstorage } from 'rxdb/plugins/storage-localstorage';
export const db = await createRxDatabase({
name: 'mydb',
storage: getRxStorageLocalstorage()
});
await db.addCollections({
humans: {
schema: {
version: 0,
primaryKey: 'passportId',
type: 'object',
properties: {
passportId: { type: 'string', maxLength: 100 },
firstName: { type: 'string' },
lastName: { type: 'string' },
age: { type: 'number' }
},
required: ['passportId', 'firstName', 'lastName']
}
}
});
```
### Create the Supabase Client
Make a single Supabase client and reuse it across your app. In the browser, use the anon key (RLS-protected). On trusted servers you may use the service role keyβbut never ship that to clients.
#### Production
```ts
//> client
import { createClient } from '@supabase/supabase-js';
export const supabase = createClient(
'https://xyzcompany.supabase.co',
'eyJhbGciOi...'
);
```
#### Vite
```ts
//> client
import { createClient } from '@supabase/supabase-js';
export const supabase = createClient(
import.meta.env.VITE_SUPABASE_URL!, // e.g. https://xyzcompany.supabase.co
import.meta.env.VITE_SUPABASE_ANON_KEY! // anon key for browsers
// optional options object here
);
```
#### Local Development
```ts
//> client
import { createClient } from '@supabase/supabase-js';
export const supabase = createClient(
'http://127.0.0.1:54321',
'eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9...'
);
```
### Start Replication
Connect your RxDB collection to the Supabase table to start the replication.
```ts
//> client
import { replicateSupabase } from 'rxdb/plugins/replication-supabase';
const replication = replicateSupabase({
tableName: 'humans',
client: supabase,
collection: db.humans,
replicationIdentifier: 'humans-supabase',
live: true,
pull: {
batchSize: 50,
// optional: shape incoming docs
modifier: (doc) => {
// map nullable age-field
if (!doc.age) delete doc.age;
return doc;
}
// optional: customize the pull query before fetching
queryBuilder: ({ query }) => {
// Add filters, joins, or other PostgREST query modifiers
// This runs before checkpoint filtering and ordering
return query.eq("status", "active");
},
},
push: {
batchSize: 50
},
// optional overrides if your column names differ:
// modifiedField: '_modified',
// deletedField: '_deleted'
});
// (optional) observe errors and wait for the first sync barrier
replication.error$.subscribe(err => console.error('[replication]', err));
await replication.awaitInitialReplication();
```
:::note Nullable values must be mapped
Supabase returns `null` for nullable columns, but in RxDB you often model those fields as optional (i.e., they can be undefined/missing). To avoid schema errors, map `null` β `undefined` in the `pull.modifier` (usually by deleting the key).
:::
## Using Joins
You can use the `pull.queryBuilder` to use joins and also pull data from related tables.
To do that, you have to create a **new** query object in the `pull.queryBuilder` with the `.select()` method and return it.
```ts
const replication = replicateSupabase({
pull: {
queryBuilder: (/* ignore the passed query instance from here */) => {
/**
* Create a totally new query instance
* and return that.
*/
return supabase.from('humans').select('*, pets(*), toys(*)');
}
}
});
```
### Do other things with the replication state
The `RxSupabaseReplicationState` which is returned from `replicateSupabase()` allows you to run all functionality of the normal [RxReplicationState](./replication.md).
## FAQ
Why use Supabase and RxDB as a cloud document database for Node.js and TypeScript?
Supabase and RxDB offer the best of both worlds for Node.js and TypeScript applications. Supabase provides a powerful PostgreSQL backend. You can use PostgreSQL as a document database by storing JSON data. RxDB provides a reactive local document store. You achieve real-time synchronization between the local document store and the Supabase backend. You benefit from strong TypeScript typing on the client and robust SQL querying on the server.
## Follow Up
- **Replication API Reference:** Learn the core concepts and lifecycle hooks β [Replication](./replication.md)
- **Offline-First Guide:** Caching, retries, and conflict strategies β [Local-First](./articles/local-first-future.md)
- **Supabase Essentials:**
- Row Level Security (RLS) β https://supabase.com/docs/guides/auth/row-level-security
- Realtime β https://supabase.com/docs/guides/realtime
- Local dev with the Supabase CLI β https://supabase.com/docs/guides/cli
- **Community:** Questions or feedback? Join our Discord β [Chat](./chat)
---
## Google Drive Sync
import {Steps} from '@site/src/components/steps';
# Replication with Google Drive
The `replication-google-drive` plugin allows you to replicate your client-side [RxDB](./) database to a folder in the user's Google Drive. This enables cross-device [sync](./replication.md) for single users without requiring any backend server.
## Overview
The replication uses the Google Drive API v3 and v2.
- **[Offline-First](./offline-first.md):** Users can work offline. Changes are synced when they go online.
- **No Backend Required:** You don't need to host your own database server.
- **Cross-Device:** Users can access their data from multiple devices by signing into the same Google account.
- **Realtime Sync:** Uses [WebRTC](./replication-webrtc.md) for peer-to-peer signaling to achieve near real-time updates. Uses the same google-drive folder instead of a signaling-server.
:::info
This plugin is in **beta** since RxDB version 17.0.0.
:::
## Usage
### Enable Google Drive API
You need to enable the Google Drive API in the [Google Cloud Console](https://console.cloud.google.com/) and create credentials (OAuth 2.0 Client ID) for your application.
### Authenticate the User
Your application must handle the OAuth flow to get an `accessToken` from Google. You can use libraries like [`@react-oauth/google`](https://www.npmjs.com/package/@react-oauth/google) or the Google Identity Services SDK.
### Start Replication
Once you have the `accessToken`, you can start the replication.
```ts
import { replicateGoogleDrive } from 'rxdb/plugins/replication-google-drive';
const replicationState = await replicateGoogleDrive({
replicationIdentifier: 'my-app-drive-sync',
collection: myRxCollection, // RxCollection
googleDrive: {
oauthClientId: 'YOUR_GOOGLE_CLIENT_ID',
authToken: 'USER_ACCESS_TOKEN',
folderPath: 'my-app-data/user-1'
},
live: true,
pull: {
batchSize: 60,
modifier: doc => doc // (optional) modify invalid data
},
push: {
batchSize: 60,
modifier: doc => doc // (optional) modify before sending
}
});
// Observe replication states
replicationState.error$.subscribe(err => {
console.error('Replication error:', err);
});
replicationState.awaitInitialReplication().then(() => {
console.log('Initial replication done');
});
```
## Signaling & WebRTC
Google Drive does not provide real-time events for file changes. If a user changes data on **User Device A**, **User Device B** would not know about it until it periodically polls the Drive API.
To achieve real-time updates, this plugin uses **WebRTC** to signal changes between connected devices.
1. Devices create "signal files" in a `signaling` subfolder on Google Drive.
2. Other devices detect these files, read the WebRTC connection data, and establish a direct P2P connection with each other.
3. When a device makes a write, it sends a "RESYNC" signal via WebRTC to all connected peers to notify them about the change.
### Polyfill for Node.js
WebRTC is native in browsers but requires a polyfill in Node.js.
```ts
import wrtc from 'node-datachannel/polyfill'; // or 'wrtc' package
// ...
const replicationState = await replicateGoogleDrive({
// ...
signalingOptions: {
wrtc // Pass the polyfill here
}
});
```
## Options
### googleDrive
- **oauthClientId** `string`: The OAuth 2.0 Client ID of your application.
- **authToken** `string`: The valid access token associated with the user.
- **folderPath** `string`: The path to the folder in Google Drive where data should be stored.
- The plugin will ensure this folder exists.
- It must **not** be the root folder.
- It creates subfolders `docs` (for data) and `signaling` (for WebRTC).
- **apiEndpoint** `string` (optional): Defaults to `https://www.googleapis.com`. Useful for mocking or proxies.
- **transactionTimeout** `number` (optional): Default `10000` (10s). The plugin uses a `transaction` file in Drive to ensure data integrity during writes. This is the timeout after which a lock is considered stale.
### pull & push
Standard RxDB [Replication Options](./replication.md) for batch size, modifiers, etc.
## Technical Details
### File Mapping
- Each RxDB document corresponds to **one JSON file** in the `docs` subfolder.
- The filename is `[primaryKey].json`.
- This simple mapping makes it easy to inspect or backup data manually.
### Checkpointing
- The replication relies on the `modifiedTime` of files in Google Drive.
### Conflict Resolution
- Conflicts are handled using the standard RxDB [conflict handling](./replication.md#conflict-handling) strategies.
- The plugin assumes a master-slave replication pattern where the client (RxDB) merges changes.
- If the `transaction` file is locked by another device, the write retries until the lock is released or times out.
## Limitations
- **Rate Limits:** Google Drive API has strict rate limits. The plugin attempts to handle 429 errors with exponential backoff, but heavy concurrent writes might hit these limits.
- **Latency:** Changes take time to propagate and appear in listings (eventual consistency), which the plugin handles internally.
- **Signaling Delay:** The initial WebRTC handshake requires writing and reading files from Drive, which can take a few seconds. Once connected, signaling is instant.
## Testing
For testing, it is recommended to use [google-drive-mock](https://github.com/pubkey/google-drive-mock). It simulates the Google Drive API so you can run tests without real credentials.
---
## Microsoft OneDrive Sync
import {Steps} from '@site/src/components/steps';
# Replication with Microsoft OneDrive
The `replication-microsoft-onedrive` plugin allows you to replicate your client-side [RxDB](./) database to a folder in the user's Microsoft OneDrive. This enables cross-device [sync](./replication.md) for single users without requiring any backend server.
## Overview
The replication uses the Microsoft Graph API.
- **[Offline-First](./offline-first.md):** Users can work offline. Changes are synced when they go online.
- **No Backend Required:** You don't need to host your own database server.
- **Cross-Device:** Users can access their data from multiple devices by signing into the same Microsoft account.
- **Realtime Sync:** Uses [WebRTC](./replication-webrtc.md) for peer-to-peer signaling to achieve near real-time updates. Uses the same onedrive folder instead of a signaling-server.
:::info
This plugin is in **beta** since RxDB version 17.0.0.
:::
## Usage
### Enable Microsoft Graph API
You need to register your application in the [Azure portal](https://portal.azure.com/) and create credentials (OAuth 2.0 Client ID) with `Files.ReadWrite` permissions for your application.
### Authenticate the User
Your application must handle the OAuth flow to get an `accessToken` from Microsoft. You can use libraries like `@azure/msal-browser` or `@azure/msal-react`.
### Start Replication
Once you have the `accessToken`, you can start the replication.
```ts
import { replicateMicrosoftOneDrive } from 'rxdb/plugins/replication-microsoft-onedrive';
const replicationState = await replicateMicrosoftOneDrive({
replicationIdentifier: 'my-app-onedrive-sync',
collection: myRxCollection, // RxCollection
oneDrive: {
authToken: 'USER_ACCESS_TOKEN',
folderPath: 'my-app-data/user-1'
},
live: true,
pull: {
batchSize: 60,
modifier: doc => doc // (optional) modify invalid data
},
push: {
batchSize: 60,
modifier: doc => doc // (optional) modify before sending
}
});
// Observe replication states
replicationState.error$.subscribe(err => {
console.error('Replication error:', err);
});
replicationState.awaitInitialReplication().then(() => {
console.log('Initial replication done');
});
```
## Signaling & WebRTC
Microsoft OneDrive does not provide real-time events for file changes that a client can easily subscribe to in the browser. If a user changes data on **User Device A**, **User Device B** would not know about it until it periodically polls the API.
To achieve real-time updates, this plugin uses **WebRTC** to signal changes between connected devices.
1. Devices create "signal files" in a `signaling` subfolder on OneDrive.
2. Other devices detect these files, read the WebRTC connection data, and establish a direct P2P connection with each other.
3. When a device makes a write, it sends a "RESYNC" signal via WebRTC to all connected peers to notify them about the change.
### Polyfill for Node.js
WebRTC is native in browsers but requires a polyfill in Node.js.
```ts
import wrtc from 'node-datachannel/polyfill'; // or 'wrtc' package
// ...
const replicationState = await replicateMicrosoftOneDrive({
// ...
signalingOptions: {
wrtc // Pass the polyfill here
}
});
```
## Options
### oneDrive
- **authToken** `string`: The valid access token associated with the user.
- **folderPath** `string`: The path to the folder in Microsoft OneDrive where data should be stored.
- The plugin will ensure this folder exists.
- It must **not** be the root folder.
- It creates subfolders `docs` (for data) and `signaling` (for WebRTC).
- **apiEndpoint** `string` (optional): Defaults to `https://graph.microsoft.com/v1.0/me/drive`. Useful for mocking or proxies.
- **transactionTimeout** `number` (optional): Default `10000` (10s). The plugin uses a `transaction.json` file in OneDrive to ensure data integrity during writes. This is the timeout after which a lock is considered stale.
### pull & push
Standard RxDB [Replication Options](./replication.md) for batch size, modifiers, etc.
## Technical Details
### File Mapping
- Each RxDB document corresponds to **one JSON file** in the `docs` subfolder.
- The filename is `[primaryKey].json`.
- This simple mapping makes it easy to inspect or backup data manually.
### Checkpointing
- The replication relies on the `lastModifiedDateTime` of files in Microsoft OneDrive.
### Conflict Resolution
- Conflicts are handled using the standard RxDB [conflict handling](./replication.md#conflict-handling) strategies.
- The plugin assumes a master-slave replication pattern where the client (RxDB) merges changes.
- If the `transaction.json` file is locked by another device, the write retries until the lock is released or times out.
## Limitations
- **Rate Limits:** Microsoft Graph API has strict rate limits. The plugin attempts to handle 429 errors with exponential backoff, but heavy concurrent writes might hit these limits.
- **Latency:** Changes take time to propagate and appear in listings (eventual consistency), which the plugin handles internally.
- **Signaling Delay:** The initial WebRTC handshake requires writing and reading files from OneDrive, which can take a few seconds. Once connected, signaling is instant.
## Testing
For testing, it is recommended to use [microsoft-onedrive-mock](https://github.com/pubkey/microsoft-onedrive-mock). It simulates the Microsoft Graph API so you can run tests without real credentials.
---
## RxDB & NATS - Realtime Sync
import {Steps} from '@site/src/components/steps';
# Replication with NATS
With this RxDB plugin you can run a two-way realtime replication with a [NATS](https://nats.io/) server.
The replication itself uses the [RxDB Sync Engine](./replication.md) which handles conflicts, errors and retries.
On the client side the official [NATS npm package](https://www.npmjs.com/package/nats) is used to connect to the NATS server.
NATS is a messaging system that by itself does not have a validation or granulary access control build in.
Therefore it is not recommended to directly replicate the NATS server with an untrusted RxDB client application. Instead you should replicated from NATS to your Node.js server side RxDB database.
## Precondition
For the replication endpoint the NATS cluster must have enabled [JetStream](https://docs.nats.io/nats-concepts/jetstream) and store all message data as [structured JSON](https://docs.nats.io/using-nats/developer/sending/structure).
The easiest way to start a compatible NATS server is to use the official docker image:
```docker run --rm --name rxdb-nats -p 4222:4222 nats:2.9.17 -js```
## Usage
### Install the nats package
```bash
npm install nats --save
```
### Start the Replication
To start the replication, import the `replicateNats()` method from the RxDB plugin and call it with the collection
that must be replicated.
The replication runs *per [RxCollection](./rx-collection.md)*, you can replicate multiple RxCollections by starting a new replication for each of them.
```typescript
import {
replicateNats
} from 'rxdb/plugins/replication-nats';
const replicationState = replicateNats({
collection: myRxCollection,
replicationIdentifier: 'my-nats-replication-collection-A',
// in NATS, each stream need a name
streamName: 'stream-for-replication-A',
/**
* The subject prefix determines how the documents are stored in NATS.
* For example the document with id 'alice'
* will have the subject 'foobar.alice'
*/
subjectPrefix: 'foobar',
connection: { servers: 'localhost:4222' },
live: true,
pull: {
batchSize: 30
},
push: {
batchSize: 30
}
});
```
## Handling deletes
RxDB requires you to never [fully delete documents](./replication.md#data-layout-on-the-server). This is needed to be able to replicate the deletion state of a document to other instances. The NATS replication will set a boolean `_deleted` field to all documents to indicate the deletion state. You can change this by setting a different `deletedField` in the sync options.
---
## Appwrite Realtime Sync for Local-First Apps
import {Tabs} from '@site/src/components/tabs';
import {Steps} from '@site/src/components/steps';
import {VideoBox} from '@site/src/components/video-box';
import {RxdbMongoDiagramPlain} from '@site/src/components/mongodb-sync';
# RxDB Appwrite Replication
This replication plugin allows you to synchronize documents between RxDB and an Appwrite server. It supports both push and pull replication, live updates via Appwrite's real-time subscriptions, [offline-capability](./offline-first.md) and [conflict resolution](./transactions-conflicts-revisions.md).
## Why you should use RxDB with Appwrite?
**Appwrite** is a secure, open-source backend server that simplifies backend tasks like user authentication, storage, database management, and real-time APIs.
**[RxDatabase](./rx-database.md)** is a reactive database for the frontend that offers offline-first capabilities and rich client-side data handling.
Combining the two provides several benefits:
1. [Offline-First](./offline-first.md): RxDB keeps all data locally, so your application remains fully functional even when the network is unavailable. When connectivity returns, the RxDB β Appwrite replication automatically resolves and synchronizes changes.
2. **Real-Time Sync**: With Appwriteβs real-time subscriptions and RxDBβs live replication, you can build collaborative features that update across all clients instantaneously.
3. [Conflict Handling](./transactions-conflicts-revisions.md): RxDB offers flexible conflict resolution strategies, making it simpler to handle concurrent edits across multiple users or devices.
4. **Scalable & Secure**: Appwrite is built to handle production loads with granular access controls, while RxDB easily scales across various storage backends on the client side.
5. **Simplicity & Modularity**: RxDBβs plugin-based architecture, combined with Appwriteβs Cloud offering makes it one of the easiest way to build local-first [realtime apps](./articles/realtime-database.md) that scale.
## Preparing the Appwrite Server
You can either use the appwrite cloud or self-host the Appwrite server. In this tutorial we use the Cloud which is recommended for beginners because it is way easier to set up. You can later decide to self-host if needed.
### Set up an Appwrite Endpoint and Project
#### Self-Hosted
##### Docker
Ensure docker and docker-compose is installed and your version are up to date:
```bash
docker-compose -v
```
##### Run the installation script
The installation script runs inside of a docker container. It will create a docker-compose file and an `.env` file.
```bash
docker run -it --rm \
--volume /var/run/docker.sock:/var/run/docker.sock \
--volume "$(pwd)"/appwrite:/usr/src/code/appwrite:rw \
--entrypoint="install" \
appwrite/appwrite:1.6.1
```
##### Start/Stop
After the installation is done, you can manually stop and start the appwrite instance with docker compose:
```bash
# stop
docker-compose down
# start
docker-compose up
```
#### Appwrite Cloud
##### Create a Cloud Account
Got to the Appwrite Console, create an account and login.
#### Create a Project
At the console click the `+ Create Project` button to create a new project. Remember the `project-id` which will be used later.
### Create an Appwrite Database and Collection
After creating an Appwrite project you have to create an Appwrite Database and a collection, you can either do this in code with the node-appwrite SDK or in the Appwrite Console as shown in this video:
### Add your documents attributes
In the appwrite collection, create all attributes of your documents. You have to define all the fields that your document in your [RxDB schema](./rx-schema.md) knows about. Notice that Appwrite does not allow for nested attributes. So when you use RxDB with Appwrite, you should also not have nested attributes in your RxDB schema.
### Add a `deleted` attribute
Appwrite (natively) hard-deletes documents. But for offline-handling RxDB needs soft-deleted documents on the server so that the deletion state can be replicated with other clients.
In RxDB, `_deleted` indicates that a document is removed locally and you need a similar field in your Appwrite collection on the Server: You must define a deletedField with any name to mark documents as "deleted" in the remote collection. Mostly you would use a boolean field named `deleted` (set it to `required`). The plugin will treat any document with `{ [deletedField]: true }` as deleted and replicate that state to local RxDB.
### Set the Permission on the Appwrite Collection
Appwrite uses permissions to control data access on the collection level. Make sure that in the Console at `Collection -> Settings -> Permissions` you have set the permission according to what you want to allow your clients to do. For testing, just enable all of them (Create, Read, Update and Delete).
## Setting up the RxDB - Appwrite Replication
Now that we have set up the Appwrite server, we can go to the client side code and set up RxDB and the replication:
### Install the Appwrite SDK and RxDB:
```bash
npm install appwrite rxdb
```
### Import the Appwrite SDK and RxDB
```ts
import {
replicateAppwrite
} from 'rxdb/plugins/replication-appwrite';
import {
createRxDatabase,
addRxPlugin,
RxCollection
} from 'rxdb/plugins/core';
import {
getRxStorageLocalstorage
} from 'rxdb/plugins/storage-localstorage';
import { Client } from 'appwrite';
```
### Create a Database with a Collection
```ts
const db = await createRxDatabase({
name: 'mydb',
storage: getRxStorageLocalstorage()
});
const mySchema = {
title: 'my schema',
version: 0,
primaryKey: 'id',
type: 'object',
properties: {
id: {
type: 'string',
maxLength: 100
},
name: {
type: 'string'
}
},
required: ['id', 'name']
};
await db.addCollections({
humans: {
schema: mySchema
}
});
const collection = db.humans;
```
### Configure the Appwrite Client
#### Appwrite Cloud
```ts
const client = new Client();
client.setEndpoint('https://cloud.appwrite.io/v1');
client.setProject('YOUR_APPWRITE_PROJECT_ID');
```
#### Self-Hosted
```ts
const client = new Client();
client.setEndpoint('http://localhost/v1');
client.setProject('YOUR_APPWRITE_PROJECT_ID');
```
### Start the Replication
```ts
const replicationState = replicateAppwrite({
replicationIdentifier: 'my-appwrite-replication',
client,
databaseId: 'YOUR_APPWRITE_DATABASE_ID',
collectionId: 'YOUR_APPWRITE_COLLECTION_ID',
deletedField: 'deleted', // Field that represents deletion in Appwrite
collection,
pull: {
batchSize: 10,
},
push: {
batchSize: 10
},
/*
* ...
* You can set all other options for RxDB replication states
* like 'live' or 'retryTime'
* ...
*/
});
```
### Do other things with the replication state
The `RxAppwriteReplicationState` which is returned from `replicateAppwrite()` allows you to run all functionality of the normal [RxReplicationState](./replication.md).
## Limitations of the Appwrite Replication Plugin
- Appwrite primary keys only allow for the characters `a-z`, `A-Z`, `0-9`, and underscore `_` (They cannot start with a leading underscore). Also the primary key has a max length of 36 characters.
- The Appwrite replication **only works on browsers**. This is because the Appwrite SDK does not support subscriptions in Node.js.
- Appwrite does not allow for bulk write operations so on push one HTTP request will be made per document. Reads run in bulk so this is mostly not a problem.
- Appwrite does not allow for transactions or "update-if" calls which can lead to overwriting documents instead of properly handling [conflicts](./transactions-conflicts-revisions.md#conflicts) when multiple clients edit the same document in parallel. This is not a problem for inserts because "insert-if-not" calls are made.
- Nested attributes in Appwrite collections are only possible via experimental relationship attributes, and compatibility with RxDB is not tested. Users opting to use these experimental relationship attributes with RxDB do so at their own risk.
---
## RxDB Server - Deploy Your Data
# RxDB Server
The RxDB Server Plugin makes it possible to spawn a server on top of a RxDB database that offers multiple types of endpoints for various usages. It can spawn basic CRUD REST endpoints or even realtime replication endpoints that can be used by the client devices to replicate data. The RxServer plugin is designed to be used in Node.js but you can also use it in Deno, Bun or the [Electron](./electron-database.md) "main" process. You can use it either as a **standalone server** or add it on top of an **existing http server** (like express) in nodejs.
## Starting a RxServer
To create an `RxServer`, you have to install the `rxdb-server` package with `npm install rxdb-server --save` and then you can import the `createRxServer()` function and create a server on a given [RxDatabase](./rx-database.md) and adapter.
After adding the endpoints to the server, do not forget to call `myServer.start()` to start the actually http-server.
```ts
import { createRxServer } from 'rxdb-server/plugins/server';
/**
* We use the express adapter which is the one that comes with RxDB core
* Make sure you have express installed in the correct version!
* @see https://github.com/pubkey/rxdb-server/blob/master/package.json
*/
import { RxServerAdapterExpress } from 'rxdb-server/plugins/adapter-express';
const myServer = await createRxServer({
database: myRxDatabase,
adapter: RxServerAdapterExpress,
port: 443
});
// add endpoints here (see below)
// after adding the endpoints, start the server
await myServer.start();
```
### Using RxServer with Fastify
There is also a [RxDB Premium π](/premium/) adapter to use the RxServer with [Fastify](https://fastify.dev/) instead of express. Fastify has shown to have better performance and in general is more modern.
```ts
import { createRxServer } from 'rxdb-server/plugins/server';
import { RxServerAdapterFastify } from 'rxdb-premium/plugins/server-adapter-fastify';
const myServer = await createRxServer({
database: myRxDatabase,
adapter: RxServerAdapterFastify,
port: 443
});
await myServer.start();
```
### Using RxServer with Koa
There is also a [RxDB Premium π](/premium/) adapter to use the RxServer with [Koa](https://koajs.com/) instead of express. Koa has shown to have better performance compared to express.
```ts
import { createRxServer } from 'rxdb-server/plugins/server';
import { RxServerAdapterKoa } from 'rxdb-premium/plugins/server-adapter-koa';
const myServer = await createRxServer({
database: myRxDatabase,
adapter: RxServerAdapterKoa,
port: 443
});
await myServer.start();
```
## RxServer Endpoints
On top of the RxServer you can add different types of **endpoints**. An endpoint is always connected to exactly one [RxCollection](./rx-collection.md) and it only serves data from that single collection.
For now there are only two endpoints implemented, the [replication endpoint](#replication-endpoint) and the [REST endpoint](#rest-endpoint). Others will be added in the future.
An endpoint is added to the server by calling the add endpoint method like `myRxServer.addReplicationEndpoint()`. Each needs a different `name` string as input which will define the resulting endpoint url.
The endpoint urls is a combination of the given `name` and schema `version` of the collection, like `/my-endpoint/0`.
```ts
const myEndpoint = server.addReplicationEndpoint({
name: 'my-endpoint',
collection: myServerCollection
});
console.log(myEndpoint.urlPath) // > 'my-endpoint/0'
```
Notice that it is **not required** that the server side schema version is equal to the client side schema version. You might want to change server schemas more often and then only do a [migration](./migration-schema.md) on the server, not on the clients.
## Replication Endpoint
The replication endpoint allows clients that connect to it to replicate data with the server via the [RxDB Sync Engine](./replication.md). There is also the [Replication Server](./replication-server.md) plugin that is used on the client side to connect to the endpoint.
The endpoint is added to the server with the `addReplicationEndpoint()` method. It requires a specific collection and the endpoint will only provided replication for documents inside of that collection.
```ts
// > server.ts
const endpoint = server.addReplicationEndpoint({
name: 'my-endpoint',
collection: myServerCollection
});
```
Then you can start the [Server Replication](./replication-server.md) on the client:
```ts
// > client.ts
const replicationState = await replicateServer({
collection: usersCollection,
replicationIdentifier: 'my-server-replication',
url: 'http://localhost:80/my-endpoint/0',
push: {},
pull: {}
});
```
## REST endpoint
The REST endpoint exposes various methods to access the data from the RxServer with non-RxDB tools via plain HTTP operations. You can use it to connect apps that are programmed in different programming languages than JavaScript or to access data from other third party tools.
Creating a REST endpoint on a RxServer:
```ts
const endpoint = await server.addRestEndpoint({
name: 'my-endpoint',
collection: myServerCollection
});
```
```ts
// plain http request with fetch
const request = await fetch('http://localhost:80/' + endpoint.urlPath + '/query', {
method: 'POST',
headers: {
'Accept': 'application/json',
'Content-Type': 'application/json'
},
body: JSON.stringify({ selector: {} })
});
const response = await request.json();
```
There is also the `client-rest` plugin that provides type-save interactions with the REST endpoint:
```ts
// using the client (optional)
import { createRestClient } from 'rxdb-server/plugins/client-rest';
const client = createRestClient('http://localhost:80/' + endpoint.urlPath, {/* headers */});
const response = await client.query({ selector: {} });
```
The REST endpoint exposes the following paths:
- **query [POST]**: Fetch the results of a NoSQL query.
- **query/observe [GET]**: Observe a query's results via [Server Send Events](https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events/Using_server-sent_events).
- **get [POST]**: Fetch multiple documents by their primary key.
- **set [POST]**: Write multiple documents at once.
- **delete [POST]**: Delete multiple documents by their primary key.
## CORS
When creating a server or adding endpoints, you can specify a [CORS](https://developer.mozilla.org/en-US/docs/Web/HTTP/CORS) string.
Endpoint cors always overwrite server cors. The default is the wildcard `*` which allows all requests.
```ts
const myServer = await startRxServer({
database: myRxDatabase,
cors: 'http://example.com'
port: 443
});
const endpoint = await server.addReplicationEndpoint({
name: 'my-endpoint',
collection: myServerCollection,
cors: 'http://example.com'
});
```
## Auth handler
To authenticate users and to make user-specific data available on server requests, an `authHandler` must be provided that parses the headers and returns the actual auth data that is used to authenticate the client and in the [queryModifier](#query-modifier) and [changeValidator](#change-validator).
An auth handler gets the given headers object as input and returns the auth data in the format `{ data: {}, validUntil: 1706579817126}`.
The `data` field can contain any data that can be used afterwards in the queryModifier and changeValidator.
The `validUntil` field contains the unix timestamp in milliseconds at which the authentication is no longer valid and the client will get disconnected.
For example your authHandler could get the `Authorization` header and parse the [JSON web token](https://jwt.io/) to identify the user and store the user id in the `data` field for later use.
## Query modifier
The query modifier is a JavaScript function that is used to restrict which documents a client can fetch or replicate from the server.
It gets the auth data and the actual NoSQL query as input parameter and returns a modified NoSQL query that is then used internally by the server.
You can pass a different query modifier to each endpoint so that you can have different endpoints for different use cases on the same server.
For example you could use a query modifier that get the `userId` from the auth data and then restricts the query to only return documents that have the same `userId` set.
```ts
function myQueryModifier(authData, query) {
query.selector.userId = { $eq: authData.data.userid };
return query;
}
const endpoint = await server.addReplicationEndpoint({
name: 'my-endpoint',
collection: myServerCollection,
queryModifier: myQueryModifier
});
```
The RxServer will use the queryModifier at many places internally to determine which queries to run or if a document is allowed to be seen/edited by a client.
:::note
For performance reasons the `queryModifier` and `changeValidator` **MUST NOT** be `async` and return a promise. If you need async data to run them, you should gather that data in the `RxServerAuthHandler` and store it in the auth data to access it later.
:::
## Change validator
The change validator is a JavaScript function that is used to restrict which document writes are allowed to be done by a client.
For example you could restrict clients to only change specific document fields or to not do any document writes at all.
It can also be used to validate change document data before storing it at the server.
In this example we restrict clients from doing inserts and only allow updates. For that we check if the change contains an `assumedMasterState` property and return false to block the write.
```ts
function myChangeValidator(authData, change) {
if(change.assumedMasterState) {
return false;
} else {
return true;
}
}
const endpoint = await server.addReplicationEndpoint({
name: 'my-endpoint',
collection: myServerCollection,
changeValidator: myChangeValidator
});
```
## Server-only indexes
Normal RxDB schema indexes get the `_deleted` field prepended because all [RxQueries](./rx-query.md) automatically only search for documents with `_deleted=false`.
When you use RxDB on a server, this might not be optimal because there can be the need to query for documents where the value of `_deleted` does not matter. Mostly this is required in the [pull.stream$](./replication.md#checkpoint-iteration) of a replication when a [queryModifier](#query-modifier) is used to add an additional field to the query.
To set indexes without `_deleted`, you can use the `internalIndexes` field of the schema like the following:
```json
{
"version": 0,
"primaryKey": "id",
"type": "object",
"properties": {
"id": {
"type": "string",
"maxLength": 100
},
"name": {
"type": "string",
"maxLength": 100
}
},
"internalIndexes": [
["name", "id"]
]
}
```
:::note
Indexes come with a performance burden. You should only use the indexes you need and make sure you **do not** accidentally set the `internalIndexes` in your client side [RxCollections](./rx-collection.md).
:::
## Server-only fields
All endpoints can be created with the `serverOnlyFields` set which defines some fields to only exist on the server, not on the clients. Clients will not see that fields and cannot do writes where one of the `serverOnlyFields` is set.
Notice that when you use `serverOnlyFields` you likely need to have a different schema on the server than the schema that is used on the clients.
```ts
const endpoint = await server.addReplicationEndpoint({
name: 'my-endpoint',
collection: col,
// here the field 'my-secretss' is defined to be server-only
serverOnlyFields: ['my-secrets']
});
```
:::note
For performance reasons, only top-level fields can be used as `serverOnlyFields`. Otherwise the server would have to deep-clone all document data which is too expensive.
:::
## Readonly fields
When you have fields that should only be modified by the server, but not by the client, you can ensure that by comparing the fields value in the [changeValidator](#change-validator).
```ts
const myChangeValidator = function(authData, change){
if(change.newDocumentState.myReadonlyField !== change.assumedMasterState.myReadonlyField){
throw new Error('myReadonlyField is readonly');
}
}
```
## $regex queries not allowed
`$regex` queries are not allowed to run at the server to prevent [ReDos Attacks](https://owasp.org/www-community/attacks/Regular_expression_Denial_of_Service_-_ReDoS).
## Conflict handling
To [detect and handle conflicts](./replication.md#conflict-handling), the conflict handler from the endpoints RxCollection is used.
## FAQ
Why are the server plugins in a different github repo and npm package?
The RxServer and its other plugins are in a different github repository because:
It has too many dependencies that you do not want to install if you only use RxDB at the client side
It has a different license (SSPL) to prevent large cloud vendors from "stealing" the revenue, similar to MongoDB's license.
Why can't endpoints be added dynamically?
After `RxServer.start()` is called, you can no longer add endpoints. This is because many of the supported
server libraries do not allow dynamic routing for performance and security reasons.
---
## RxServer Scaling - Vertical or Horizontal
# Scaling the RxServer
The [RxDB Server](./rx-server.md) run in JavaScript and JavaScript runs on a single process on the operating system. This can make the CPU performance limit to be the main bottleneck when serving requests to your users. To mitigate that problem, there are a wide range of methods to scale up the server so that it can serve more requests at the same time faster.
## Vertical Scaling
Vertical Scaling aka "scaling up" has the goal to get more power out of a single server by utilizing more of the servers compute. Vertical scaling should be the first step when you decide it is time to scale.
### Run multiple JavaScript processes
To utilize more compute power of your server, the first step is to scale vertically by running the RxDB server on **multiple processes** in parallel.
RxDB itself is already build to support multiInstance-usage on the client, like when the user has opened multiple browser tabs at once. The same method works also on the server side in Node.js. You can spawn multiple JavaScript processes that use the same [RxDatabase](./rx-database.md) and the instances will automatically communicate with each other and distribute their data and events with the [BroadcastChannel](https://github.com/pubkey/broadcast-channel).
By default the [multiInstance param](./rx-database.md#multiinstance) is set to `true` when calling `createRxDatabase()`, so you do not have to change anything. To make all processes accessible through the same endpoint, you can put a load-balancer like [nginx](https://nginx.org/en/docs/http/load_balancing.html) in front of them.
### Using workers to split up the load
Another way to increases the server capacity is to put the storage into a [Worker thread](./rx-storage-worker.md) so that the "main" thread with the webserver can handle more requests. This might be easier to set up compared to using multiple JavaScript processes and a load balancer.
### Use an in-memory storage at the user facing level
Another way to serve more requests to your end users, is to use an [in-memory](./rx-storage-memory.md) storage that has the [best](./rx-storage-performance.md) read- and write performance. It outperforms persistent storages by a factor of 10x.
So instead of directly serving requests from the persistence layer, you add an in-memory layer on top of that. You could either do a [replication](./replication.md) from your memory database to the persistent one, or you use the [memory mapped](./rx-storage-memory-mapped.md) storage which has this build in.
```ts
import { getRxStorageMemory } from 'rxdb/plugins/storage-memory';
import { replicateRxCollection } from 'rxdb/plugins/replication';
import { getRxStorageFilesystemNode } from 'rxdb-premium/plugins/storage-filesystem-node';
import { getMemoryMappedRxStorage } from 'rxdb-premium/plugins/storage-memory-mapped';
const myRxDatabase = await createRxDatabase({
name: 'mydb',
storage: getMemoryMappedRxStorage({
storage: getRxStorageFilesystemNode({
basePath: path.join(__dirname, 'my-database-folder')
})
})
});
await myDatabase.addCollections({/* ... */});
const myServer = await startRxServer({
database: myRxDatabase,
port: 443
});
```
But notice that you have to check your persistence requirements. When a write happens to the memory layer and the server crashes while it has not persisted, in rare cases the write operation might get lost. You can remove that risk by setting `awaitWritePersistence: true` on the [memory mapped storage](./rx-storage-memory-mapped.md) settings.
## Horizontal Scaling
To scale the RxDB Server above a single physical hardware unit, there are different solutions where the decision depends on the exact use case.
### Single Datastore with multiple branches
The most common way to use multiple servers with RxDB is to split up the server into a tree with a root "datastore" and multiple "branches". The datastore contains the persisted data and only servers as a replication endpoint for the branches. The branches themself will replicate data to and from the datastore and server requests to the end users.
This is mostly useful on read-heavy applications because reads will directly run on the branches without ever reaching the main datastore and you can always add more branches to **scale up**. Even adding additional layers of "datastores" is possible so the tree can grow (or shrink) with the demand.
### Moving the branches to "the edge"
Instead of running the "branches" of the tree on the same physical location as the datastore, it often makes sense to move the branches into a datacenter near the end users. Because the RxDB [replication algorithm](./replication.md) is made to work with slow and even partially offline users, using it for physically separated servers will work the same way. Latency is not that important because writes and reads will not decrease performance by blocking each other and the replication can run in the background without blocking other servers during transaction.
### Replicate Databases for Microservices
If your application is build with a [microservice architecture](https://en.wikipedia.org/wiki/Microservices) and your microservices are also build in Node.js, you can scale the database horizontally by moving the database into the microservices and use the [RxDB replication](./replication.md) to do a realtime sync between the microservices and a main "datastore" server. The "datastore" server would then only handle the replication requests or do some additional things like logging or [backups](./backup.md). The compute for reads and writes will then mainly be done on the microservices themself. This simplifies setting up more and more microservices without decreasing the performance of the whole system.
### Use a self-scaling RxStorage
An alternative to scaling up the RxDB servers themself, you can also switch to a [RxStorage](./rx-storage.md) which scales up internally. For example the [FoundationDB storage](./rx-storage-foundationdb.md) or [MongoDB](./rx-storage-mongodb.md) can work on top of a cluster that can increase load by adding more servers to itself. With that you can always add more Node.js RxDB processes that connect to the same cluster and server requests from it.
---
## Transactions, Conflicts and Revisions
In contrast to most SQL databases, RxDB does not have the concept of relational ACID transactions. Instead, RxDB has to apply different techniques that better suit the offline-first, client-side world where it is not possible to create a transaction between multiple maybe-offline client devices.
## Why RxDB does not have transactions
When talking about transactions, we mean [ACID transactions](https://en.wikipedia.org/wiki/ACID) that guarantee the properties of atomicity, consistency, isolation and durability.
With an ACID transaction you can mutate data dependent on the current state of the database. It is ensured that no other database operations happen in between your transaction and after the transaction has finished, it is guaranteed that the new data is actually written to the disk.
To implement ACID transactions on a **single server**, the database has to keep track on who is running transactions and then schedule these transactions so that they can run in isolation.
As soon as you have to split your database on **multiple servers**, transaction handling becomes way more difficult. The servers have to communicate with each other to find a consensus about which transaction can run and which has to wait. Network connections might break, or one server might complete its part of the transaction and then be required to roll back its changes because of an error on another server.
But with RxDB you have **multiple clients** that can go randomly online or offline. The users can have different devices and the clock of these devices can go off by any time. To support ACID transactions here, RxDB would have to make the whole world stand still for all clients, while one client is doing a write operation. And even that can only work when all clients are online. Implementing that might be possible, but at the cost of an unpredictable amount of performance loss and not being able to support [offline-first](./offline-first.md).
> A single write operation to a document is the only atomic thing you can do in [RxDatabase](./rx-database.md).
The benefits of not having to support transactions:
- Clients can read and write data without blocking each other.
- Clients can write data while being **offline** and then replicate with a server when they are **online** again, called [offline-first](./offline-first.md).
- Creating a compatible backend for the replication is easy so that RxDB can replicate with any existing infrastructure.
- Optimizations like [Sharding](./rx-storage-sharding.md) can be used.
## Revisions
Working without transactions leads to having undefined state when doing multiple database operations at the same time. Most client-side databases rely on a last-write-wins strategy on write operations. This might be a viable solution for some cases, but often this leads to strange problems that are hard to debug.
Instead, to ensure that the behavior of RxDB is **always predictable**, RxDB relies on **revisions** for version control. Revisions work similar to [Lamport Clocks](https://martinfowler.com/articles/patterns-of-distributed-systems/lamport-clock.html).
Each document is stored together with its revision string, that looks like `1-9dcca3b8e1a` and consists of:
- The revision height, a number that starts with `1` and is increased with each write to that document.
- The database instance token.
An operation to the RxDB data layer does not only contain the new document data, but also the previous document data with its revision string. If the previous revision matches the revision that is currently stored in the database, the write operation can succeed. If the previous revision is **different** than the revision that is currently stored in the database, the operation will throw a `409 CONFLICT` error.
## Conflicts
There are two types of conflicts in RxDB, the **local conflict** and the **replication conflict**.
### Local conflicts
A local conflict can happen when a write operation assumes a different previous document state, than what is currently stored in the database. This can happen when multiple parts of your application do simultaneous writes to the same document. This can happen on a single browser tab, or when multiple tabs write at once or when a write appears while the document gets replicated from a remote server replication.
When a local conflict appears, RxDB will throw a `409 CONFLICT` error. The calling code must then handle the error properly, depending on the application logic.
Instead of handling local conflicts, in most cases it is easier to ensure that they cannot happen, by using `incremental` database operations like [incrementalModify()](./rx-document.md), [incrementalPatch()](./rx-document.md) or [incrementalUpsert()](./rx-collection.md). These write operations have a built-in way to handle conflicts by re-applying the mutation functions to the conflicting document state.
## Replication conflicts
A replication conflict appears when multiple clients write to the same documents at once and these documents are then replicated to the backend server.
When you replicate with the [GraphQL replication](./replication-graphql.md) and the [replication primitives](./replication.md), RxDB assumes that conflicts are **detected** and **resolved** at the client side.
When a document is sent to the backend and the backend detected a conflict (by comparing revisions or other properties), the backend will respond with the actual document state so that the client can compare this with the local document state and create a new, resolved document state that is then pushed to the server again. You can read more about the replication conflicts [here](./replication.md#conflict-handling).
## Custom conflict handler
A conflict handler is an object with two JavaScript functions:
- Detect if two document states are equal
- Solve existing conflicts
Because the conflict handler also is used for conflict detection, it will run many times on pull-, push- and write operations of RxDB. Most of the time it will detect that there is no conflict and then return.
Lets have a look at the [default conflict handler](https://github.com/pubkey/rxdb/blob/master/src/replication-protocol/default-conflict-handler.ts) of RxDB to learn how to create a custom one:
```ts
import { deepEqual } from 'rxdb/plugins/utils';
export const defaultConflictHandler: RxConflictHandler = {
isEqual(a, b) {
/**
* isEqual() is used to detect conflicts or to detect if a
* document has to be pushed to the remote.
* If the documents are deep equal,
* we have no conflict.
* Because deepEqual is CPU expensive, on your custom conflict handler you might only
* check some properties, like the updatedAt time or revisions
* for better performance.
*/
return deepEqual(a, b);
},
resolve(i) {
/**
* The default conflict handler will always
* drop the fork state and use the master state instead.
*
* In your custom conflict handler you likely want to merge properties
* of the realMasterState and the newDocumentState instead.
*/
return i.realMasterState;
}
};
```
To overwrite the default conflict handler, you have to specify a custom `conflictHandler` property when creating a collection with `addCollections()`.
```js
const myCollections = await myDatabase.addCollections({
// key = collectionName
humans: {
schema: mySchema,
conflictHandler: myCustomConflictHandler
}
});
```
---
## Efficient RxDB Queries via Query Cache
# QueryCache
RxDB uses a `QueryCache` which optimizes the reuse of queries at runtime. This makes sense especially when RxDB is used in UI-applications where people move for- and backwards on different routes or pages and the same queries are used many times. Because of the [event-reduce algorithm](https://github.com/pubkey/event-reduce) cached queries are even valuable for optimization, when changes to the database occur between now and the last execution.
## Cache Replacement Policy
To not let RxDB fill up all the memory, a `cache replacement policy` is defined that clears up the cached queries. This is implemented as a function which runs regularly, depending on when queries are created and the database is idle. The default policy should be good enough for most use cases but defining custom ones can also make sense.
## The default policy
The default policy starts cleaning up queries depending on how much queries are in the cache and how much document data they contain.
* It will never uncache queries that have subscribers to their results
* It tries to always have less than 100 queries without subscriptions in the cache.
* It prefers to uncache queries that have never executed and are older than 30 seconds
* It prefers to uncache queries that have not been used for longer time
## Other references to queries
With JavaScript, it is not possible to count references to variables. Therefore it might happen that an uncached `RxQuery` is still referenced by the users code and used to get results. This should never be a problem, uncached queries must still work. Creating the same query again however, will result in having two `RxQuery` instances instead of one.
## Using a custom policy
A cache replacement policy is a normal JavaScript function according to the type `RxCacheReplacementPolicy`.
It gets the `RxCollection` as first parameter and the `QueryCache` as second. Then it iterates over the cached `RxQuery` instances and uncaches the desired ones with `uncacheRxQuery(rxQuery)`. When you create your custom policy, you should have a look at the [default](https://github.com/pubkey/rxdb/blob/master/src/query-cache.ts).
To apply a custom policy to a [RxCollection](./rx-collection.md), add the function as attribute `cacheReplacementPolicy`.
```ts
const collection = await myDatabase.addCollections({
humans: {
schema: mySchema,
cacheReplacementPolicy: function(){ /* ... */ }
}
});
```
---
## Creating Plugins
Creating your own plugin is very simple. A plugin is basically a javascript-object which overwrites or extends RxDB's internal classes, prototypes, and hooks.
A basic plugin:
```javascript
const myPlugin = {
rxdb: true, // this must be true so rxdb knows that this is a rxdb-plugin
/**
* (optional) init() method
* that is called when the plugin is added to RxDB for the first time.
*/
init() {
// import other plugins or initialize stuff
},
/**
* every value in this object can manipulate the prototype of the keynames class
* You can manipulate every prototype in this list:
* @link https://github.com/pubkey/rxdb/blob/master/src/plugin.ts#L22
*/
prototypes: {
/**
* add a function to RxCollection so you can call 'myCollection.hello()'
*
* @param {object} prototype of RxCollection
*/
RxCollection: (proto) => {
proto.hello = function() {
return 'world';
};
}
},
/**
* some methods are static and can be overwritten in the overwritable-object
*/
overwritable: {
validatePassword: function(password) {
if (password && typeof password !== 'string' || password.length < 10)
throw new TypeError('password is not valid');
}
},
/**
* you can add hooks to the hook-list
*/
hooks: {
/**
* add a `foo` property to each document. You can then call myDocument.foo (='bar')
*/
createRxDocument: {
/**
* You can either add the hook running 'before' or 'after'
* the hooks of other plugins.
*/
after: function(doc) {
doc.foo = 'bar';
}
}
}
};
// now you can import the plugin into rxdb
addRxPlugin(myPlugin);
```
# Properties
## rxdb
The `rxdb`-property signals that this plugin is an rxdb-plugin. The value should always be `true`.
## prototypes
The `prototypes`-property contains a function for each of RxDB's internal prototype that you want to manipulate. Each function gets the prototype-object of the corresponding class as parameter and then can modify it. You can see a list of all available prototypes [here](https://github.com/pubkey/rxdb/blob/master/src/plugin.ts)
## overwritable
Some of RxDB's functions are not inside of a class-prototype but are static. You can set and overwrite them with the `overwritable`-object. You can see a list of all overwritables [here](https://github.com/pubkey/rxdb/blob/master/src/overwritable.ts).
# hooks
Sometimes you don't want to overwrite an existing RxDB-method, but extend it. You can do this by adding hooks which will be called each time the code jumps into the hooks corresponding call. You can find a list of all hooks [here](https://github.com/pubkey/rxdb/blob/master/src/hooks.ts).
# options
[RxDatabase](./rx-database.md) and [RxCollection](./rx-collection.md) have an additional options-parameter, which can be filled with any data required be the plugin.
```javascript
const collection = myDatabase.addCollections({
foo: {
schema: mySchema,
options: { // anything can be passed into the options
foo: ()=>'bar'
}
}
})
// Afterwards you can use these options in your plugin.
collection.options.foo(); // 'bar'
```
---
## Error Messages
# RxDB Error Messages
When RxDB has an error, an `RxError` object is thrown instead of a normal JavaScript `Error`. This `RxError` contains additional properties such as a `code` field and `parameters`. By default the full human readable error messages are not included into the RxDB build. This is because error messages have a high entropy and cannot be compressed well. Therefore only an error message with the correct error-code and parameters is thrown but without the full text.
When you enable the [DevMode Plugin](./dev-mode.md) the full error messages are added to the `RxError`. This should only be done in development, not in production builds to keep a small build size.
## All RxDB error messages
import { ErrorMessages } from '@site/src/components/error-messages';
---
## Testing
Writing tests for your RxDB application is crucial to ensure reliability. Because RxDB runs in many different environments (Browser, [Node.js](nodejs-database.md), [React Native](react-native-database.md), [Electron](electron-database.md), ...), testing strategies might vary. However, there are some common patterns that make testing easier and faster.
## Use the `memory` RxStorage
For unit tests, you should generally use the [`memory` RxStorage](rx-storage-memory.md). It keeps data only in memory, which has several advantages:
- **Speed**: It is much faster than writing to disc.
- **Isolation**: Each test run starts with a clean state; you don't have to delete database files between tests.
```ts
import { createRxDatabase } from 'rxdb';
import { getRxStorageMemory } from 'rxdb/plugins/storage-memory';
const db = await createRxDatabase({
name: 'test-db',
storage: getRxStorageMemory()
});
```
## The `using` Keyword
RxDB supports the [Explicit Resource Management](https://www.typescriptlang.org/docs/handbook/release-notes/typescript-5-2.html#using-declarations-and-explicit-resource-management) `using` keyword (available in TypeScript 5.2+). This automatically closes the database when the variable goes out of scope, which is perfect for tests.
Instead of manually calling `await db.close()`, you can do:
```ts
describe('my test suite', () => {
it('should insert a document', async () => {
// Did you know?
// using 'using' ensures db.close() is called automatically
// at the end of the test function.
await using db = await createRxDatabase({
name: 'test-db',
storage: getRxStorageMemory()
});
await db.addCollections({ ... });
// ... run your tests
});
});
```
## Cleanup
When running many tests, it is important to ensure that all databases are cleaned up after your tests run. Having non-closed `RxDatabase` instances after some tests can significantly decrease performance because background tasks and event listeners are still active.
A good practice is to verify that no database instances or connections are left open. You can check internal RxDB states to ensure everything is closed.
```ts
import { dbCount } from 'rxdb/plugins/core';
import assert from 'assert';
describe('cleanup', () => {
it('ensure every db is cleaned up', () => {
assert.strictEqual(dbCount(), 0);
});
});
```
## Multi-Tab Simulation
To test multi-tab behavior (like [Leader Election](leader-election.md) or [Replication](replication.md)) within a single Node.js process or test runner, you can create multiple [`RxDatabase`](rx-database.md) instances with the **same name** and storage. RxDB will treat them as if they were running in different tabs or processes. Notice that for this, [ignoreDuplicate](./rx-database.md#ignoreduplicate) must be set to `true` because otherwise it will not allow to create multiple databases with the same name in a single JavaScript process.
```ts
import { createRxDatabase } from 'rxdb';
import { getRxStorageMemory } from 'rxdb/plugins/storage-memory';
// Simulate Tab 1
const db1 = await createRxDatabase({
name: 'test-db', // same name
storage: getRxStorageMemory(),
ignoreDuplicate: true // must be set to true
});
await db1.addCollections({ ... });
// Simulate Tab 2
const db2 = await createRxDatabase({
name: 'test-db', // same name
storage: getRxStorageMemory(),
ignoreDuplicate: true // must be set to true
});
await db2.addCollections({ ... });
// insert at "tab one"
await db1.todos.insert({ id: "foobar"});
// read at "tab two"
const doc = await db2.todos.findOne("foobar").exec(true);
assert.ok(doc);
```
This works because the `memory` storage (and others) are shared within the same JavaScript process.
---
## Seamless Schema Data Migration with RxDB
# Migrate Database Data on schema changes
The RxDB Data Migration Plugin helps developers easily update stored data in their apps when they make changes to the data structure by changing the schema of a [RxCollection](./rx-collection.md). This is useful when developers release a new version of the app with a different schema.
Imagine you have your awesome messenger-app distributed to many users. After a while, you decide that in your new version, you want to change the schema of the messages-collection. Instead of saving the message-date like `2017-02-12T23:03:05+00:00` you want to have the unix-timestamp like `1486940585` to make it easier to compare dates. To accomplish this, you change the schema and **increase the version-number** and you also change your code where you save the incoming messages. But one problem remains: what happens with the messages which are already stored in the database on the user's device in the old schema?
With RxDB you can provide migrationStrategies for your collections that automatically (or on call) transform your existing data from older to newer schemas. This assures that the client's data always matches your newest code-version.
# Add the migration plugin
To enable the data migration, you have to add the `migration-schema` plugin.
```ts
import { addRxPlugin } from 'rxdb';
import { RxDBMigrationSchemaPlugin } from 'rxdb/plugins/migration-schema';
addRxPlugin(RxDBMigrationSchemaPlugin);
```
## Providing strategies
Upon creation of a collection, you have to provide migrationStrategies when your schema's version-number is greater than `0`. To do this, you have to add an object to the `migrationStrategies` property where a function for every schema-version is assigned. A migrationStrategy is a function which gets the old document-data as a parameter and returns the new, transformed document-data. If the strategy returns `null`, the document will be removed instead of migrated.
```javascript
myDatabase.addCollections({
messages: {
schema: messageSchemaV1,
migrationStrategies: {
// 1 means, this transforms data from version 0 to version 1
1: function(oldDoc){
oldDoc.time = new Date(oldDoc.time).getTime(); // string to unix
return oldDoc;
}
}
}
});
```
Asynchronous strategies can also be used:
```javascript
myDatabase.addCollections({
messages: {
schema: messageSchemaV1,
migrationStrategies: {
1: function(oldDoc){
oldDoc.time = new Date(oldDoc.time).getTime(); // string to unix
return oldDoc;
},
/**
* 2 means, this transforms data from version 1 to version 2
* this returns a promise which resolves with the new document-data
*/
2: function(oldDoc){
// in the new schema (version: 2) we defined 'senderCountry' as required field (string)
// so we must get the country of the message-sender from the server
const coordinates = oldDoc.coordinates;
return fetch('http://myserver.com/api/countryByCoordinates/'+coordinates+'/')
.then(response => {
const response = response.json();
oldDoc.senderCountry = response;
return oldDoc;
});
}
}
}
});
```
you can also filter which documents should be migrated:
```js
myDatabase.addCollections({
messages: {
schema: messageSchemaV1,
migrationStrategies: {
// 1 means, this transforms data from version 0 to version 1
1: function(oldDoc){
oldDoc.time = new Date(oldDoc.time).getTime(); // string to unix
return oldDoc;
},
/**
* this removes all documents older then 2017-02-12
* they will not appear in the new collection
*/
2: function(oldDoc){
if(oldDoc.time < 1486940585) return null;
else return oldDoc;
}
}
}
});
```
## autoMigrate
By default, the migration automatically happens when the collection is created. Calling `RxDatabase.addCollections()` returns only when the migration has finished.
If you have lots of data or the migrationStrategies take a long time, it might be better to start the migration 'by hand' and show the migration-state to the user as a loading-bar.
```javascript
const messageCol = await myDatabase.addCollections({
messages: {
schema: messageSchemaV1,
autoMigrate: false, // <- migration will not run at creation
migrationStrategies: {
1: async function(oldDoc){
...
anything that takes very long
...
return oldDoc;
}
}
}
});
// check if migration is needed
const needed = await messageCol.migrationNeeded();
if(needed === false) {
return;
}
// start the migration
messageCol.startMigration(10); // 10 is the batch-size, how many docs will run at parallel
const migrationState = messageCol.getMigrationState();
// 'start' the observable
migrationState.$.subscribe({
next: state => console.dir(state),
error: error => console.error(error),
complete: () => console.log('done')
});
// the emitted states look like this:
{
status: 'RUNNING' // oneOf 'RUNNING' | 'DONE' | 'ERROR'
count: {
total: 50, // amount of documents which must be migrated
handled: 0, // amount of handled docs
percent: 0 // percentage [0-100]
}
}
```
If you don't want to show the state to the user, you can also use `.migratePromise()`:
```js
const migrationPromise = messageCol.migratePromise(10);
await migratePromise;
```
## migrationStates()
`RxDatabase.migrationStates()` returns an `Observable` that emits all migration states of any collection of the database.
Use this when you add collections dynamically and want to show a loading-state of the migrations to the user.
```js
const allStatesObservable = myDatabase.migrationStates();
allStatesObservable.subscribe(allStates => {
allStates.forEach(migrationState => {
console.log(
'migration state of ' +
migrationState.collection.name
);
});
});
```
## Migrating attachments
When you store [RxAttachment](./rx-attachment.md)s together with your document, they can also be changed, added or removed while running the migration.
You can do this by mutating the `oldDoc._attachments` property.
```js
import { createBlob } from 'rxdb';
const migrationStrategies = {
1: async function(oldDoc){
// do nothing with _attachments to keep all attachments and have them in the new collection version.
return oldDoc;
},
2: async function(oldDoc){
// set _attachments to an empty object to delete all existing ones during the migration.
oldDoc._attachments = {};
return oldDoc;
},
3: async function(oldDoc){
// update the data field of a single attachment to change its data.
oldDoc._attachments.myFile.data = await createBlob(
'my new text',
oldDoc._attachments.myFile.content_type
);
return oldDoc;
}
}
```
## Migration on multi-tab in browsers
If you use RxDB in a multiInstance environment, like a browser, it will ensure that exactly one tab is running a migration of a collection.
Also the `migrationState.$` events are emitted between browser tabs.
## Migration and Replication
If you use any of the [RxReplication](./replication.md) plugins, the migration will also run on the internal replication-state storage. It will migrate all `assumedMasterState` documents
so that after the migration is done, you do not have to re-run the replication from scratch.
RxDB assumes that you run the exact same migration on the servers and the clients. Notice that the replication `pull-checkpoint` will not be migrated. Your backend must be compatible with pull-checkpoints of older versions.
## Migration should be run on all database instances
If you have multiple database instances (for example, if you are running replication inside of a [Worker](./rx-storage-worker.md) or [SharedWorker](./rx-storage-shared-worker.md) and have created a database instance inside of the worker), schema migration should be started on all database instances. All instances must know about all migration strategies and any updated schema versions.
---
## Migration Storage
# Storage Migration
The storage migration plugin can be used to migrate all data from one existing RxStorage into another. This is useful when:
- You want to migrate from one [RxStorage](./rx-storage.md) to another one.
- You want to migrate to a new major RxDB version while keeping the previous saved data. This function only works from the previous major version upwards. Do not use it to migrate like rxdb v9 to v14.
The storage migration **drops deleted documents** and filters them out during the migration.
:::warning Do never change the schema while doing a storage migration
When you migrate between storages, you might want to change the schema in the same process. You should never do that because it will lead to problems afterwards and might make your database unusable.
When you also want to change your schema, first run the storage migration and afterwards run a normal [schema migration](./migration-schema.md).
:::
## Usage
Lets say you want to migrate from [LocalStorage RxStorage](./rx-storage-localstorage.md) to the [IndexedDB RxStorage](./rx-storage-indexeddb.md).
```ts
import { migrateStorage } from 'rxdb/plugins/migration-storage';
import { getRxStorageIndexedDB } from 'rxdb-premium/plugins/storage-indexeddb';
import { getRxStorageLocalstorage } from 'rxdb-old/plugins/storage-localstorage';
// create the new RxDatabase
const db = await createRxDatabase({
name: dbLocation,
storage: getRxStorageIndexedDB(),
multiInstance: false
});
await migrateStorage({
database: db as any,
/**
* Name of the old database,
* using the storage migration requires that the
* new database has a different name.
*/
oldDatabaseName: 'myOldDatabaseName',
oldStorage: getRxStorageLocalstorage(), // RxStorage of the old database
batchSize: 500, // batch size
parallel: false, // <- true if it should migrate all collections in parallel. False (default) if should migrate in serial
afterMigrateBatch: (input: AfterMigrateBatchHandlerInput) => {
console.log('storage migration: batch processed');
}
});
```
:::note
Only collections that exist in the new database at the time you call migrateStorage() will have their data migrated.
- If your old database had collections `['users', 'posts', 'comments']` but your new database only defines `['users', 'posts']`, then only users and posts data will be migrated.
- Any collections missing from the new database will simply be skipped - no data for them will be read or written.
This allows you to selectively migrate only certain collections if desired, by choosing which collections to define before invoking `migrateStorage()`.
:::
## Migrate from a previous RxDB major version
To migrate from a previous RxDB major version, you have to install the 'old' RxDB in the `package.json`
```json
{
"dependencies": {
"rxdb-old": "npm:rxdb@14.17.1",
}
}
```
Then you can run the migration by providing the old storage:
```ts
/* ... */
import { migrateStorage } from 'rxdb/plugins/migration-storage';
import { getRxStorageLocalstorage } from 'rxdb-old/plugins/storage-localstorage'; // <- import from the old RxDB version
await migrateStorage({
database: db as any,
/**
* Name of the old database,
* using the storage migration requires that the
* new database has a different name.
*/
oldDatabaseName: 'myOldDatabaseName',
oldStorage: getRxStorageLocalstorage(), // RxStorage of the old database
batchSize: 500, // batch size
parallel: false,
afterMigrateBatch: (input: AfterMigrateBatchHandlerInput) => {
console.log('storage migration: batch processed');
}
});
/* ... */
```
## Disable Version Check on [RxDB Premium π](/premium/)
RxDB Premium has a check in place that ensures that you do not accidentally use the wrong RxDB core and π Premium version together which could break your database state.
This can be a problem during migrations where you have multiple versions of RxDB in use and it will throw the error `Version mismatch detected`.
You can disable that check by importing and running the `disableVersionCheck()` function from RxDB Premium.
```ts
// RxDB Premium v15 or newer:
import {
disableVersionCheck
} from 'rxdb-premium-old/plugins/shared';
disableVersionCheck();
// RxDB Premium v14:
// for esm
import {
disableVersionCheck
} from 'rxdb-premium-old/dist/es/shared/version-check.js';
disableVersionCheck();
// for cjs
import {
disableVersionCheck
} from 'rxdb-premium-old/dist/lib/shared/version-check.js';
disableVersionCheck();
```
---
## Attachments
import { DefaultCompressibleTypes } from '@site/src/components/default-compressible-types';
# Attachments
Attachments are binary data files that can be attachment to an `RxDocument`, like a file that is attached to an email.
Using attachments instead of adding the data to the normal document, ensures that you still have a good **performance** when querying and writing documents, even when a big amount of data, like an image file has to be stored.
- You can store string, binary files, images and whatever you want side by side with your documents.
- Deleted documents automatically loose all their attachments data.
- Not all replication plugins support the replication of attachments.
- Attachments can be stored [encrypted](./encryption.md).
Internally, attachment data is stored as `Blob` objects. Blob is the canonical internal type because it is immutable, carries MIME type metadata via `Blob.type`, provides synchronous size via `Blob.size`, and is [structured-cloneable](https://developer.mozilla.org/en-US/docs/Web/API/Web_Workers_API/Structured_clone_algorithm) (works with Worker/Electron `postMessage` and IndexedDB). Conversion to `ArrayBuffer` only happens at system boundaries that require it: encryption (Web Crypto), compression (CompressionStream), digest hashing, and WebSocket serialization.
## Add the attachments plugin
To enable the attachments, you have to add the `attachments` plugin.
```ts
import { addRxPlugin } from 'rxdb';
import { RxDBAttachmentsPlugin } from 'rxdb/plugins/attachments';
addRxPlugin(RxDBAttachmentsPlugin);
```
## Enable attachments in the schema
Before you can use attachments, you have to ensure that the attachments-object is set in the schema of your `RxCollection`.
```javascript
const mySchema = {
version: 0,
type: 'object',
properties: {
// .
// .
// .
},
attachments: {
encrypted: true // if true, the attachment-data will be encrypted with the db-password
}
};
const myCollection = await myDatabase.addCollections({
humans: {
schema: mySchema
}
});
```
## putAttachment()
Adds an attachment to a `RxDocument`. Returns a Promise with the new attachment.
```javascript
import { createBlob } from 'rxdb';
const attachment = await myDocument.putAttachment(
{
id: 'cat.txt', // (string) name of the attachment
data: createBlob('meowmeow', 'text/plain'), // (Blob) data of the attachment
type: 'text/plain' // (string) type of the attachment-data like 'image/jpeg'
}
);
```
:::warning
Expo/React-Native does not support the `Blob` API natively. Make sure you use your own polyfill that properly supports `blob.arrayBuffer()` when using RxAttachments or use the `putAttachmentBase64()` and `getDataBase64()` so that you do not have to create blobs.
:::
## putAttachments()
Write multiple attachments to a `RxDocument` in a single atomic operation. This is more efficient than calling `putAttachment()` multiple times because it only performs one write to the storage. Returns a Promise with an array of the new attachments.
```javascript
import { createBlob } from 'rxdb';
const attachments = await myDocument.putAttachments([
{
id: 'cat.txt',
data: createBlob('meowmeow', 'text/plain'),
type: 'text/plain'
},
{
id: 'dog.txt',
data: createBlob('woof', 'text/plain'),
type: 'text/plain'
}
]);
```
## putAttachmentBase64()
Same as `putAttachment()` but accepts a plain base64 string instead of a `Blob`.
```ts
const attachment = await doc.putAttachmentBase64({
id: 'cat.txt',
length: 4,
data: 'bWVvdw==',
type: 'text/plain'
});
```
## getAttachment()
Returns an `RxAttachment` by its id. Returns `null` when the attachment does not exist.
```javascript
const attachment = myDocument.getAttachment('cat.jpg');
```
## allAttachments()
Returns an array of all attachments of the `RxDocument`.
```javascript
const attachments = myDocument.allAttachments();
```
## allAttachments$
Gets an Observable which emits a stream of all attachments from the document. Re-emits each time an attachment gets added or removed from the RxDocument.
```javascript
const all = [];
myDocument.allAttachments$.subscribe(
attachments => all = attachments
);
```
## RxAttachment
The attachments of RxDB are represented by the type `RxAttachment` which has the following attributes/methods.
### doc
The `RxDocument` which the attachment is assigned to.
### id
The id as `string` of the attachment.
### type
The type as `string` of the attachment.
### length
The length of the data of the attachment as `number`.
### digest
The hash of the attachments data as `string`.
:::note
The digest is NOT calculated by RxDB, instead it is calculated by the RxStorage. The only guarantee is that the digest will change when the attachments data changes.
:::
### rev
The revision-number of the attachment as `number`.
### remove()
Removes the attachment. Returns a Promise that resolves when done.
```javascript
const attachment = myDocument.getAttachment('cat.jpg');
await attachment.remove();
```
## getData()
Returns a Promise which resolves the attachment's data as `Blob`. (async)
```javascript
const attachment = myDocument.getAttachment('cat.jpg');
const blob = await attachment.getData(); // Blob
```
## getDataBase64()
Returns a Promise which resolves the attachment's data as **base64** `string`.
```javascript
const attachment = myDocument.getAttachment('cat.jpg');
const base64Database = await attachment.getDataBase64(); // 'bWVvdw=='
```
## getStringData()
Returns a Promise which resolves the attachment's data as `string`.
```javascript
const attachment = await myDocument.getAttachment('cat.jpg');
const data = await attachment.getStringData(); // 'meow'
```
## Inline attachments on insert and upsert
Instead of inserting a document first and then calling `putAttachment()` separately, you can include attachments directly in the document data when using `insert()`, `bulkInsert()`, `upsert()`, `bulkUpsert()`, or `incrementalUpsert()`. Provide `_attachments` as an array of `{ id, type, data }` objects.
```javascript
import { createBlob } from 'rxdb';
// insert with inline attachments
const doc = await myCollection.insert({
name: 'foo',
_attachments: [
{
id: 'photo.jpg',
type: 'image/jpeg',
data: myJpegBlob
},
{
id: 'notes.txt',
type: 'text/plain',
data: createBlob('some notes', 'text/plain')
}
]
});
const attachment = doc.getAttachment('photo.jpg');
```
### Upsert behavior with attachments
When upserting a document that already exists, attachments from the new data are **merged** with the document's existing attachments by default. This means existing attachments not mentioned in the upsert data are preserved.
To replace all existing attachments instead, pass `{ deleteExistingAttachments: true }` as the second argument:
```javascript
// Merge (default): keeps existing attachments, adds/updates new ones
const doc = await myCollection.upsert(docData);
// Replace: only the attachments in docData will exist after the upsert
const doc2 = await myCollection.upsert(docData, { deleteExistingAttachments: true });
```
This option works with `upsert()`, `bulkUpsert()`, and `incrementalUpsert()`.
# Attachment compression
Storing many attachments can be a problem when the disc space of the device is exceeded.
Therefore it can make sense to compress the attachments before storing them in the [RxStorage](./rx-storage.md).
With the `attachments-compression` plugin you can compress the attachments data on write and decompress it on reads.
This happens internally and will not change how you use the API. The compression is run with the [Compression Streams API](https://developer.mozilla.org/en-US/docs/Web/API/Compression_Streams_API) which is only supported on [newer browsers](https://caniuse.com/?search=compressionstream).
## MIME-type-aware compression
The compression plugin is MIME-type-aware. It only compresses attachment types that benefit from compression (text, JSON, SVG, etc.) and passes through already-compressed formats (JPEG, PNG, MP4, etc.) as-is. This avoids wasting CPU cycles on files that won't shrink.
A built-in default list of compressible types is used automatically. You can override it with the `compressibleTypes` option in your schema:
```ts
import {
wrappedAttachmentsCompressionStorage
} from 'rxdb/plugins/attachments-compression';
import { getRxStorageIndexedDB } from 'rxdb-premium/plugins/storage-indexeddb';
// create a wrapped storage with attachment-compression.
const storageWithAttachmentsCompression = wrappedAttachmentsCompressionStorage({
storage: getRxStorageIndexedDB()
});
const db = await createRxDatabase({
name: 'mydatabase',
storage: storageWithAttachmentsCompression
});
// set the compression mode at the schema level
const mySchema = {
version: 0,
type: 'object',
properties: {
// ..
},
attachments: {
compression: 'deflate', // <- Specify the compression mode here. OneOf ['deflate', 'gzip']
// Optional: override which MIME types get compressed.
// Supports wildcard prefix matching (e.g. 'text/*' matches 'text/plain', 'text/html', etc.).
// If omitted, a built-in default list of compressible types is used.
compressibleTypes: [
'text/*',
'application/json',
'application/xml',
'image/svg+xml'
// ... add your own patterns
]
}
};
/* ... create your collections as usual and store attachments in them. */
```
The default compressible types include the following MIME type patterns. Binary formats like `image/jpeg`, `image/png`, `video/*`, and `audio/*` are **not** in the default list and will be stored without re-compression.
---
## RxPipeline - Automate Data Flows in RxDB
# RxPipeline
The RxPipeline plugin enables you to run operations depending on writes to a collection.
Whenever a write happens on the source collection of a pipeline, a handler is called to process the writes and run operations on another collection.
You could have a similar behavior by observing the collection stream and process data on emits:
```ts
mySourceCollection.$.subscribe(event => {/* ...process...*/});
```
While this could work in some cases, it causes many problems that are fixed by using the pipeline plugin instead:
- In an RxPipeline, only the [Leading Instance](./leader-election.md) runs the operations. For example when you have multiple browser tabs open, only one will run the processing and when that tab is closed, another tab will become elected leader and continue the pipeline processing.
- On sudden stops and restarts of the JavaScript process, the processing will continue at the correct checkpoint and not miss out any documents even on unexpected crashes.
- Reads/Writes on the destination collection are halted while the pipeline is processing. This ensures your queries only return fully processed documents and no partial results. So when you run a query to the destination collection directly after a write to the source collection, you can be sure your query results are up to date and the pipeline has already been run at the moment the query resolved:
```ts
await mySourceCollection.insert({/* ... */});
/**
* Because our pipeline blocks reads to the destination,
* we know that the result array contains data created
* on top of the previously inserted documents.
*/
const result = myDestinationCollection.find().exec();
```
## Creating a RxPipeline
Pipelines are created on top of a source [RxCollection](./rx-collection.md) and have another `RxCollection` as destination. An identifier is used to identify the state of the pipeline so that different pipelines have a different processing checkpoint state. A plain JavaScript function `handler` is used to process the data of the source collection writes.
```ts
const pipeline = await mySourceCollection.addPipeline({
identifier: 'my-pipeline',
destination: myDestinationCollection,
handler: async (docs) => {
/**
* Here you can process the documents and write to
* the destination collection.
*/
for (const doc of docs) {
await myDestinationCollection.insert({
id: doc.primary,
category: doc.category
});
}
}
});
```
## Use Cases for RxPipeline
The RxPipeline is a handy building block for different features and plugins. You can use it to aggregate data or restructure local data.
### UseCase: Re-Index data that comes from replication
Sometimes you want to [replicate](./replication.md) atomic documents over the wire but locally you want to split these documents for better indexing. For example you replicate email documents that have multiple receivers in a string-array. While string-arrays cannot be indexed, locally you need a way to query for all emails of a given receiver.
To handle this case you can set up a RxPipeline that writes the mapping into a separate collection:
```ts
const pipeline = await emailCollection.addPipeline({
identifier: 'map-email-receivers',
destination: emailByReceiverCollection,
handler: async (docs) => {
for (const doc of docs) {
// remove previous mapping
await emailByReceiverCollection.find({emailId: doc.primary}).remove();
// add new mapping
if(!doc.deleted) {
await emailByReceiverCollection.bulkInsert(
doc.receivers.map(receiver => ({
emailId: doc.primary,
receiver: receiver
}))
);
}
}
}
});
```
With this you can efficiently query for "all emails that a person received" by running:
```ts
const mailIds = await emailByReceiverCollection.find({
receiver: 'foobar@example.com'
}).exec();
```
### UseCase: Fulltext Search
You can utilize the pipeline plugin to index text data for efficient fulltext search.
```ts
const pipeline = await emailCollection.addPipeline({
identifier: 'email-fulltext-search',
destination: mailByWordCollection,
handler: async (docs) => {
for (const doc of docs) {
// remove previous mapping
await mailByWordCollection.find({emailId: doc.primary}).remove();
// add new mapping
if(!doc.deleted) {
const words = doc.text.split(' ');
await mailByWordCollection.bulkInsert(
words.map(word => ({
emailId: doc.primary,
word: word
}))
);
}
}
}
});
```
With this you can efficiently query for "all emails that contain a given word" by running:
```ts
const mailIds = await emailByReceiverCollection.find({word: 'foobar'}).exec();
```
### UseCase: Download data based on source documents
When you have to fetch data for each document of a collection from a server, you can use the pipeline to ensure all documents have their data downloaded and no document is missed.
```ts
const pipeline = await emailCollection.addPipeline({
identifier: 'download-data',
destination: serverDataCollection,
handler: async (docs) => {
for (const doc of docs) {
const response = await fetch('https://example.com/doc/' + doc.primary);
const serverData = await response.json();
await serverDataCollection.upsert({
id: doc.primary,
data: serverData
});
}
}
});
```
## RxPipeline methods
### awaitIdle()
You can await the idleness of a pipeline with `await myRxPipeline.awaitIdle()`. This will await a promise that resolves when the pipeline has processed all documents and is not running anymore.
### close()
`await myRxPipeline.close()` stops the pipeline so that it is no longer doing stuff. This is automatically called when the RxCollection or [RxDatabase](./rx-database.md) of the pipeline is closed.
### remove()
`await myRxPipeline.remove()` removes the pipeline and all metadata which it has stored. Recreating the pipeline afterwards will start processing all source documents from scratch.
## Using RxPipeline correctly
### Pipeline handlers must be idempotent
Because a JavaScript process can exit at any time, like when the user closes a browser tab, the pipeline handler function must be idempotent. This means when it only runs partially and is started again with the same input, it should still end up in the correct result.
### Pipeline handlers must not throw
Pipeline handlers must never throw. If you run operations inside of the handler that might cause errors, you must wrap the handler's code with a `try-catch` by yourself and also handle retries. If your handler throws, your pipeline will be stuck and no longer be usable, which should never happen.
### Be careful when doing http requests in the handler
When you run http requests inside of your handler, you no longer have an [offline first](./offline-first.md) application because reads to the destination collection will be blocked until all handlers have finished. When your client is offline, therefore the collection will be blocked for reads and writes.
### Pipelines temporarily block external reads and writes
While a pipeline is running, **all reads and writes to its destination collection are blocked**. This guarantees that queries never observe partially processed data, but it also means that pipelines can block each other if they interact incorrectly.
Problems occur when multiple pipelines:
- read or write across the same collections, or
- wait for each other using `awaitIdle()` from inside a pipeline handler.
```ts
// Example of a deadlock
// Pipeline A: files β files (reads folders)
const pipelineA = await db.files.addPipeline({
identifier: 'file-path-sync',
destination: db.files,
handler: async (docs) => {
const folders = await folders.find().exec(); // can block
/* ... */
}
});
// Pipeline B: files β folders (waits for A)
await db.folders.addPipeline({
identifier: 'file-count',
destination: db.folders,
handler: async () => {
await pipelineA.awaitIdle(); // β may deadlock
/* ... */
}
});
```
To prevent deadlocks, consider:
- Never call `awaitIdle()` inside a pipeline handler.
- Avoid circular dependencies between pipelines.
- Prefer one-directional data flow.
---
## Signals & Custom Reactivity with RxDB
import {Tabs} from '@site/src/components/tabs';
import {Steps} from '@site/src/components/steps';
# Signals & Co. - Custom reactivity adapters instead of RxJS Observables
RxDB internally uses the [rxjs library](https://rxjs.dev/) for observables and streams. All functionalities of RxDB like [query](./rx-query.md#observe) results or [document fields](./rx-document.md#observe) that expose values that change over time return a rxjs `Observable` that allows you to observe the values and update your UI accordingly depending on the changes to the database state.
However there are many reasons to use other reactivity libraries that use a different datatype to represent changing values. For example when you use **signals** in angular or react, the **template refs** of vue or state libraries like MobX and redux.
RxDB allows you to pass a custom reactivity factory on [RxDatabase](./rx-database.md) creation so that you can easily access values wrapped with your custom datatype in a convenient way.
## Adding a reactivity factory
### Angular
In angular we use [Angular Signals](https://angular.dev/guide/signals) as custom reactivity objects.
#### Import
```ts
import { createReactivityFactory } from 'rxdb/plugins/reactivity-angular';
import { Injectable, inject } from '@angular/core';
```
#### Set the reactivity factory
Set the factory as `reactivity` option when calling `createRxDatabase`.
```ts
const database = await createRxDatabase({
name: 'mydb',
storage: getRxStorageLocalstorage(),
reactivity: createReactivityFactory(inject(Injector))
});
// add collections/sync etc...
```
#### Use the Signal in an Angular component
```ts
import { Component, inject } from '@angular/core';
import { CommonModule } from '@angular/common';
import { DbService } from '../db.service';
@Component({
selector: 'app-todos-list',
standalone: true,
imports: [CommonModule],
template: `
{{ t.title }}
`,
})
export class TodosListComponent {
private dbService = inject(DbService);
// RxDB query - Angular Signal
readonly todosSignal = this.dbService.db.todos.find().$$;
}
```
An example of how signals are used in angular with RxDB, can be found at the [RxDB Angular Example](https://github.com/pubkey/rxdb/blob/master/examples/angular/src/app/components/heroes-list/heroes-list.component.ts#L46)
### React
For React, we use the [Preact Signals](https://preactjs.com/guide/v10/signals/) for custom reactivity.
#### Install Preact Signals
```bash
npm install @preact/signals-core --save
```
#### Import
```ts
import {
PreactSignalsRxReactivityFactory
} from 'rxdb/plugins/reactivity-preact-signals';
```
#### Set the reactivity factory
```ts
const database = await createRxDatabase({
name: 'mydb',
storage: getRxStorageLocalstorage(),
reactivity: PreactSignalsRxReactivityFactory
});
// add collections/sync etc...
```
#### Use the Signal in a React component
```tsx
import { useEffect, useState } from 'preact/hooks';
import { getDatabase } from './db';
export function TodosList() {
const [db, setDb] = useState(null);
useEffect(() => {
getDatabase().then(setDb);
}, []);
if (!db) return null;
// RxQuery -> Preact Signal
const todosSignal = db.todos.find().$$;
return (
{todosSignal.value.map((doc: any) => (
{doc.title}
))}
);
}
```
### Vue
For Vue, we use the [Vue Shallow Refs](https://vuejs.org/api/reactivity-advanced) for custom reactivity.
#### Import
```ts
import { VueRxReactivityFactory } from 'rxdb/plugins/reactivity-vue';
```
#### Set the reactivity factory
```ts
const database = await createRxDatabase({
name: 'mydb',
storage: getRxStorageLocalstorage(),
reactivity: VueRxReactivityFactory
});
// add collections/sync etc...
```
#### Use the Shallow Ref in a Vue component
```html
```
## Accessing custom reactivity objects
All observable data in RxDB is marked by the single dollar sign `$` like [RxCollection](./rx-collection.md).$ for events or `RxDocument.myField$` to get the observable for a document field. To make custom reactivity objects distinguable, they are marked with double-dollar signs `$$` instead. Here are some example on how to get custom reactivity objects from RxDB specific instances:
```ts
// RxDocument
// get signal that represents the document field 'foobar'
const signal = myRxDocument.get$$('foobar');
// same as above
const signal = myRxDocument.foobar$$;
// get signal that represents whole document over time
const signal = myRxDocument.$$;
// get signal that represents the deleted state of the document
const signal = myRxDocument.deleted$$;
```
```ts
// RxQuery
// get signal that represents the query result set over time
const signal = collection.find().$$;
// get signal that represents the query result set over time
const signal = collection.findOne().$$;
```
```ts
// RxLocalDocument
// get signal that represents the whole local document state
const signal = myRxLocalDocument.$$;
// get signal that represents the foobar field
const signal = myRxLocalDocument.get$$('foobar');
```
---
## RxState - Reactive Persistent State with RxDB
RxState is a flexible state library build on top of the [RxDB Database](https://rxdb.info/). While RxDB stores similar documents inside of collections, RxState can store any complex JSON data without having a predefined schema.
The state is automatically persisted through RxDB and states changes are propagated between browser tabs. Even setting up replication is simple by using the RxDB [Replication feature](./replication.md).
## Creating a RxState
A `RxState` instance is created on top of a [RxDatabase](./rx-database.md). The state will automatically be persisted with the [storage](./rx-storage.md) that was used when setting up the RxDatabase. To use it you first have to import the `RxDBStatePlugin` and add it to RxDB with `addRxPlugin()`.
To create a state call the `addState()` method on the database instance. Calling `addState` multiple times will automatically de-duplicated and only create a single RxState object.
```javascript
import { createRxDatabase, addRxPlugin } from 'rxdb';
import { getRxStorageLocalstorage } from 'rxdb/plugins/storage-localstorage';
// first add the RxState plugin to RxDB
import { RxDBStatePlugin } from 'rxdb/plugins/state';
addRxPlugin(RxDBStatePlugin);
const database = await createRxDatabase({
name: 'heroesdb',
storage: getRxStorageLocalstorage(),
});
// create a state instance
const myState = await database.addState();
// you can also create states with a given namespace
const myChildState = await database.addState('myNamepsace');
```
## Writing data and Persistence
Writing data to the state happen by a so called `modifier`. It is a simple JavaScript function that gets the current value as input and returns the new, modified value.
For example to increase the value of `myField` by one, you would use a modifier that increases the current value:
```ts
// initially set value to zero
await myState.set('myField', v => 0);
// increase value by one
await myState.set('myField', v => v + 1);
// update value to be 42
await myState.set('myField', v => 42);
```
The modifier is used instead of a direct assignment to ensure correct behavior when other JavaScript realms write to the state at the same time, like other browser tabs or webworkers. On conflicts, the modifier will just be run again to ensure deterministic and correct behavior. Therefore mutation is `async`, you have to `await` the call to the set function when you care about the moment when the change actually happened.
## Get State Data
The state stored inside of a RxState instance can be seen as a big single JSON object that contains all data.
You can fetch the whole object or partially get a single properties or nested ones.
Fetching data can either happen with the `.get()` method or by accessing the field directly like `myRxState.myField`.
```ts
// get root state data
const val = myState.get();
// get single property
const val = myState.get('myField');
const val = myState.myField;
// get nested property
const val = myState.get('myField.childfield');
const val = myState.myField.childfield;
// get nested array property
const val = myState.get('myArrayField[0].foobar');
const val = myState.myArrayField[0].foobar;
```
## Observability
Instead of fetching the state once, you can also observe the state with either rxjs observables or [custom reactivity handlers](#rxstate-with-signals-and-hooks) like signals or hooks.
Rxjs observables can be created by either using the `.get$()` method or by accessing the top level property suffixed with a dollar sign like `myState.myField$`.
```ts
const observable = myState.get$('myField');
const observable = myState.myField$;
// then you can subscribe to that observable
observable.subscribe(newValue => {
// update the UI
});
```
Subscription works across multiple JavaScript realms like browser tabs or Webworkers.
## RxState with signals and hooks
With the double-dollar sign you can also access [custom reactivity](./reactivity.md) instances like signals or hooks. These are easier to use compared to rxjs, depending on which JavaScript framework you are using.
For example in angular to use signals, you would first add a reactivity factory to your database and then access the signals of the RxState:
```ts
import { RxReactivityFactory, createRxDatabase } from 'rxdb/plugins/core';
import { toSignal } from '@angular/core/rxjs-interop';
const reactivityFactory: RxReactivityFactory = {
fromObservable(obs, initialValue) {
return toSignal(obs, { initialValue });
}
};
const database = await createRxDatabase({
name: 'mydb',
storage: getRxStorageLocalstorage(),
reactivity: reactivityFactory
});
const myState = await database.addState();
const mySignal = myState.get$$('myField');
const mySignal = myState.myField$$;
```
## Cleanup RxState operations
For faster writes, changes to the state are only written as list of operations to disc. After some time you might have too
many operations written which would delay the initial state creation. To automatically merge the state operations into a single operation and clear the old operations, you should add the [Cleanup Plugin](./cleanup.md) before creating the [RxDatabase](./rx-database.md):
```ts
import { addRxPlugin } from 'rxdb';
import { RxDBCleanupPlugin } from 'rxdb/plugins/cleanup';
addRxPlugin(RxDBCleanupPlugin);
```
## Correctness over Performance
RxState is optimized for correctness, not for performance. Compared to other state libraries, RxState directly persists data to storage and ensures write conflicts are handled properly. Other state libraries are handles mainly in-memory and lazily persist to disc without caring about conflicts or multiple browser tabs which can cause problems and hard to reproduce bugs.
RxState still uses RxDB which has a range of [great performing storages](./rx-storage-performance.md) so the write speed is more than sufficient. Also to further improve write performance you can use more RxState instances (with an different namespace) to split writes across multiple storage instances.
Reads happen directly in-memory which makes RxState read performance comparable to other state libraries.
## RxState Replication
Because the state data is stored inside of an internal [RxCollection](./rx-collection.md) you can easily use the [RxDB Replication](./replication.md) to sync data between users or devices of the same user.
For example with the [P2P WebRTC replication](./replication-webrtc.md) you can start the replication on the collection and automatically sync the RxState operations between users directly:
```ts
import {
replicateWebRTC,
getConnectionHandlerSimplePeer
} from 'rxdb/plugins/replication-webrtc';
const database = await createRxDatabase({
name: 'heroesdb',
storage: getRxStorageLocalstorage(),
});
const myState = await database.addState();
const replicationPool = await replicateWebRTC(
{
collection: myState.collection,
topic: 'my-state-replication-pool',
connectionHandlerCreator: getConnectionHandlerSimplePeer({}),
pull: {},
push: {}
}
);
```
---
## Master Local Documents in RxDB
# Local Documents
Local documents are a special class of documents which are used to store local metadata.
They come in handy when you want to store settings or additional data next to your documents.
- Local Documents can exist on a [RxDatabase](./rx-database.md) or [RxCollection](./rx-collection.md).
- Local Document do not have to match the collections schema.
- Local Documents do not get replicated.
- Local Documents will not be found on queries.
- Local Documents can not have attachments.
- Local Documents will not get handled by the [migration-schema](./migration-schema.md).
- The id of a local document has the `maxLength` of `128` characters.
:::note
While local documents can be very useful, in many cases the [RxState](./rx-state.md) API is more convenient.
:::
## Add the local documents plugin
To enable the local documents, you have to add the `local-documents` plugin.
```ts
import { addRxPlugin } from 'rxdb';
import { RxDBLocalDocumentsPlugin } from 'rxdb/plugins/local-documents';
addRxPlugin(RxDBLocalDocumentsPlugin);
```
## Activate the plugin for a RxDatabase or RxCollection
For better performance, the local document plugin does not create a storage for every database or collection that is created.
Instead you have to set `localDocuments: true` when you want to store local documents in the instance.
```js
// activate local documents on a RxDatabase
const myDatabase = await createRxDatabase({
name: 'mydatabase',
storage: getRxStorageLocalstorage(),
localDocuments: true // <- activate this to store local documents in the database
});
myDatabase.addCollections({
messages: {
schema: messageSchema,
localDocuments: true // <- activate this to store local documents in the collection
}
});
```
:::note
If you want to store local documents in a `RxCollection` but **NOT** in the `RxDatabase`, you **MUST NOT** set `localDocuments: true` in the `RxDatabase` because it will only slow down the initial database creation.
:::
## insertLocal()
Creates a local document for the database or collection. Throws if a local document with the same id already exists. Returns a Promise which resolves the new `RxLocalDocument`.
```javascript
const localDoc = await myCollection.insertLocal(
'foobar', // id
{ // data
foo: 'bar'
}
);
// you can also use local-documents on a database
const localDoc = await myDatabase.insertLocal(
'foobar', // id
{ // data
foo: 'bar'
}
);
```
## upsertLocal()
Creates a local document for the database or collection if not exists. Overwrites the if exists. Returns a Promise which resolves the `RxLocalDocument`.
```javascript
const localDoc = await myCollection.upsertLocal(
'foobar', // id
{ // data
foo: 'bar'
}
);
```
## getLocal()
Find a `RxLocalDocument` by its id. Returns a Promise which resolves the `RxLocalDocument` or `null` if not exists.
```javascript
const localDoc = await myCollection.getLocal('foobar');
```
## getLocal$()
Like `getLocal()` but returns an `Observable` that emits the document or `null` if not exists.
```javascript
const subscription = myCollection.getLocal$('foobar').subscribe(documentOrNull => {
console.dir(documentOrNull); // > RxLocalDocument or null
});
```
## RxLocalDocument
A `RxLocalDocument` behaves like a normal `RxDocument`.
```javascript
const localDoc = await myCollection.getLocal('foobar');
// access data
const foo = localDoc.get('foo');
// change data
localDoc.set('foo', 'bar2');
await localDoc.save();
// observe data
localDoc.get$('foo').subscribe(value => { /* .. */ });
// remove it
await localDoc.remove();
```
:::note
Because the local document does not have a schema, accessing the documents data-fields via pseudo-proxy will not work.
:::
```javascript
const foo = localDoc.foo; // undefined
const foo = localDoc.get('foo'); // works!
localDoc.foo = 'bar'; // does not work!
localDoc.set('foo', 'bar'); // works
```
For the usage with typescript, you can have access to the typed data of the document over `toJSON()`
```ts
declare type MyLocalDocumentType = {
foo: string
}
const localDoc = await myCollection.upsertLocal(
'foobar', // id
{ // data
foo: 'bar'
}
);
// typescript will know that foo is a string
const foo: string = localDoc.toJSON().foo;
```
---
## Cleanup
# π§Ή Cleanup
To make the [replication](./replication.md) work, and for other reasons, RxDB has to keep deleted documents in storage so that it can replicate their deletion state.
This ensures that when a client is [offline](./offline-first.md), the deletion state is still known and can be replicated with the backend when the client goes online again.
Keeping too many deleted documents in the storage, can slow down queries or fill up too much disc space.
With the cleanup plugin, RxDB will run cleanup cycles that clean up deleted documents when it can be done safely.
## Installation
```ts
import { addRxPlugin } from 'rxdb';
import { RxDBCleanupPlugin } from 'rxdb/plugins/cleanup';
addRxPlugin(RxDBCleanupPlugin);
```
## Create a database with cleanup options
You can set a specific cleanup policy when a [RxDatabase](./rx-database.md) is created. For most use cases, the defaults should be ok.
```ts
import { createRxDatabase } from 'rxdb';
import { getRxStorageLocalstorage } from 'rxdb/plugins/storage-localstorage';
const db = await createRxDatabase({
name: 'heroesdb',
storage: getRxStorageLocalstorage(),
cleanupPolicy: {
/**
* The minimum time in milliseconds for how long
* a document has to be deleted before it is
* purged by the cleanup.
* [default=one month]
*/
minimumDeletedTime: 1000 * 60 * 60 * 24 * 31, // one month,
/**
* The minimum amount of that that the RxCollection must have existed.
* This ensures that at the initial page load, more important
* tasks are not slowed down because a cleanup process is running.
* [default=60 seconds]
*/
minimumCollectionAge: 1000 * 60, // 60 seconds
/**
* After the initial cleanup is done,
* a new cleanup is started after [runEach] milliseconds
* [default=5 minutes]
*/
runEach: 1000 * 60 * 5, // 5 minutes
/**
* If set to true,
* RxDB will await all running replications
* to not have a replication cycle running.
* This ensures we do not remove deleted documents
* when they might not have already been replicated.
* [default=true]
*/
awaitReplicationsInSync: true,
/**
* If true, it will only start the cleanup
* when the current instance is also the leader.
* This ensures that when RxDB is used in multiInstance mode,
* only one instance will start the cleanup.
* [default=true]
*/
waitForLeadership: true
}
});
```
## Calling cleanup manually
You can manually run a cleanup per collection by calling [RxCollection](./rx-collection.md).cleanup().
```ts
/**
* Manually run the cleanup with the
* minimumDeletedTime from the cleanupPolicy.
*/
await myRxCollection.cleanup();
/**
* Overwrite the minimumDeletedTime
* be setting it explicitly (time in milliseconds)
*/
await myRxCollection.cleanup(1000);
/**
* Purge all deleted documents no
* matter when they where deleted
* by setting minimumDeletedTime to zero.
*/
await myRxCollection.cleanup(0);
```
## Using the cleanup plugin to empty a collection
When you have a collection with documents and you want to empty it by purging all documents, the recommended way is to call `myRxCollection.remove()`. However, this will destroy the JavaScript class of the collection and stop all listeners and observables.
Sometimes the better option might be to manually delete all documents and then use the cleanup plugin to purge the deleted documents:
```ts
// delete all documents
await myRxCollection.find().remove();
// purge all deleted documents
await myRxCollection.cleanup(0);
```
## FAQ
When does the cleanup run
The cleanup cycles are optimized to run only when the database is idle and it is unlikely that another database interaction performance will be decreased in the meantime. For example, by default, the cleanup does not run in the first 60 seconds of the creation of a collection to ensure the initial page load of your website does not slow down. Also, we use mechanisms like the `requestIdleCallback()` API to improve the correct timing of the cleanup cycle.
---
## Backup
# π₯ Backup Plugin
With the backup plugin you can write the current database state and ongoing changes into folders on the filesystem.
The files are written in plain json together with their attachments so that you can read them out with any software or tools, without being bound to RxDB.
This is useful to:
- Consume the database content with other software that cannot replicate with RxDB
- Write a backup of the database to a remote server by mounting the backup folder on the other server.
The backup plugin works only in [node.js](./nodejs-database.md), not in a browser. It is intended to have a backup strategy when using RxDB on the server side like with the [RxServer](./rx-server.md). To run backups on the client side, you should use one of the [replication](./replication.md) plugins instead.
## Installation
```javascript
import { addRxPlugin } from 'rxdb';
import { RxDBBackupPlugin } from 'rxdb/plugins/backup';
addRxPlugin(RxDBBackupPlugin);
```
## one-time backup
Write the whole database to the filesystem **once**.
When called multiple times, it will continue from the last checkpoint and not start all over again.
```javascript
const backupOptions = {
// if false, a one-time backup will be written
live: false,
// the folder where the backup will be stored
directory: '/my-backup-folder/',
// if true, attachments will also be saved
attachments: true
}
const backupState = myDatabase.backup(backupOptions);
await backupState.awaitInitialBackup();
// call again to run from the last checkpoint
const backupState2 = myDatabase.backup(backupOptions);
await backupState2.awaitInitialBackup();
```
## live backup
When `live: true` is set, the backup will write all ongoing changes to the backup directory.
```javascript
const backupOptions = {
// set live: true to have an ongoing backup
live: true,
directory: '/my-backup-folder/',
attachments: true
}
const backupState = myDatabase.backup(backupOptions);
// you can still await the initial backup write, but further changes will still be processed.
await backupState.awaitInitialBackup();
```
## writeEvents$
You can listen to the `writeEvents$` Observable to get notified about written backup files.
```javascript
const backupOptions = {
live: false,
directory: '/my-backup-folder/',
attachments: true
}
const backupState = myDatabase.backup(backupOptions);
const subscription = backupState.writeEvents$.subscribe(writeEvent => console.dir(writeEvent));
/*
> {
collectionName: 'humans',
documentId: 'foobar',
files: [
'/my-backup-folder/foobar/document.json'
],
deleted: false
}
*/
```
## Limitations
- It is currently not possible to import from a written backup. If you need this functionality, please make a pull request.
---
## Leader Election
# Leader-Election
RxDB comes with a leader-election which elects a leading instance between different instances in the same javascript runtime.
Before you read this, please check out on how many of your open browser-tabs you have opened the same website more than once. Count them, I will wait..
So if you would now inspect the traffic that these open tabs produce, you can see that many of them send exact the same data over wire for every tab. No matter if the data is sent with an open websocket or by polling.
## Use-case-example
Imagine we have a website which displays the current temperature of the visitors location in various charts, numbers or heatmaps. To always display the live-data, the website opens a [websocket](./articles/websockets-sse-polling-webrtc-webtransport.md) to our API-Server which sends the current temperature every 10 seconds. Using the way most sites are currently build, we can now open it in 5 browser-tabs and it will open 5 websockets which send data 6*5=30 times per minute. This will not only waste the power of your clients device, but also wastes your api-servers resources by opening redundant connections.
## Solution
The solution to this redundancy is the usage of a [leader-election](https://en.wikipedia.org/wiki/Leader_election)-algorithm which makes sure that always exactly one tab is managing the remote-data-access. The managing tab is the elected leader and stays leader until it is closed. No matter how many tabs are opened or closed, there must be always exactly **one** leader.
You could now start implementing a messaging-system between your browser-tabs, hand out which one is leader, solve conflicts and reassign a new leader when the old one 'dies'.
Or just use RxDB which does all these things for you.
## Add the leader election plugin
To enable the leader election, you have to add the `leader-election` plugin.
```javascript
import { addRxPlugin } from 'rxdb';
import { RxDBLeaderElectionPlugin } from 'rxdb/plugins/leader-election';
addRxPlugin(RxDBLeaderElectionPlugin);
```
## Code-example
To make it easy, here is an example where the temperature is pulled every ten seconds and saved to a collection. The pulling starts at the moment where the opened tab becomes the leader.
```javascript
import { getRxStorageLocalstorage } from 'rxdb/plugins/storage-localstorage';
const db = await createRxDatabase({
name: 'weatherDB',
storage: getRxStorageLocalstorage(),
password: 'myPassword',
multiInstance: true
});
await db.addCollections({
temperature: {
schema: mySchema
}
});
db.waitForLeadership()
.then(() => {
console.log('Long lives the king!'); // <- runs when db becomes leader
setInterval(async () => {
const temp = await fetch('https://example.com/api/temp/');
db.temperature.insert({
degrees: temp,
time: new Date().getTime()
});
}, 1000 * 10);
});
```
## Handle Duplicate Leaders
On rare occasions, it can happen that [more than one leader](https://github.com/pubkey/broadcast-channel/blob/master/.github/README.md#handle-duplicate-leaders) is elected. This can happen when the CPU is on 100% or for any other reason the JavaScript process is fully blocked for a long time.
For most cases this is not really a problem because on duplicate leaders, both browser tabs replicate with the same backend anyways.
To handle the duplicate leader event, you can access the leader elector and set a handler:
```ts
import {
getLeaderElectorByBroadcastChannel
} from 'rxdb/plugins/leader-election';
const leaderElector = getLeaderElectorByBroadcastChannel(broadcastChannel);
leaderElector.onduplicate = async () => {
// Duplicate leader detected -> reload the page.
location.reload();
}
```
## Live-Example
In this example the leader is marked with the crown β
## Try it out
Run the [angular-example](https://github.com/pubkey/rxdb/tree/master/examples/angular) where the leading tab is marked with a crown on the top-right-corner.
## Notice
The leader election is implemented via the [broadcast-channel module](https://github.com/pubkey/broadcast-channel#using-the-leaderelection).
The leader is elected between different processes on the same javascript-runtime. Like multiple tabs in the same browser or multiple Node.js processes on the same machine. It will not run between different replicated instances.
---
## Streamlined RxDB Middleware
# Middleware
RxDB middleware-hooks (also called pre and post hooks) are functions which are passed control during execution of asynchronous functions.
The hooks are specified on RxCollection-level and help to create a clear what-happens-when-structure of your code.
Hooks can be defined to run **parallel** or **in series** one after another.
Hooks can be **synchronous** or **asynchronous** when they return a `Promise`.
To stop the operation at a specific hook, throw an error.
## List
RxDB supports the following hooks:
- preInsert
- postInsert
- preSave
- postSave
- preRemove
- postRemove
- postCreate
### Why is there no validate-hook?
Different to mongoose, the validation on document-data is running on the field-level for every change to a document.
This means if you set the value `lastName` of a RxDocument, then the validation will only run on the changed field, not the whole document.
Therefore it is not useful to have validate-hooks when a document is written to the database.
## Use Cases
Middleware is useful for atomizing model logic and avoiding nested blocks of async code.
Here are some other ideas:
- complex validation
- removing dependent documents
- asynchronous defaults
- asynchronous tasks that a certain action triggers
- triggering custom events
- notifications
## Usage
All hooks have the plain data as first parameter, and all but `preInsert` also have the `RxDocument`-instance as second parameter. If you want to modify the data in the hook, change attributes of the first parameter.
All hook functions are also `this`-bound to the `RxCollection`-instance.
### Insert
An insert-hook receives the data-object of the new document.
#### lifecycle
- RxCollection.insert is called
- preInsert series-hooks
- preInsert parallel-hooks
- schema validation runs
- new document is written to database
- postInsert series-hooks
- postInsert parallel-hooks
- event is emitted to [RxDatabase](./rx-database.md) and [RxCollection](./rx-collection.md)
#### preInsert
```js
// series
myCollection.preInsert(function(plainData){
// set age to 50 before saving
plainData.age = 50;
}, false);
// parallel
myCollection.preInsert(function(plainData){
}, true);
// async
myCollection.preInsert(function(plainData){
return new Promise(res => setTimeout(res, 100));
}, false);
// stop the insert-operation
myCollection.preInsert(function(plainData){
throw new Error('stop');
}, false);
```
#### postInsert
```js
// series
myCollection.postInsert(function(plainData, rxDocument){
}, false);
// parallel
myCollection.postInsert(function(plainData, rxDocument){
}, true);
// async
myCollection.postInsert(function(plainData, rxDocument){
return new Promise(res => setTimeout(res, 100));
}, false);
```
### Save
A save-hook receives the document which is saved.
#### lifecycle
- RxDocument.save is called
- preSave series-hooks
- preSave parallel-hooks
- updated document is written to database
- postSave series-hooks
- postSave parallel-hooks
- event is emitted to RxDatabase and RxCollection
#### preSave
```js
// series
myCollection.preSave(function(plainData, rxDocument){
// modify anyField before saving
plainData.anyField = 'anyValue';
}, false);
// parallel
myCollection.preSave(function(plainData, rxDocument){
}, true);
// async
myCollection.preSave(function(plainData, rxDocument){
return new Promise(res => setTimeout(res, 100));
}, false);
// stop the save-operation
myCollection.preSave(function(plainData, rxDocument){
throw new Error('stop');
}, false);
```
#### postSave
```js
// series
myCollection.postSave(function(plainData, rxDocument){
}, false);
// parallel
myCollection.postSave(function(plainData, rxDocument){
}, true);
// async
myCollection.postSave(function(plainData, rxDocument){
return new Promise(res => setTimeout(res, 100));
}, false);
```
### Remove
A remove-hook receives the document which is removed.
#### lifecycle
- RxDocument.remove is called
- preRemove series-hooks
- preRemove parallel-hooks
- deleted document is written to database
- postRemove series-hooks
- postRemove parallel-hooks
- event is emitted to RxDatabase and RxCollection
#### preRemove
```js
// series
myCollection.preRemove(function(plainData, rxDocument){
}, false);
// parallel
myCollection.preRemove(function(plainData, rxDocument){
}, true);
// async
myCollection.preRemove(function(plainData, rxDocument){
return new Promise(res => setTimeout(res, 100));
}, false);
// stop the remove-operation
myCollection.preRemove(function(plainData, rxDocument){
throw new Error('stop');
}, false);
```
#### postRemove
```js
// series
myCollection.postRemove(function(plainData, rxDocument){
}, false);
// parallel
myCollection.postRemove(function(plainData, rxDocument){
}, true);
// async
myCollection.postRemove(function(plainData, rxDocument){
return new Promise(res => setTimeout(res, 100));
}, false);
```
### postCreate
This hook is called whenever a `RxDocument` is constructed.
You can use `postCreate` to modify every RxDocument-instance of the collection.
This adds a flexible way to add specific behavior to every document. You can also use it to add custom getter/setter to documents. PostCreate-hooks cannot be **asynchronous**.
```js
myCollection.postCreate(function(plainData, rxDocument){
Object.defineProperty(rxDocument, 'myField', {
get: () => 'foobar',
});
});
const doc = await myCollection.findOne().exec();
console.log(doc.myField);
// 'foobar'
```
:::note
This hook does not run on already created or cached documents. Make sure to add `postCreate`-hooks before interacting with the collection.
:::
---
## CRDT - Conflict-free replicated data type Database
# RxDB CRDT Plugin
Whenever there are multiple instances in a distributed system, data writes can cause conflicts. Two different clients could do a write to the same document at the same time or while they are both offline. When the clients replicate the document state with the server, a conflict emerges that must be resolved by the system.
In [RxDB](./), conflicts are normally resolved by setting a `conflictHandler` when creating a collection. The conflict handler is a JavaScript function that gets the two conflicting states of the same document and it will return the resolved document state.
The [default conflict handler](./replication.md#conflict-handling) will always drop the fork state and use the master state to ensure that clients that have been offline for a long time, do not overwrite other clients changes when they go online again.
With CRDTs (short for [Conflict-free replicated data type](https://en.wikipedia.org/wiki/Conflict-free_replicated_data_type)), all document
writes are represented as CRDT operations in plain JSON. The CRDT operations are stored together with the document and each time a conflict arises, the CRDT conflict handler will automatically merge the operations in a deterministic way. Using CRDTs is an easy way to "magically" handle all conflict problems in your application by storing the deltas of writes together with the document data.
## RxDB CRDT operations
In RxDB, a CRDT operation is defined with [NoSQL](./articles/in-memory-nosql-database.md) update operators, like you might know them from [MongoDB update operations](https://www.mongodb.com/docs/manual/reference/operator/update/) or the [RxDB update plugin](./rx-document.md#update).
To run the operators, RxDB uses the [mingo library](https://github.com/kofrasa/mingo#updating-documents).
A CRDT operator example:
```js
const myCRDTOperation = {
// increment the points field by +1
$inc: {
points: 1
},
// set the modified field to true
$set: {
modified: true
}
};
```
### Operators
At the moment, not all possible operators are implemented in [mingo](https://github.com/kofrasa/mingo#updating-documents), if you need additional ones, you should make a pull request there.
The following operators can be used at this point in time:
- `$min`
- `$max`
- `$inc`
- `$set`
- `$unset`
- `$push`
- `$addToSet`
- `$pop`
- `$pullAll`
- `$rename`
For the exact definition on how each operator behaves, check out the [MongoDB documentation on update operators](https://www.mongodb.com/docs/manual/reference/operator/update/).
## Installation
To use CRDTs with RxDB, you need the following:
- Add the CRDT plugin via `addRxPlugin`.
- Add a field to your schema that defines where to store the CRDT operations via `getCRDTSchemaPart()`
- Set the `crdt` options in your schema.
- Do **NOT** set a custom conflict handler, the plugin will use its own one.
```ts
// import the relevant parts from the CRDT plugin
import {
getCRDTSchemaPart,
RxDBcrdtPlugin
} from 'rxdb/plugins/crdt';
// add the CRDT plugin to RxDB
import { addRxPlugin } from 'rxdb';
addRxPlugin(RxDBcrdtPlugin);
// create a database
import { createRxDatabase } from 'rxdb';
import { getRxStorageLocalstorage } from 'rxdb/plugins/storage-localstorage';
const myDatabase = await createRxDatabase({
name: 'heroesdb',
storage: getRxStorageLocalstorage()
});
// create a schema with the CRDT options
const mySchema = {
version: 0,
primaryKey: 'id',
type: 'object',
properties: {
id: {
type: 'string',
maxLength: 100
},
points: {
type: 'number',
maximum: 100,
minimum: 0
},
crdts: getCRDTSchemaPart() // use this field to store the CRDT operations
},
required: ['id', 'points'],
crdt: { // CRDT options
field: 'crdts'
}
}
// add a collection
await db.addCollections({
users: {
schema: mySchema
}
});
// insert a document
const myDocument = await db.users.insert({id: 'alice', points: 0});
// run a CRDT operation that increments the 'points' by one
await myDocument.updateCRDT({
ifMatch: {
$inc: {
points: 1
}
}
});
```
## Conditional CRDT operations
By default, all CRDTs operations will be run to build the current document state. But in many cases, more granular operations are required to better reflect the desired business logic. For these cases, conditional CRDTs can be used.
For example if you have a field `points` with a `maximum` of `100`, you might want to only run the `$inc` operation, if the `points` value is less than `100`.
In an conditional CRDT, you can specify a `selector` and the operation sets `ifMatch` and `ifNotMatch`. At each time the CRDT is applied to the document state, first the selector will run and evaluate which operations path must be used.
```ts
await myDocument.updateCRDT({
// only if the selector matches, the ifMatch operation will run
selector: {
age: {
$lt: 100
}
},
// an operation that runs if the selector matches
ifMatch: {
$inc: {
points: 1
}
},
// if the selector does NOT match, you could run a different operation instead
ifNotMatch: {
// ...
}
});
```
## Running multiples operations at once
By default, one CRDT operation is applied to the document in a single database write.
To represent more complex logic chains, it might make sense to use multiple CRDTs and write them at once inside of a single atomic document write.
For these cases, the `updateCRDT()` method allows to pass an array of operations.
```ts
await myDocument.updateCRDT([
{
selector: { /** ... **/ },
ifMatch: { /** ... **/ }
},
{
selector: { /** ... **/ },
ifMatch: { /** ... **/ }
},
{
selector: { /** ... **/ },
ifMatch: { /** ... **/ }
},
{
selector: { /** ... **/ },
ifMatch: { /** ... **/ }
}
]);
```
## CRDTs on inserts
When CRDTs are enabled with the plugin, all insert operations are automatically mapped as CRDT operation with the `$set` operator.
```ts
// Calling RxCollection.insert()
await myRxCollection.insert({
id: 'foo',
points: 1
});
// is exactly equal to calling insertCRDT()
await myRxCollection.insertCRDT({
ifMatch: {
$set: {
id: 'foo',
points: 1
}
}
});
```
When the same document is inserted in multiple client instances and then replicated, a conflict will emerge and the insert-CRDTs will overwrite each other in a deterministic order.
You can use `insertCRDT()` to make conditional insert operations with any logic. To check for the previous existence of a document, use the `$exists` query operation on the primary key of the document.
```ts
await myRxCollection.insertCRDT({
selector: {
// only run if the document did not exist before.
id: { $exists: false }
},
ifMatch: {
// if the document did not exist, insert it
$set: {
id: 'foo',
points: 1
}
},
ifNotMatch: {
// if document existed already, increment the points by +1
$inc: {
points: 1
}
}
});
```
## Deleting documents
You can delete a document with a CRDT operation by setting `_deleted` to true. Calling `RxDocument.remove()` will do exactly the same when CRDTs are activated.
```ts
await doc.updateCRDT({
ifMatch: {
$set: {
_deleted: true
}
}
});
// OR
await doc.remove();
```
## CRDTs with replication
CRDT operations are stored inside of a special field besides your 'normal' document fields.
When replicating document data with the [RxDB replication](./replication.md) or the [CouchDB replication](./replication-couchdb.md) or even any custom replication, the CRDT operations must be replicated together with the document data as if they would be 'normal' a document property.
When any instances makes a write to the document, it is required to update the CRDT operations accordingly. For example if your custom backend updates a document, it must also do that by adding a CRDT operation. In [dev-mode](./dev-mode.md) RxDB will refuse to store any document data where the document properties do not match the result of the CRDT operations.
## Why not automerge.js or yjs?
There are already CRDT libraries out there that have been considered to be used with RxDB. The biggest ones are [automerge](https://github.com/automerge/automerge) and [yjs](https://github.com/yjs/yjs). The decision was made to not use these but instead go for a more NoSQL way of designing the CRDT format because:
- Users do not have to learn a new syntax but instead can use the NoSQL query operations which they already know to manipulate the JSON data of a document.
- RxDB is often used to [replicate](./replication.md) data with any custom backend on an already existing infrastructure. Using NoSQL operators instead of binary data in CRDTs, makes it easy to implement the exact same logic on these backends so that the backend can also do document writes and still be compliant to the RxDB CRDT plugin.
So instead of using YJS or Automerge with a database, you can use RxDB with the CRDT plugin to have a more database specific CRDT approach. This gives you additional features for free such as [schema validation](./schema-validation.md) or [data migration](./migration-schema.md).
## When to not use CRDTs
CRDT can only be use when your business logic allows to represent document changes via static json operators.
If you can have cases where user interaction is required to correctly merge conflicting document states, you cannot use CRDTs for that.
Also when CRDTs are used, it is no longer allowed to do non-CRDT writes to the document properties.
## CRDT Alternative
While the CRDT plugin can automatically merge concurrent document updates, it is not the only way to resolve conflicts in RxDB.
An alternative approach to CRDT is to use RxDB's built-in [conflict handling system](./transactions-conflicts-revisions.md).
> Why use conflict handlers instead of CRDT?
Conflict handlers offer a **simpler and more flexible** way to manage data conflicts. Instead of encoding changes as CRDT operations, you define how RxDB should decide which document version "wins" with plain JavaScript code. This approach is easier to reason about because it works directly with your domain logic. For example, you can compare timestamps, prioritize certain fields, or even involve user interaction to resolve conflicts.
Conflict handlers are:
* **Easier to understand**: you work with plain document states instead of CRDT operations.
* **Fully customizable**: you can define any merge strategy, from simple last-write-wins to complex rule-based logic.
* **Compatible with all data types**: unlike CRDTs, which are best suited for numeric or set-based updates.
* **Transparent**: you always know which state is being written and why.
### Downsides of CRDTs
CRDTs are powerful for automatic conflict-free merging, but they also come with trade-offs:
* **Higher conceptual complexity**: CRDTs require understanding of operation semantics, version vectors, and merge determinism.
* **Limited flexibility**: you can only express changes that fit the supported JSON-style update operators.
* **Difficult debugging**: when merges don't behave as expected, it can be hard to trace the sequence of CRDT operations that led to a state.
* **Overhead for simple cases**: if your data rarely conflicts or needs human oversight, using CRDTs can add unnecessary complexity.
### When to choose conflict handlers
Use conflict handlers as CRDT alternative if:
* You want full control over merge logic.
* Your data model includes contextual or user-specific decisions.
* You prefer a straightforward, rule-based resolution system over automatic merges.
Use CRDTs if:
* Your app performs frequent offline writes that can be merged deterministically.
* Your data can be represented as additive, numeric, or array-based updates.
* You want minimal manual intervention during replication.
Both methods are first-class citizens in RxDB. CRDTs focus on **automatic, deterministic merging**, while conflict handlers emphasize **clarity, flexibility, and control**.
### Example: merging different fields with conflict handlers instead of CRDT
For example, imagine two users edit different fields of the same document at the same time. One updates a `name`, the other updates a `score`. A custom conflict handler can merge both changes so no data is lost:
```ts
const mergeFieldsHandler = {
isEqual: (a, b) => JSON.stringify(a) === JSON.stringify(b),
resolve: (input) => {
return {
...input.realMasterState,
name: input.newDocumentState.name ?? input.realMasterState.name,
score: Math.max(input.newDocumentState.score, input.realMasterState.score)
};
}
};
```
In this example, if the two versions change different properties, the final merged document includes both updates. This kind of logic is often easier to reason about than designing equivalent CRDT operations.
## FAQ
Are you looking for a distributed database with conflict-free replication?
RxDB provides a distributed database with conflict-free replication. You build offline-first applications using local data storage. RxDB synchronizes data across multiple client devices. The CRDT plugin resolves data conflicts automatically during replication. You maintain continuous data consistency without manual merge logic.
---
## Populate and Link Docs in RxDB
# Population
There are no joins in RxDB but sometimes we still want references to documents in other collections. This is where population comes in. You can specify a relation from one [RxDocument](./rx-document.md) to another [RxDocument](./rx-document.md) in the same or another [RxCollection](./rx-collection.md) of the same database.
Then you can get the referenced document with the population getter.
This works exactly like population with [mongoose](http://mongoosejs.com/docs/populate.html).
## Schema with ref
The `ref` keyword in properties describes to which collection the field value belongs to (has a relationship).
```javascript
export const refHuman = {
title: 'human related to other human',
version: 0,
primaryKey: 'name',
properties: {
name: {
type: 'string'
},
bestFriend: {
ref: 'human', // refers to collection human
type: 'string' // ref-values must always be string or ['string', 'null'] (primary of foreign RxDocument)
}
}
};
```
You can also have a one-to-many reference by using a string array.
```js
export const schemaWithOneToManyReference = {
version: 0,
primaryKey: 'name',
type: 'object',
properties: {
name: {
type: 'string'
},
friends: {
type: 'array',
ref: 'human',
items: {
type: 'string'
}
}
}
};
```
## populate()
### via method
To get the referred RxDocument, you can use the `populate()` method.
It takes the field path as attribute and returns a Promise which resolves to the foreign document or null if not found.
```javascript
await humansCollection.insert({
name: 'Alice',
bestFriend: 'Carol'
});
await humansCollection.insert({
name: 'Bob',
bestFriend: 'Alice'
});
const doc = await humansCollection.findOne('Bob').exec();
const bestFriend = await doc.populate('bestFriend');
console.dir(bestFriend); //> RxDocument[Alice]
```
### via getter
You can also get the populated RxDocument with the direct getter. To do this, you have to add an underscore suffix `_` to the field name.
This also works on nested values.
```javascript
await humansCollection.insert({
name: 'Alice',
bestFriend: 'Carol'
});
await humansCollection.insert({
name: 'Bob',
bestFriend: 'Alice'
});
const doc = await humansCollection.findOne('Bob').exec();
const bestFriend = await doc.bestFriend_; // notice the underscore `_`
console.dir(bestFriend); //> RxDocument[Alice]
```
## Example with nested reference
```javascript
const myCollection = await myDatabase.addCollections({
human: {
schema: {
version: 0,
type: 'object',
properties: {
name: {
type: 'string'
},
family: {
type: 'object',
properties: {
mother: {
type: 'string',
ref: 'human'
}
}
}
}
}
}
});
/**
* We assume myDocument is a document from the collection
*/
const mother = await myDocument.family.mother_;
console.dir(mother); //> RxDocument
```
## Example with array
```javascript
const myCollection = await myDatabase.addCollections({
human: {
schema: {
version: 0,
type: 'object',
properties: {
name: {
type: 'string'
},
friends: {
type: 'array',
ref: 'human',
items: {
type: 'string'
}
}
}
}
}
});
//[insert other humans here]
await myCollection.insert({
name: 'Alice',
friends: [
'Bob',
'Carol',
'Dave'
]
});
const doc = await humansCollection.findOne('Alice').exec();
const friends = await doc.friends_;
console.dir(friends); //> Array.
```
---
## ORM
# Object-Data-Relational-Mapping
Like [mongoose](http://mongoosejs.com/docs/guide.html#methods), RxDB has ORM capabilities which can be used to add specific behavior to documents and collections.
## statics
Statics are defined collection-wide and can be called on the collection.
### Add statics to a collection
To add static functions, pass a `statics` object when you create your collection. The object contains functions, mapped to their function names.
```javascript
const heroes = await myDatabase.addCollections({
heroes: {
schema: mySchema,
statics: {
scream: function(){
return 'AAAH!!';
}
}
}
});
console.log(heroes.scream());
// 'AAAH!!'
```
You can also use the `this` keyword which resolves to the collection:
```javascript
const heroes = await myDatabase.addCollections({
heroes: {
schema: mySchema,
statics: {
whoAmI: function(){
return this.name;
}
}
}
});
console.log(heroes.whoAmI());
// 'heroes'
```
## Instance Methods
Instance methods are defined collection-wide. They can be called on the [RxDocuments](./rx-document.md) of the collection.
### Add instance methods to a collection
```javascript
const heroes = await myDatabase.addCollections({
heroes: {
schema: mySchema,
methods: {
scream: function(){
return 'AAAH!!';
}
}
}
});
const doc = await heroes.findOne().exec();
console.log(doc.scream());
// 'AAAH!!'
```
Here you can also use the `this` keyword:
```javascript
const heroes = await myDatabase.addCollections({
heroes: {
schema: mySchema,
methods: {
whoAmI: function(){
return 'I am ' + this.name + '!!';
}
}
}
});
await heroes.insert({
name: 'Skeletor'
});
const doc = await heroes.findOne().exec();
console.log(doc.whoAmI());
// 'I am Skeletor!!'
```
## attachment-methods
Attachment methods are defined collection-wide. They can be called on the [RxAttachments](./rx-attachment.md) of the RxDocuments of the collection.
```javascript
const heroes = await myDatabase.addCollections({
heroes: {
schema: mySchema,
attachments: {
scream: function(){
return 'AAAH!!';
}
}
}
});
const doc = await heroes.findOne().exec();
const attachment = await doc.putAttachment({
id: 'cat.txt',
data: 'meow I am a kitty',
type: 'text/plain'
});
console.log(attachment.scream());
// 'AAAH!!'
```
---
## Fulltext Search π
# Fulltext Search
To run fulltext search queries on the local data, RxDB has a fulltext search plugin based on [flexsearch](https://github.com/nextapps-de/flexsearch) and [RxPipeline](./rx-pipeline.md). On each write to a given source [RxCollection](./rx-collection.md), an indexer is running to map the written document data into a fulltext search index.
The index can then be queried efficiently with complex fulltext search operations.
## Benefits of using a local fulltext search
1. Efficient Search and Indexing
The plugin utilizes the [FlexSearch library](https://github.com/nextapps-de/flexsearch), known for its speed and memory efficiency. This ensures that search operations are performed quickly, even with large datasets. The search engine can handle multi-field queries, partial matching, and complex search operations, providing users with highly relevant results.
2. Local Data Indexing
With the plugin, all search operations are performed on the local data stored within the RxDB collections. This means that users can execute fulltext search queries without the need for an external server or database, which is especially beneficial for offline-first applications. The local indexing ensures that search queries are executed quickly, reducing the latency typically associated with remote database queries. Also when used in multiple browser tabs, it is ensured that through [Leader Election](./leader-election.md), only exactly one tabs is doing the work of indexing without having an overhead in the other browser tabs.
3. Real-time Indexing
The plugin integrates seamlessly with RxDB's reactive nature. Every time a document is written to an [RxCollection](./rx-collection.md), an indexer updates the fulltext search index in real-time. This ensures that search results are always up-to-date, reflecting the most current state of the data without requiring manual reindexing.
4. Persistent indexing
The fulltext search index is efficiently persisted within the [RxCollection](./rx-collection.md), ensuring that the index remains intact across app restarts. When documents are added or updated in the collection, the index is incrementally updated in real-time, meaning only the changes are processed rather than reindexing the entire dataset. This incremental approach not only optimizes performance but also ensures that subsequent app launches are quick, as there's no need to reindex all the data from scratch, making the search feature both reliable and fast from the moment the app starts. When using an [encrypted storage](./encryption.md) the index itself and incremental updates to it are stored fully encrypted and are only decrypted in-memory.
5. Complex Query Support
The FlexSearch-based plugin allows for [sophisticated search queries](https://github.com/nextapps-de/flexsearch?tab=readme-ov-file#index.search), including multi-term and contextual searches. Users can perform complex searches that go beyond simple keyword matching, enabling more advanced use cases like searching for documents with specific phrases, relevance-based sorting, or even phonetic matching.
6. Offline-First Support and Privacy
As RxDB is designed with [offline-first applications](./offline-first.md) in mind, the fulltext search plugin supports this paradigm by ensuring that all search operations can be performed offline. This is crucial for applications that need to function in environments with intermittent or no internet connectivity, offering users a consistent and reliable search experience with [zero latency](./articles/zero-latency-local-first.md).
## Using the RxDB Fulltext Search
The flexsearch search is a [RxDB Premium Package π](/premium/) which must be purchased and imported from the `rxdb-premium` npm package.
Step 1: Add the `RxDBFlexSearchPlugin` to RxDB.
```ts
import { RxDBFlexSearchPlugin } from 'rxdb-premium/plugins/flexsearch';
import { addRxPlugin } from 'rxdb/plugins/core';
addRxPlugin(RxDBFlexSearchPlugin);
```
Step 2: Create a `RxFulltextSearch` instance on top of a collection with the `addFulltextSearch()` function.
```ts
import { addFulltextSearch } from 'rxdb-premium/plugins/flexsearch';
const flexSearch = await addFulltextSearch({
// unique identifier. Used to store metadata and continue indexing on restarts/reloads.
identifier: 'my-search',
// The source collection on whose documents the search is based on
collection: myRxCollection,
/**
* Transforms the document data to a given searchable string.
* This can be done by returning a single string property of the document
* or even by concatenating and transforming multiple fields like:
* doc => doc.firstName + ' ' + doc.lastName
*/
docToString: doc => doc.firstName,
/**
* (Optional)
* Amount of documents to index at once.
* See https://rxdb.info/rx-pipeline.html
*/
batchSize: number;
/**
* (Optional)
* lazy: Initialize the in memory fulltext index at the first search query.
* instant: Directly initialize so that the index is already there on the first query.
* Default: 'instant'
*/
initialization: 'instant',
/**
* (Optional)
* @link https://github.com/nextapps-de/flexsearch#index-options
*/
indexOptions: {},
});
```
Step 3: Run a search operation:
```ts
// find all documents whose searchstring contains "foobar"
const foundDocuments = await flexSearch.find('foobar');
/**
* You can also use search options as second parameter
* @link https://github.com/nextapps-de/flexsearch#search-options
*/
const foundDocuments = await flexSearch.find('foobar', { limit: 10 });
```
---
## Optimize Client-Side Queries with RxDB
# Query Optimizer
The query optimizer can be used to determine which index is the best to use for a given query.
Because RxDB is used in client side applications, it cannot do any background checks or measurements to optimize the query plan because that would cause significant performance problems.
:::note
The query optimizer is part of the [RxDB Premium π](/premium/) plugin that must be purchased. It is not part of the default RxDB module.
:::
## Usage
```ts
import {
findBestIndex
} from 'rxdb-premium/plugins/query-optimizer';
import {
getRxStorageIndexedDB
} from 'rxdb-premium/plugins/indexeddb';
const bestIndexes = await findBestIndex({
schema: myRxJsonSchema, // see Schema Validation
/**
* In this example we use the IndexedDB RxStorage,
* but any other storage can be used for testing.
*/
storage: getRxStorageIndexedDB(),
/**
* Multiple queries can be optimized at the same time
* which decreases the overall runtime.
*/
queries: {
/**
* Queries can be mapped by a query id,
* here we use myFirstQuery as query id.
*/
myFirstQuery: {
selector: {
age: {
$gt: 10
}
},
},
mySecondQuery: {
selector: {
age: {
$gt: 10
},
lastName: {
$eq: 'Nakamoto'
}
},
}
},
testData: [/** data for the documents. **/]
});
```
## Important details
- This is a build time tool. You should use it to find the best indexes for your queries during **build time**. Then you store these results and you application can use the best indexes during **run time**.
- It makes no sense to run time optimization with a different [RxStorage](./rx-storage.md) (+settings) that what you use in production. The result of the query optimizer is heavily dependent on the RxStorage and JavaScript runtime. For example it makes no sense to run the optimization in Node.js and then use the optimized indexes in the browser.
- It is very important that you use **production like** `testData`. Finding the best index heavily depends on data distribution and amount of stored/queried documents. For example if you store and query users with an `age` field, it makes no sense to just use a random number for the age because in production the `age` of your users is not equally distributed.
- The higher you set `runs`, the more test cycles will be performed and the more **significant** will be the time measurements which leads to a better index selection.
---
## How Local-First and WebMCP make your app accessible to agents
import {Steps} from '@site/src/components/steps';
import {VideoBox} from '@site/src/components/video-box';
import {Tabs} from '@site/src/components/tabs';
import {QuoteBlock} from '@site/src/components/quoteblock';
# How Local-First and WebMCP make your app accessible to agents
Over the past few years, the **[Local-First](./articles/local-first-future.md) architecture** has emerged as a new standard for building fast, offline-capable applications. Now, the long-awaited introduction of **WebMCP** makes local-first even more useful. By keeping data local, AI Agents can access, query, and mutate application states instantaneously on the client side, bypassing the latency and security vulnerabilities of traditional cloud APIs.
## What is WebMCP?
[WebMCP](https://webmachinelearning.github.io/webmcp/) (Web Model Context Protocol) is an experimental browser API that allows your web application to seamlessly expose "tools" for AI Agents. WebMCP is an adaptation of the Model Context Protocol (MCP) standardized for use within web browsers, currently incubated through the W3C Web Machine Learning community group.
When an AI Agent is active, it can discover these tools and call them programmatically with arguments as defined by a strict JSON Schema, via the `navigator.modelContext` API.
```ts
// Example: Registering a simple WebMCP tool natively
navigator.modelContext.registerTool({
name: 'get_weather',
description: 'Returns the current weather for a city',
inputSchema: {
type: 'object',
properties: {
city: { type: 'string' }
},
required: ['city']
}
}, async (params) => {
return fetch(`/api/weather?city=${params.city}`);
});
```
## The End of Scraping
WebMCP transforms the browser from a visual document viewer into a semantic capability surface.
For years, automation and AI have relied on simulating human inputs: guessing CSS class names `.button-primary`, parsing accessibility trees, and breaking whenever a layout changes. This "pixels-as-APIs" approach is slow, brittle, and highly token-dependent.
WebMCP provides a formalized machine interface alongside the human interface. By exposing deterministic, schema-validated tools, WebMCP closes the execution gap. AI Agents no longer guess how to interact with an application; they are given a precise contract defining exactly what operations are available and what payloads they accept.
I personally think these AI agents are inevitable. Like we adapted to Mobile from Desktop, its time to build websites and services for AI agents.
## Benefits of WebMCP
WebMCP introduces massive structural advantages over traditional browser automation:
- **Token Efficiency**: Providing structured JSON schemas to LLMs requires far fewer tokens than dumping raw DOM layout elements or accessibility trees into the prompt context.
- **Bypass Bot Protection**: Rather than forcing AI to use brittle DOM-scraping (like Selenium) that triggers CAPTCHAs or Cloudflare blocks, WebMCP gives them a sanctioned, highly-structured "front door" API.
- **Better Understanding**: The agent does not have to arbitrarily parse pixels or DOM layouts. Instead, it works directly on the deterministic data structures it receives.
- **Less Hallucination**: Because the agent receives exact data with high precision rather than inferring state from a UI, it is significantly less prone to hallucinating facts.
- **Access Control**: Developers have granular control over exactly what an agent can and cannot do by explicitly defining and exposing only specific WebMCP tools.
## Why Local-First and RxDB work great with WebMCP
WebMCP is uniquely powerful when paired with [local-first](/articles/local-first-future.md) databases like RxDB:
- **Unlimited Options**: Traditional websites must code a specific WebMCP tool for every possible action (e.g. `getProductsByPrice`, `searchProductsByColor`, `additemToCart`). By exposing a local-first database, the AI Agent has unlimited generic query and mutation options mathematically bound only by your schema.
- **Zero Latency**: Agents query data instantly from the local database on the user's device.
- **Offline Capable**: Because the data and the API are local, AI Agents can assist users completely offline.
- **Privacy First**: Sensitive user data can stay on the device while still being queryable by the on-device AI model.
- **Direct Access**: Agents can bypass the UI entirely and find exactly what they need with basic NoSQL queries.
- **LLM-Friendly NoSQL**: Writing NoSQL query objects (like [Mongo-style queries](./rx-query.md)) is significantly easier and more deterministic for LLMs to generate and validate than orchestrating complex, string-based SQL JOIN queries.
- **Native JSONSchema**: WebMCP relies entirely on JSONSchema to define tools and parameters. Because [RxDB schemas are *already* written in JSONSchema](./rx-schema.md), there is zero translation overheadβthe agent receives the exact structural contract it expects.
### Example Use Cases
Exposing your local database to AI agents unlocks new user experiences beyond what is possible with traditional websites.
**Online Shop**
An AI Agent searches your local catalog for items based on complex criteria, such as "Find all in-stock items cheaper than $50 that have a blue color". The agent can issue a highly efficient NoSQL query via WebMCP to retrieve all in-stock items under $50, and then seamlessly read through the returned JSON block to filter out the blue ones using its own internal LLM reasoning. Without a local-first database, developers would have to manually implement and maintain specific server-side WebMCP endpoints (e.g., `getItemsUnder50`) for every possible filtering combination the user might ask for, or expose a dynamic endpoint which queries the backend database but has unpredictable performance risks and massive security problems.
**Grocery Shopping List**
An agent can manage your list by using WebMCP modification tools in real time. If a user says "Move all drinks from Shop A to Shop B", the AI first uses `rxdb_query` to find all items categorized as drinks assigned to Shop A. It then uses the `rxdb_upsert` tool to update those specific documents to belong to Shop B. Alternatively, if a user says "Add everything I need to bake a cheesecake", the AI determines the ingredients and uses the `rxdb_insert` tool multiple times to instantly populate the local UI with the new shopping items, and `rxdb_delete` to remove items when the user says "I already have butter".
**Geotracking App**
Using the [RxDB continuous queries and observables](./reactivity.md) (`rxdb_wait_changes`), an agent monitors live tracking data to notify the user when a specific object starts moving.
## The RxDB WebMCP Plugin
RxDB provides a plugin `rxdb/plugins/webmcp` that lets you expose your collections to WebMCP with just a single function call.
The plugin dynamically reads your RxDB schema to assemble a prompt description and the tool's `inputSchema` so the AI knows exactly what data shapes are available and how to query them. It automatically registers the following WebMCP tools for each tracked collection:
**Read Operations**:
- `rxdb_query`: Run complex NoSQL queries against the local database.
- `rxdb_count`: Count the number of documents matching a specific query.
- `rxdb_changes`: Fetch the replication changestream since a given checkpoint.
- `rxdb_wait_changes`: Listen to live UI updates by pausing until a matching document changes occurs.
**Write Operations**:
- `rxdb_insert`: Insert new documents into the local collection.
- `rxdb_upsert`: Overwrite existing documents or insert them if they don't exist.
- `rxdb_delete`: Remove items from the local database by ID.
:::note
State-modifying tools like insert/upsert/delete can be disabled via the [`readOnly`](#readonly-default-false) option.
:::
### Quick Start
#### Add the plugin
First, ensure the plugin is added to your RxDB configuration:
```ts
import { addRxPlugin } from 'rxdb';
import { RxDBWebMCPPlugin } from 'rxdb/plugins/webmcp';
addRxPlugin(RxDBWebMCPPlugin);
```
#### Create a database
First, initialize your RxDatabase instance:
```ts
import { createRxDatabase } from 'rxdb';
import { getRxStorageLocalstorage } from 'rxdb/plugins/storage-localstorage';
const db = await createRxDatabase({
name: 'mydatabase',
storage: getRxStorageLocalstorage()
});
```
#### Add a Collection
Next, [add a collection](./rx-collection.md) with a simple schema. Providing an accurate schema is critical because the AI agent will use this exact schema to understand your data shape:
```ts
await db.addCollections({
todos: {
schema: {
title: 'Todo App Schema',
version: 0,
primaryKey: 'id',
type: 'object',
properties: {
id: { type: 'string', maxLength: 100 },
name: { type: 'string', description: 'The task title' },
done: { type: 'boolean' }
},
required: ['id', 'name', 'done']
}
}
});
```
#### Register Collections
Finally, activate WebMCP on the whole database or specific collections:
```ts
// Expose all collections in the DB to WebMCP (Read-only by default)
db.registerWebMCP();
// Or expose only a specific collection:
db.collections.todos.registerWebMCP();
```
### Options
The `registerWebMCP` method accepts an optional object:
#### `readOnly` (default: `false`)
By default, WebMCP allows modifier tools.
If you explicitly want the agent to only be able to query the database, enable `readOnly`.
```ts
db.registerWebMCP({
readOnly: true
});
```
This skips registering `rxdb_insert`, `rxdb_upsert`, and `rxdb_delete` tools.
#### `awaitReplicationsInSync` (default: `true`)
Because [replications](./replication.md) pull remote data into the local RxDB asynchronously, an AI Agent's query might miss data if a replication is still catching up. By default, WebMCP query invocations await (`awaitInSync()`) all running replications for that collection before returning the query results.
Set this to `false` if you want to allow queries without waiting for replication to be in sync.
```ts
db.registerWebMCP({
awaitReplicationsInSync: false
});
```
:::warning
If the application is offline and the replication is configured to retry infinitely, querying WebMCP with this option enabled may hang indefinitely while awaiting replication sync. Use wisely.
:::
### Logs and Errors
Both `registerWebMCP` methods (`db.registerWebMCP()` and `db.collections.humans.registerWebMCP()`) return an object containing two RxJS Subjects: `log$` and `error$`.
You can subscribe to these to monitor the AI agent's actions:
```ts
const { log$, error$ } = db.registerWebMCP();
log$.subscribe(info => {
// Log all tool calls, arguments, and responses
console.log('WebMCP Agent Action', info);
});
error$.subscribe(err => {
// Audit failed tool executions
console.error('WebMCP Agent Error', err);
});
```
### Pro Tip: Schema Descriptions for Better LLM Results
Because WebMCP sends your collection's [JSON schema](./rx-schema.md) directly to the AI Agent, the LLM uses the schema to understand the data model.
Providing detailed, highly accurate descriptions for your properties significantly improves the LLM's ability to construct valid and precise queries.
#### Bad Example
```ts
const productSchema = {
version: 0,
title: 'Product',
primaryKey: 'sku',
type: 'object',
properties: {
sku: {
type: 'string',
maxLength: 100
},
price: {
type: 'number'
}
},
required: ['sku', 'price']
};
```
#### Good Example
```ts
const productSchema = {
version: 0,
title: 'Store Product Inventory Item',
description: 'A physical item sold in our e-commerce store. Contains pricing and categorical SKU lookup data.',
primaryKey: 'sku',
type: 'object',
properties: {
sku: {
type: 'string',
maxLength: 100,
description: 'The Stock Keeping Unit identifier. Consists of a category prefix and a 6-digit number.'
},
price: {
type: 'number',
minimum: 0,
multipleOf: 0.01,
description: 'The price of the product in Euro (β¬).'
}
},
required: ['sku', 'price']
};
```
### Security and Prompt Injection
Architecturally, WebMCP turns the browser into a "capability surface" with explicit contracts. Security boundaries are clearer because only declared tools are visible and inputs are strictly validated against schemas.
However, be aware that WebMCP **does not completely eliminate prompt injection risks**. It significantly narrows the surface compared to DOM-level automation, but an Agent mimicking a well-behaved query against your schema can still produce corrupted behavior if the prompt itself contains malicious instructions. Ensure your application logic (and RxDB schema validation) assumes agent-provided payloads are untrusted.
:::note Beta Status
The WebMCP plugin is currently in **Beta**. APIs and behaviors are subject to change as the official [W3C WebMCP specification](https://webmachinelearning.github.io/webmcp/) and browser implementations evolve.
:::
## FAQ
What is WebMCP?
WebMCP (Web Model Context Protocol) is an experimental browser API that allows your web application to seamlessly expose "tools" for AI Agents. It acts as a standardized translation layer between your application's functionality and LLMs running within the browser, enabling natural language interactions with your web application's data. Wait for formal browser support for use in production environments.
How to mix the RxDB WebMCP with own WebMCP tools?
You can easily mix the RxDB WebMCP tools with your own tools. Since db.registerWebMCP() internally just calls navigator.modelContext.registerTool(), you can simply call this native method yourself to register any additional tools that interact with your frontend logic, external APIs, or other non-database components.
How to use the WebMCP polyfill for browsers without native support?
Since most browsers do not yet natively implement the navigator.modelContext API, you can use the WebMCP-org polyfill package @mcp-b/global to add support in any browser today.
Install the package:
```bash
npm install @mcp-b/global
```
Then import it once at the entry point of your application, before any WebMCP tools are registered:
```ts
import '@mcp-b/global';
// navigator.modelContext is now available
import { addRxPlugin } from 'rxdb';
import { RxDBWebMCPPlugin } from 'rxdb/plugins/webmcp';
addRxPlugin(RxDBWebMCPPlugin);
```
The polyfill sets up the navigator.modelContext interface so that your registered tools are accessible to AI agents running in the browser, even without native browser support.
How can I try WebMCP in Chrome?
WebMCP is currently in an early preview phase. You can test it today in Chrome Canary (version 145+) by following these steps:
1. **Enable the flag**: Go to `chrome://flags`, search for "WebMCP for testing", enable it, and relaunch Chrome.
2. **Install the inspector extension**: Install the [Model Context Tool Inspector Extension](https://chromewebstore.google.com/detail/model-context-tool-inspec/gbpdfapgefenggkahomfgkhfehlcenpd) to view registered tools, execute them manually, and test with an agent using Gemini API integration.
3. **Use a live demo**: You can test the integration directly on demo pages like the [RxDB WebMCP Quickstart](https://pubkey.github.io/rxdb-quickstart/).
## Follow up
To learn more about WebMCP and see it in action, check out these resources:
- [WebMCP Chrome Developer Blog Post](https://developer.chrome.com/blog/webmcp-epp?hl=en)
- [RxDB WebMCP Quickstart Repository](https://github.com/pubkey/rxdb-quickstart)
- [Live WebMCP RxDB Demo](https://pubkey.github.io/rxdb-quickstart/)
- Read: [Why Local-First Software Is the Future and what are its Limitations](/articles/local-first-future.md)
---
## Boost Your RxDB with Powerful Third-Party Plugins
# Third Party Plugins
* [rxdb-hooks](https://github.com/cvara/rxdb-hooks) A set of hooks to integrate RxDB into react applications.
* [rxdb-flexsearch](https://github.com/serenysoft/rxdb-flexsearch) The full text search for RxDB using [FlexSearch](https://github.com/nextapps-de/flexsearch).
* [rxdb-orion](https://github.com/serenysoft/rxdb-orion) Enables replication with [Laravel Orion](https://tailflow.github.io/laravel-orion-docs).
* [rxdb-supabase](https://github.com/marceljuenemann/rxdb-supabase) Enables replication with [Supabase](https://supabase.com/).
* [rxdb-utils](https://github.com/rafamel/rxdb-utils) Additional features for RxDB like models, timestamps, default values, view and more.
* [loki-async-reference-adapter](https://github.com/jonnyreeves/loki-async-reference-adapter) Simple async adapter for LokiJS, suitable to use RxDB's [Lokijs RxStorage](./rx-storage-lokijs.md) with React Native.
---
## π Discover RxDB Storage Benchmarks
## RxStorage Performance comparison
A big difference in the RxStorage implementations is the **performance**. In difference to a server side database, RxDB is bound to the limits of the JavaScript runtime and depending on the runtime, there are different possibilities to store and fetch data. For example in the browser it is only possible to store data in a [slow IndexedDB](./slow-indexeddb.md) or OPFS instead of a filesystem while on React-Native you can use the [SQLite storage](./rx-storage-sqlite.md).
Therefore the performance can be completely different depending on where you use RxDB and what you do with it. Here you can see some performance measurements and descriptions on how the different [storages](./rx-storage.md) work and how their performance is different.
## Persistent vs Semi-Persistent storages
The "normal" storages are always persistent. This means each RxDB write is directly written to disc and all queries run on the disc state. This means a good startup performance because nothing has to be done on startup.
In contrast, semi-persistent storages like [memory mapped](./rx-storage-memory-mapped.md) store all data in memory on startup and only save to disc occasionally (or on exit). Therefore it has a very fast read/write performance, but loading all data into memory on the first page load can take longer for big amounts of documents. Also these storages can only be used when all data fits into the memory at least once. In general it is recommended to stay on the persistent storages and only use semi-persistent ones, when you know for sure that the dataset will stay small (less than 2k documents).
## Performance comparison
In the following you can find some performance measurements and comparisons. Notice that these are only a small set of possible RxDB operations. If performance is really relevant for your use case, you should do your own measurements with usage-patterns that are equal to how you use RxDB in production.
### Measurements
Here the following metrics are measured:
- time-to-first-insert: Many storages run lazy, so it makes no sense to compare the time which is required to create a database with collections. Instead we measure the **time-to-first-insert** which is the whole timespan from database creation until the first single document write is done.
- insert documents (bulk): Insert 500 documents with a single bulk-insert operation.
- find documents by id (bulk): Here we fetch 100% of the stored documents with a single `findByIds()` call.
- insert documents (serial): Insert 50 documents, one after each other.
- find documents by id (serial): Here we find 50 documents in serial with one `findByIds()` call per document.
- find documents by query: Here we fetch 100% of the stored documents with a single `find()` call.
- find documents by query: Here we fetch all of the stored documents with a 4 `find()` calls that run in parallel. Each fetching 25% of the documents.
- count documents: Counts 100% of the stored documents with a single `count()` call. Here we measure 4 runs at once to have a higher number that is easier to compare.
## Browser based Storages Performance Comparison
The performance patterns of the browser based storages are very diverse. The [IndexedDB storage](./rx-storage-indexeddb.md) is recommended for mostly all use cases so you should start with that one. Later you can do performance testings and switch to another storage like [OPFS](./rx-storage-opfs.md) or [memory-mapped](./rx-storage-memory-mapped.md).
## Node/Native based Storages Performance Comparison
For most client-side native applications ([react-native](./react-native-database.md), [electron](./electron-database.md), [capacitor](./capacitor-database.md)), using the [SQLite RxStorage](./rx-storage-sqlite.md) is recommended. For non-client side applications like a server, use the [MongoDB storage](./rx-storage-mongodb.md) instead.
---
## RxDB NoSQL Performance Tips
# Performance tips for RxDB and other [NoSQL](./articles/in-memory-nosql-database.md) databases
In this guide, you'll find techniques to improve the performance of RxDB operations and queries. Notice that all your performance optimizations should be done with a correct tracking of the metrics, otherwise you might change stuff into the wrong direction.
## Use bulk operations
When you run write operations on multiple documents, make sure you use bulk operations instead of single document operations.
```ts
// wrong β
for(const docData of dataAr){
await myCollection.insert(docData);
}
// right βοΈ
await myCollection.bulkInsert(dataAr);
```
## Help the query planner by adding operators that better restrict the index range
Often on complex queries, RxDB (and other databases) do not pick the optimal index range when querying a result set.
You can add additional restrictive operators to ensure the query runs over a smaller index space and has a better performance.
Lets see some examples for different query types.
```ts
/**
* Adding a restrictive operator for an $or query
* so that it better limits the index space for the time-field.
*/
const orQuery = {
selector: {
$or: [
{
time: { $gt: 1234 },
},
{
time: { $eg: 1234 },
user: { $gt: 'foobar' }
},
]
time: { $gte: 1234 } // <- add restrictive operator
}
}
/**
* Adding a restrictive operator for an $regex query
* so that it better limits the index space for the user-field.
* We know that all matching fields start with 'foo' so we can
* tell the query to use that as lower constraint for the index.
*/
const regexQuery = {
selector: {
user: {
$regex: '^foo(.*)0-9$', // a complex regex with a ^ in the beginning
$gte: 'foo' // <- add restrictive operator
}
}
}
/**
* Adding a restrictive operator for a query on an enum field.
* so that it better limits the index space for the time-field.
*/
const enumQuery = {
selector: {
/**
* Here lets assume our status field has the enum type ['idle', 'in-progress', 'done']
* so our restrictive operator can exclude all documents with 'done' as status.
*/
status: {
$in: {
'idle',
'in-progress',
},
$gt: 'done' // <- add restrictive operator on status
}
}
}
```
## Set a specific index
Sometime the query planner of the database itself has no chance in picking the best index of the possible given indexes.
For queries where performance is very important, you might want to explicitly specify which index must be used.
```ts
const myQuery = myCollection.find({
selector: {
/* ... */
},
// explicitly specify index
index: [
'fieldA',
'fieldB'
]
});
```
## Try different ordering of index fields
The order of the fields in a compound index is very important for performance. When optimizing index usage, you should try out different orders on the index fields and measure which runs faster. For that it is very important to run tests on real-world data where the distribution of the data is the same as in production.
For example when there is a query on a user collection with an `age` and a `gender` field, it depends if the index `['gender', 'age']` performance better as `['age', 'gender']` based on the distribution of data:
```ts
const query = myCollection
.findOne({
selector: {
age: {
$gt: 18
},
gender: {
$eq: 'm'
}
},
/**
* Because the developer knows that 50% of the documents are 'male',
* but only 20% are below age 18,
* it makes sense to enforce using the ['gender', 'age'] index to improve performance.
* This could not be known by the query planer which might have chosen ['age', 'gender'] instead.
*/
index: ['gender', 'age']
});
```
Notice that RxDB has the [Query Optimizer Plugin](./query-optimizer.md) that can be used to automatically find the best indexes.
## Make a Query "hot" to reduce load
Having a query where the up-to-date result set is needed more than once, you might want to make the query "hot" by permanently subscribing to it. This ensures that the query result is kept up to date by RxDB ant the [EventReduce algorithm](https://github.com/pubkey/event-reduce) at any time so that at the moment you need the current results, it has them already.
For example when you use RxDB at [Node.js](./nodejs-database.md) for a webserver, you should use an outer "hot" query instead of running the same query again on every request to a route.
```ts
// wrong β
app.get('/list', (req, res) => {
const result = await myCollection.find({/* ... */}).exec();
res.send(JSON.stringify(result));
});
// right βοΈ
const query = myCollection.find({/* ... */});
query.subscribe(); // <- make it hot
app.get('/list', (req, res) => {
const result = await query.exec();
res.send(JSON.stringify(result));
});
```
## Store parts of your document data as attachment
For in-app databases like RxDB, it does not make sense to partially parse the `JSON` of a document. Instead, always the whole document json is parsed and handled. This has a better performance because `JSON.parse()` in JavaScript directly calls a C++ binding which can parse really fast compared to a partial parsing in JavaScript itself. Also by always having the full document, RxDB can de-duplicate memory caches of document across multiple queries.
The downside is that very very big documents with a complex structure can increase query time significantly. Documents fields with complex that are mostly not in use, can be move into an [attachment](./rx-attachment.md). This would lead RxDB to not fetch the attachment data each time the document is loaded from disc. Instead only when explicitly asked for.
```ts
const myDocument = await myCollection.insert({/* ... */});
const attachment = await myDocument.putAttachment(
{
id: 'otherStuff.json',
data: createBlob(JSON.stringify({/* ... */}), 'application/json'),
type: 'application/json'
}
);
```
## Process queries in a worker process
Moving database storage into a WebWorker can significantly improve performance in web applications that use RxDB or similar NoSQL databases. When database operations are executed in the main JavaScript thread, they can block or slow down the User Interface, especially during heavy or complex data operations. By offloading these operations to a WebWorker, you effectively separate the data processing workload from the UI thread. This means the main thread remains free to handle user interactions and render updates without delay, leading to a smoother and more responsive user experience. Additionally, WebWorkers allow for parallel data processing, which can expedite tasks like querying and indexing. This approach not only enhances UI responsiveness but also optimizes overall application performance by leveraging the multi-threading capabilities of modern browsers.
With RxDB you can use the [Worker](./rx-storage-worker.md) and [SharedWorker](./rx-storage-shared-worker.md) plugin to move the query processing away from the main thread.
## Use less plugins and hooks
Utilizing fewer [hooks](./middleware.md) and plugins in RxDB or similar NoSQL database systems can lead to markedly better performance. Each additional hook or plugin introduces extra layers of processing and potential overhead, which can cumulatively slow down database operations. These extensions often execute additional code or enforce extra checks with each operation, such as insertions, updates, or deletions. While they can provide valuable functionalities or custom behaviors, their overuse can inadvertently increase the complexity and execution time of basic database operations. By minimizing their use and only employing essential hooks and plugins, the system can operate more efficiently. This streamlined approach reduces the computational burden on each transaction, leading to faster response times and a more efficient overall data handling process, especially critical in high-load or real-time applications where performance is paramount.
---
## Solving IndexedDB Slowness for Seamless Apps
# Why IndexedDB is slow and what to use instead
So you have a JavaScript web application that needs to store data at the client side, either to make it [offline usable](./offline-first.md), just for caching purposes or for other reasons.
For [in-browser data storage](./articles/browser-database.md), you have some options:
- **Cookies** are sent with each HTTP request, so you cannot store more than a few strings in them.
- **WebSQL** [is deprecated](https://hacks.mozilla.org/2010/06/beyond-html5-database-apis-and-the-road-to-indexeddb/) because it never was a real standard and turning it into a standard would have been too difficult.
- [LocalStorage](./articles/localstorage.md) is a synchronous API over asynchronous IO-access. Storing and reading data can fully block the JavaScript process so you cannot use LocalStorage for more than few simple key-value pairs.
- The **FileSystem API** could be used to store plain binary files, but it is [only supported in chrome](https://caniuse.com/filesystem) for now.
- **IndexedDB** is an indexed key-object database. It can store json data and iterate over its indexes. It is [widely supported](https://caniuse.com/indexeddb) and stable.
:::note UPDATE April 2023
Since beginning of 2023, all modern browsers ship the **File System Access API** which allows to persistently store data in the browser with a way better performance. For [RxDB](https://rxdb.info/) you can use the [OPFS RxStorage](./rx-storage-opfs.md) to get about 4x performance improvement compared to IndexedDB.
:::
It becomes clear that the only way to go is IndexedDB. You start developing your app and everything goes fine.
But as soon as your app gets bigger, more complex or just handles more data, you might notice something. **IndexedDB is slow**. Not slow like a database on a cheap server, **even slower**! Inserting a few hundred documents can take up several seconds. Time which can be critical for a fast page load. Even sending data over the internet to the backend can be faster than storing it inside of an IndexedDB database.
> Transactions vs Throughput
So before we start complaining, lets analyze what exactly is slow. When you run tests on Nolans [Browser Database Comparison](http://nolanlawson.github.io/database-comparison/) you can see that inserting 1k documents into IndexedDB takes about 80 milliseconds, 0.08ms per document. This is not really slow. It is quite fast and it is very unlikely that you want to store that many document at the same time at the client side. But the key point here is that all these documents get written in a `single transaction`.
I forked the comparison tool [here](https://pubkey.github.io/client-side-databases/database-comparison/index.html) and changed it to use one transaction per document write. And there we have it. Inserting 1k documents with one transaction per write, takes about 2 seconds. Interestingly if we increase the document size to be 100x bigger, it still takes about the same time to store them. This makes clear that the limiting factor to IndexedDB performance is the transaction handling, not the data throughput.
To fix your IndexedDB performance problems you have to make sure to use as less data transfers/transactions as possible.
Sometimes this is easy, as instead of iterating over a documents list and calling single inserts, with RxDB you could use the [bulk methods](https://rxdb.info/rx-collection.html#bulkinsert) to store many document at once.
But most of the time is not so easy. Your user clicks around, data gets replicated from the backend, another browser tab writes data. All these things can happen at random time and you cannot crunch all that data in a single transaction.
Another solution is to just not care about performance at all. In a few releases the browser vendors will have optimized IndexedDB and everything is fast again. Well, IndexedDB was slow [in 2013](https://www.researchgate.net/publication/281065948_Performance_Testing_and_Comparison_of_Client_Side_Databases_Versus_Server_Side) and it is still slow today. If this trend continues, it will still be slow in a few years from now. Waiting is not an option. The chromium devs made [a statement](https://bugs.chromium.org/p/chromium/issues/detail?id=1025456#c15) to focus on optimizing read performance, not write performance.
Switching to WebSQL (even if it is deprecated) is also not an option because, like [the comparison tool shows](https://pubkey.github.io/client-side-databases/database-comparison/index.html), it has even slower transactions.
So you need a way to **make IndexedDB faster**. In the following I lay out some performance optimizations than can be made to have faster reads and writes in IndexedDB.
**HINT:** You can reproduce all performance tests [in this repo](https://github.com/pubkey/indexeddb-performance-tests). In all tests we work on a dataset of 40000 `human` documents with a random `age` between `1` and `100`.
## Batched Cursor
With [IndexedDB 2.0](https://caniuse.com/indexeddb2), new methods were introduced which can be utilized to improve performance. With the `getAll()` method, a faster alternative to the old `openCursor()` can be created which improves performance when reading data from the IndexedDB store.
Lets say we want to query all user documents that have an `age` greater than `25` out of the store.
To implement a fast batched cursor that only needs calls to `getAll()` and not to `getAllKeys()`, we first need to create an `age` index that contains the primary `id` as last field.
```ts
myIndexedDBObjectStore.createIndex(
'age-index',
[
'age',
'id'
]
);
```
This is required because the `age` field is not unique, and we need a way to checkpoint the last returned batch so we can continue from there in the next call to `getAll()`.
```ts
const maxAge = 25;
let result = [];
const tx: IDBTransaction = db.transaction([storeName], 'readonly', TRANSACTION_SETTINGS);
const store = tx.objectStore(storeName);
const index = store.index('age-index');
let lastDoc;
let done = false;
/**
* Run the batched cursor until all results are retrieved
* or the end of the index is reached.
*/
while (done === false) {
await new Promise((res, rej) => {
const range = IDBKeyRange.bound(
/**
* If we have a previous document as checkpoint,
* we have to continue from it's age and id values.
*/
[
lastDoc ? lastDoc.age : -Infinity,
lastDoc ? lastDoc.id : -Infinity,
],
[
maxAge + 0.00000001,
String.fromCharCode(65535)
],
true,
false
);
const openCursorRequest = index.getAll(range, batchSize);
openCursorRequest.onerror = err => rej(err);
openCursorRequest.onsuccess = e => {
const subResult: TestDocument[] = e.target.result;
lastDoc = lastOfArray(subResult);
if (subResult.length === 0) {
done = true;
} else {
result = result.concat(subResult);
}
res();
};
});
}
console.dir(result);
```
As the performance test results show, using a batched cursor can give a huge improvement. Interestingly choosing a high batch size is important. When you known that all results of a given `IDBKeyRange` are needed, you should not set a batch size at all and just directly query all documents via `getAll()`.
RxDB uses batched cursors in the [IndexedDB RxStorage](./rx-storage-indexeddb.md).
## IndexedDB Sharding
Sharding is a technique, normally used in server side databases, where the database is partitioned horizontally. Instead of storing all documents at one table/collection, the documents are split into so called **shards** and each shard is stored on one table/collection. This is done in server side architectures to spread the load between multiple physical servers which **increases scalability**.
When you use IndexedDB in a browser, there is of course no way to split the load between the client and other servers. But you can still benefit from sharding. Partitioning the documents horizontally into **multiple IndexedDB stores**, has shown to have a big performance improvement in write- and read operations while only increasing initial pageload slightly.
As shown in the performance test results, sharding should always be done by `IDBObjectStore` and not by database. Running a batched cursor over the whole dataset with 10 store shards in parallel is about **28% faster** then running it over a single store. Initialization time increases minimal from `9` to `17` milliseconds.
Getting a quarter of the dataset by batched iterating over an index, is even **43%** faster with sharding then when a single store is queried.
As downside, getting 10k documents by their id is slower when it has to run over the shards.
Also it can be much effort to recombined the results from the different shards into the required query result. When a query without a limit is done, the sharding method might cause a data load huge overhead.
Sharding can be used with RxDB with the [Sharding Plugin](./rx-storage-sharding.md).
## Custom Indexes
Indexes improve the query performance of IndexedDB significant. Instead of fetching all data from the storage when you search for a subset of it, you can iterate over the index and stop iterating when all relevant data has been found.
For example to query for all user documents that have an `age` greater than `25`, you would create an `age+id` index.
To be able to run a batched cursor over the index, we always need our primary key (`id`) as the last index field.
Instead of doing this, you can use a `custom index` which can improve the performance. The custom index runs over a helper field `ageIdCustomIndex` which is added to each document on write. Our index now only contains a single `string` field instead of two (age-`number` and id-`string`).
```ts
// On document insert add the ageIdCustomIndex field.
const idMaxLength = 20; // must be known to craft a custom index
docData.ageIdCustomIndex = docData.age + docData.id.padStart(idMaxLength, ' ');
store.put(docData);
// ...
```
```ts
// normal index
myIndexedDBObjectStore.createIndex(
'age-index',
[
'age',
'id'
]
);
// custom index
myIndexedDBObjectStore.createIndex(
'age-index-custom',
[
'ageIdCustomIndex'
]
);
```
To iterate over the index, you also use a custom crafted keyrange, depending on the last batched cursor checkpoint. Therefore the `maxLength` of `id` must be known.
```ts
// keyrange for normal index
const range = IDBKeyRange.bound(
[25, ''],
[Infinity, Infinity],
true,
false
);
// keyrange for custom index
const range = IDBKeyRange.bound(
// combine both values to a single string
25 + ''.padStart(idMaxLength, ' '),
Infinity,
true,
false
);
```
As shown, using a custom index can further improve the performance of running a batched cursor by about `10%`.
Another big benefit of using custom indexes, is that you can also encode `boolean` values in them, which [cannot be done](https://github.com/w3c/IndexedDB/issues/76) with normal IndexedDB indexes.
RxDB uses custom indexes in the [IndexedDB RxStorage](./rx-storage-indexeddb.md).
## Relaxed durability
Chromium based browsers allow to set [durability](https://developer.mozilla.org/en-US/docs/Web/API/IDBTransaction/durability) to `relaxed` when creating an IndexedDB transaction. Which runs the transaction in a less secure durability mode, which can improve the performance.
> The user agent may consider that the transaction has successfully committed as soon as all outstanding changes have been written to the operating system, without subsequent verification.
As shown [here](https://nolanlawson.com/2021/08/22/speeding-up-indexeddb-reads-and-writes/), using the relaxed durability mode can improve performance slightly. The best performance improvement could be measured when many small transactions have to be run. Less, bigger transaction do not benefit that much.
## Explicit transaction commits
By explicitly committing a transaction, another slight performance improvement can be achieved. Instead of waiting for the browser to commit an open transaction, we call the `commit()` method to explicitly close it.
```ts
// .commit() is not available on all browsers, so first check if it exists.
if (transaction.commit) {
transaction.commit()
}
```
The improvement of this technique is minimal, but observable as [these tests](https://nolanlawson.com/2021/08/22/speeding-up-indexeddb-reads-and-writes/) show.
## In-Memory on top of IndexedDB
To prevent transaction handling and to fix the performance problems, we need to stop using IndexedDB as a database. Instead all data is loaded into the memory on the initial page load. Here all reads and writes happen in memory which is about 100x faster. Only some time after a write occurred, the memory state is persisted into IndexedDB with a **single write transaction**. In this scenario IndexedDB is used as a filesystem, not as a database.
There are some libraries that already do that:
- LokiJS with the [IndexedDB Adapter](https://techfort.github.io/LokiJS/LokiIndexedAdapter.html)
- [Absurd-SQL](https://github.com/jlongster/absurd-sql)
- SQL.js with the [empscripten Filesystem API](https://emscripten.org/docs/api_reference/Filesystem-API.html#filesystem-api-idbfs)
- [DuckDB Wasm](https://duckdb.org/2021/10/29/duckdb-wasm.html)
### In-Memory: Persistence
One downside of not directly using IndexedDB, is that your data is not persistent all the time. And when the JavaScript process exists without having persisted to IndexedDB, data can be lost. To prevent this from happening, we have to ensure that the in-memory state is written down to the disc. One point is make persisting as fast as possible. LokiJS for example has the `incremental-indexeddb-adapter` which only saves new writes to the disc instead of persisting the whole state. Another point is to run the persisting at the correct point in time. For example the RxDB [LokiJS storage](https://rxdb.info/rx-storage-lokijs.html) persists in the following situations:
- When the database is idle and no write or query is running. In that time we can persist the state if any new writes appeared before.
- When the `window` fires the [beforeunload event](https://developer.mozilla.org/en-US/docs/Web/API/WindowEventHandlers/onbeforeunload) we can assume that the JavaScript process is exited any moment and we have to persist the state. After `beforeunload` there are several seconds time which are sufficient to store all new changes. This has shown to work quite reliable.
The only missing event that can happen is when the browser exists unexpectedly like when it crashes or when the power of the computer is shut of.
### In-Memory: Multi Tab Support
One big difference between a web application and a 'normal' app, is that your users can use the app in multiple browser tabs at the same time. But when you have all database state in memory and only periodically write it to disc, multiple browser tabs could overwrite each other and you would loose data. This might not be a problem when you rely on a client-server replication, because the lost data might already be replicated with the backend and therefore with the other tabs. But this would not work when the client is offline.
The ideal way to solve that problem, is to use a [SharedWorker](https://developer.mozilla.org/en/docs/Web/API/SharedWorker). A [SharedWorker](./rx-storage-shared-worker.md) is like a [WebWorker](https://developer.mozilla.org/en/docs/Web/API/Web_Workers_API) that runs its own JavaScript process only that the SharedWorker is shared between multiple contexts. You could create the database in the SharedWorker and then all browser tabs could request the Worker for data instead of having their own database. But unfortunately the SharedWorker API does [not work](https://caniuse.com/sharedworkers) in all browsers. Safari [dropped](https://bugs.webkit.org/show_bug.cgi?id=140344) its support and InternetExplorer or Android Chrome, never adopted it. Also it cannot be polyfilled. **UPDATE:** [Apple added SharedWorkers back in Safari 142](https://developer.apple.com/safari/technology-preview/release-notes/)
Instead, we could use the [BroadcastChannel API](https://developer.mozilla.org/en-US/docs/Web/API/Broadcast_Channel_API) to communicate between tabs and then apply a [leader election](https://github.com/pubkey/broadcast-channel#using-the-leaderelection) between them. The [leader election](./leader-election.md) ensures that, no matter how many tabs are open, always one tab is the `Leader`.
The disadvantage is that the leader election process takes some time on the initial page load (about 150 milliseconds). Also the leader election can break when a JavaScript process is fully blocked for a longer time. When this happens, a good way is to just reload the browser tab to restart the election process.
## Further read
- [Offline First Database Comparison](https://github.com/pubkey/client-side-databases)
- [Speeding up IndexedDB reads and writes](https://nolanlawson.com/2021/08/22/speeding-up-indexeddb-reads-and-writes/)
- [SQLITE ON THE WEB: ABSURD-SQL](https://hackaday.com/2021/08/24/sqlite-on-the-web-absurd-sql/)
- [SQLite in a PWA with FileSystemAccessAPI](https://anita-app.com/blog/articles/sqlite-in-a-pwa-with-file-system-access-api.html)
- [Response to this article by Oren Eini](https://ravendb.net/articles/re-why-indexeddb-is-slow-and-what-to-use-instead)
---
## Alternatives for realtime local-first JavaScript applications and local databases
# Alternatives for realtime offline-first JavaScript applications
To give you an augmented view over the topic of client side JavaScript databases, this page contains all known alternatives to **RxDB**. Remember that you are reading this inside of the RxDB documentation, so everything is **opinionated**.
If you disagree with anything or think that something is missing, make a pull request to this file on the RxDB github repository.
:::note
RxDB has these main benefits:
- RxDB is a battle proven tool [widely used](/#reviews) by companies in real projects in production.
- RxDB is not VC funded and therefore does not require you to use a specific cloud service to rip you off. RxDB can be used with your [own backend](./replication-http.md) or no backend at all.
- RxDB has a working business model of selling [premium plugins](/premium/) which ensures that RxDB will be maintained and improved continuously while many alternatives are dead already or seem to die soon.
- RxDB has years (since 2016) of performance optimization, bug fixing and feature adding. It is just working as is and there are close to zero [open issues](https://github.com/pubkey/rxdb/issues).
:::
--------------------------------------------------------------------------------
## Alternatives to RxDB
[RxDB](https://rxdb.info) is an **observable**, **replicating**, **[local first](./offline-first.md)**, **JavaScript** database. So it makes only sense to list similar projects as alternatives, not just any database or JavaScript store library. However, I will list up some projects that RxDB is often compared with, even if it only makes sense for some use cases.
Here are the alternatives to RxDB:
### Firebase
Firebase is a **platform** developed by Google for creating mobile and web applications. Firebase has many features and products, two of which are client side databases. The [Realtime Database](./articles/firebase-realtime-database-alternative.md) and the [Cloud Firestore](./articles/firestore-alternative.md).
#### Firebase - Realtime Database
The firebase realtime database was the first database in firestore. It has to be mentioned that in this context, "realtime" means **"realtime replication"**, not "realtime computing". The firebase realtime database stores data as a big unstructured JSON tree that is replicated between clients and the backend.
#### Firebase - Cloud Firestore
The firestore is the successor to the realtime database. The big difference is that it behaves more like a 'normal' database that stores data as documents inside of collections. The conflict resolution strategy of firestore is always *last-write-wins* which might or might not be suitable for your use case.
The biggest difference to RxDB is that firebase products are only able to be used on top of the Firebase cloud hosted backend, which creates a vendor lock-in. RxDB can replicate with any self hosted CouchDB server or custom GraphQL endpoints. You can even replicate Firestore to RxDB with the [Firestore Replication Plugin](./replication-firestore.md).
### Meteor
Meteor (since 2012) is one of the oldest technologies for JavaScript realtime applications. Meteor is not a library but a whole framework with its own package manager, database management and replication.
Because of how it works, it has proven to be hard to integrate it with other modern JavaScript frameworks like [angular](https://github.com/urigo/angular-meteor), [vue.js](./articles/vue-database.md) or svelte.
Meteor uses MongoDB in the backend and can replicate with a Minimongo database in the frontend.
While testing, it has proven to be impossible to make a meteor app **offline first** capable. There are [some projects](https://github.com/frozeman/meteor-persistent-minimongo2) that might do this, but all are unmaintained.
### Minimongo
Forked in Jan 2014 from meteorJSs' minimongo package, Minimongo is a client-side, in-memory, JavaScript version of MongoDB with backend replication over HTTP. Similar to MongoDB, it stores data in documents inside of [collections](./rx-collection.md) and also has the same query syntax. Minimongo has different storage adapters for IndexedDB, WebSQL, [LocalStorage](./articles/localstorage.md) and SQLite.
Compared to RxDB, Minimongo has no concept of revisions or conflict handling, which might lead to undefined behavior when used with replication or in multiple browser tabs. Minimongo has no observable queries or changestream.
### WatermelonDB
WatermelonDB is a reactive & asynchronous JavaScript database. While originally made for [React](./articles/react-database.md) and [React Native](./react-native-database.md), it can also be used with other JavaScript frameworks. The main goal of WatermelonDB is **performance** within an application with lots of data.
In React Native, WatermelonDB uses the provided SQLite database. Also there is an Expo plugin for WatermelonDB. In a browser, WatermelonDB uses the LokiJS in-memory database to store and query data. WatermelonDB is one of the rare projects that support both Flow and Typescript at the same time.
### AWS Amplify
AWS Amplify is a collection of tools and libraries to develop web- and mobile frontend applications. Similar to firebase, it provides everything needed like authentication, analytics, a REST API, storage and so on. Everything hosted in the AWS Cloud, even when they state that *"AWS Amplify is designed to be open and pluggable for any custom backend or service"*. For realtime replication, AWS Amplify can connect to an AWS App-Sync GraphQL endpoint.
### AWS Datastore
Since December 2019 the Amplify library includes the AWS Datastore which is a document-based, client side database that is able to replicate data via AWS AppSync in the background.
The main difference to other projects is the complex project configuration via the amplify cli and the bit confusing query syntax that works over functions. Complex Queries with multiple `OR/AND` statements are not possible which might change in the future.
Local development is hard because the AWS AppSync mock does not support realtime replication. It also is not really offline-first because a user login is always required.
```ts
// An AWS datastore OR query
const posts = await DataStore.query(Post, c => c.or(
c => c.rating("gt", 4).status("eq", PostStatus.PUBLISHED)
));
// An AWS datastore SORT query
const posts = await DataStore.query(Post, Predicates.ALL, {
sort: s => s.rating(SortDirection.ASCENDING).title(SortDirection.DESCENDING)
});
```
The biggest difference to RxDB is that you have to use the AWS cloud backends. This might not be a problem if your data is at AWS anyway.
### RethinkDB
RethinkDB is a backend database that pushed dynamic JSON data to the client in realtime. It was founded in 2009 and the company shut down in 2016.
Rethink db is not a client side database, it streams data from the backend to the client which of course does not work while offline.
### Horizon
Horizon is the client side library for RethinkDB which provides useful functions like authentication, permission management and subscription to a RethinkDB backend. Offline support [never made](https://github.com/rethinkdb/horizon/issues/58) it to horizon.
### Supabase
Supabase labels itself as "*an open source Firebase alternative*". It is a collection of open source tools that together mimic many Firebase features, most of them by providing a wrapper around a PostgreSQL database. While it has realtime queries that run over the wire, like with RethinkDB, Supabase has no client-side storage or replication feature and therefore is not offline first.
### CouchDB
Apache CouchDB is a server-side, document-oriented database that is mostly known for its multi-master replication feature. Instead of having a master-slave replication, with CouchDB you can run replication in any constellation without having a master server as bottleneck where the server even can go off- and online at any time. This comes with the drawback of having a slow replication with much network overhead.
CouchDB has a changestream and a query syntax similar to MongoDB.
### PouchDB
PouchDB is a JavaScript database that is compatible with most of the CouchDB API. It has an adapter system that allows you to switch out the underlying storage layer. There are many adapters like for [IndexedDB](./rx-storage-indexeddb.md), [SQLite](./rx-storage-sqlite.md), the Filesystem and so on. The main benefit is to be able to replicate data with any CouchDB compatible endpoint.
Because of the CouchDB compatibility, PouchDB has to do a lot of overhead in handling the revision tree of documents, which is why it can show bad performance for bigger datasets.
RxDB was originally build around PouchDB until the storage layer was abstracted out in version [10.0.0](./releases/10.0.0.md) so it now allows to use different `RxStorage` implementations. PouchDB has some performance issues because of how it has to store the document revision tree to stay compatible with the CouchDB API.
### Couchbase
Couchbase (originally known as Membase) is another NoSQL document database made for realtime applications.
It uses the N1QL query language which is more SQL like compared to other NoSQL query languages. In theory you can achieve replication of a Couchbase with a PouchDB database, but this has shown to be not [that easy](https://github.com/pouchdb/pouchdb/issues/7793#issuecomment-501624297).
### Cloudant
Cloudant is a cloud-based service that is based on [CouchDB](./replication-couchdb.md) and has mostly the same features.
It was originally designed for cloud computing where data can automatically be distributed between servers. But it can also be used to replicate with frontend PouchDB instances to create scalable web applications.
It was bought by IBM in 2014 and since 2018 the Cloudant Shared Plan is retired and migrated to IBM Cloud.
### Hoodie
Hoodie is a backend solution that enables offline-first JavaScript frontend development without having to write backend code. Its main goal is to abstract away configuration into simple calls to the Hoodie API.
It uses CouchDB in the backend and PouchDB in the frontend to enable offline-first capabilities.
The last commit for hoodie was one year ago and the website (hood.ie) is offline which indicates it is not an active project anymore.
### LokiJS
LokiJS is a JavaScript embeddable, in-memory database. And because everything is handled in-memory, LokiJS has awesome performance when mutating or querying data. You can still persist to a permanent storage (IndexedDB, Filesystem etc.) with one of the provided storage adapters. The persistence happens after a timeout is reached after a write, or before the JavaScript process exits. This also means you could loose data when the JavaScript process exits ungracefully like when the power of the device is shut down or the browser crashes.
While the project is not that active anymore, it is more *finished* than *unmaintained*.
In the past, RxDB supported using [LokiJS as RxStorage](./rx-storage-lokijs.md) but because the LokiJS is not maintained anymore and had too many issues, this storage option was removed in RxDB version 16.
### Gundb
GUN is a JavaScript graph database. While having many features, the **decentralized** replication is the main unique selling point. You can replicate data Peer-to-Peer without any centralized backend server. GUN has several other features that are useful on top of that, like encryption and authentication.
While testing it was really hard to get basic things running. GUN is open source, but because of how the source code [is written](https://github.com/amark/gun/blob/master/src/put.js), it is very difficult to understand what is going wrong.
### sql.js
sql.js is a javascript library to run SQLite on the web. It uses a virtual database file stored in memory and does not have any persistence. All data is lost once the JavaScript process exits. sql.js is created by compiling SQLite to WebAssembly so it has about the same features as SQLite. For older browsers there is a JavaScript fallback.
### absurd-sQL
Absurd-sql is a project that implements an IndexedDB-based persistence for sql.js. Instead of directly writing data into the IndexedDB, it treats IndexedDB like a disk and stores data in blocks there which shows to have a much better performance, mostly because of how [performance expensive](./slow-indexeddb.md) IndexedDB transactions are.
### NeDB
NeDB was a embedded persistent or in-memory database for Node.js, nw.js, [Electron](./electron-database.md) and browsers.
It is document-oriented and had the same query syntax as MongoDB.
Like LokiJS it has persistence adapters for IndexedDB etc. to persist the database state on the disc.
The last commit to NeDB was in **2016**.
### Dexie.js
Dexie.js is a minimalistic wrapper for IndexedDB. While providing a better API than plain IndexedDB, Dexie also improves performance by batching transactions and other optimizations. It also adds additional non-IndexedDB features like observable queries or multi tab support or react hooks.
Compared to RxDB, Dexie.js does not support complex (MongoDB-like) queries and requires a lot of fiddling when a document range of a specific index must be fetched.
Dexie.js is used by Whatsapp Web, Microsoft To Do and Github Desktop.
RxDB supports using [Dexie.js as Database storage](./rx-storage-dexie.md) which enhances IndexedDB via dexie with RxDB features like MongoDB-like queries etc.
### LowDB
LowDB is a small, local JSON database powered by the Lodash library. It is designed to be simple, easy to use, and straightforward. LowDB allows you to perform native JavaScript queries and persist data in a flat JSON file. Written in TypeScript, it's particularly well-suited for small projects, prototyping, or when you need a lightweight, file-based database.
As an alternative to LowDB, [RxDB](./) offers real-time reactivity, allowing developers to subscribe to database changes, a feature not natively available in LowDB. Additionally, RxDB provides robust [query capabilities](./rx-query.md), including the ability to subscribe to query results for automatic UI updates. These features make RxDB a strong alternative to LowDB for more complex and dynamic applications.
### localForage
localForage is a popular JavaScript library for offline storage that provides a simple, promise-based API. It abstracts over different storage mechanisms such as [IndexedDB](./rx-storage-indexeddb.md), WebSQL, or [localStorage](./articles/localstorage.md), making it easier to write code once and have it work seamlessly across various browsers. While localForage is great for storing data locally in a key-value manner, it doesn't provide the real-time reactive queries, [conflict handling](./transactions-conflicts-revisions.md), or revision-based replication that RxDB does. This makes localForage a useful choice for straightforward caching or persistent storage needs, but not ideal for advanced offline-first scenarios requiring multi-user collaboration or complex querying.
### MongoDB Realm
Originally Realm was a mobile database for Android and iOS. Later they added support for other languages and runtimes, also for JavaScript.
It was meant as replacement for SQLite but is more like an object store than a full SQL database.
In 2019 MongoDB bought Realm and changed the projects focus.
Now Realm is made for replication with the MongoDB Realm Sync based on the MongoDB Atlas Cloud platform. This tight coupling to the MongoDB cloud service is a big downside for most use cases.
### Apollo
The Apollo [GraphQL](./replication-graphql.md) platform is made to transfer data between a server to UI applications over GraphQL endpoints. It contains several tools like GraphQL clients in different languages or libraries to create GraphQL endpoints.
While it is has different caching features for offline usage, compared to RxDB it is not fully offline first because caching alone does not mean your application is fully usable when the user is offline.
### Replicache
Replicache is a client-side sync framework for building realtime, collaborative, [local-first](./articles/local-first-future.md) web apps. It claims to work with most backend stacks. In contrast to other local first tools, replicache does not work like a local database. Instead it runs on so called `mutators` that unify behavior on the client and server side. So instead of implementing and calling REST routes on both sides of your stack, you will implement mutators that define a specific delta behavior based on the input data. To observe data in replicache, there are `subscriptions` that notify your frontend application about changes to the state.
Replicache can be used in most frontend technologies like browsers, React/Remix, NextJS/Vercel and React Native. While Replicache can be installed and used from npm, the Replicache source code is not open source and the Replicache github repo does not allow you to inspect or debug it. Still you can use replicache for in non-commercial projects, or for companies with < $200k revenue (ARR) and < $500k in funding. (2024: Replicache will be free and Rocicorp are working on a new Zerosync product to succeed Replicache and Reflect.)
### InstantDB
InstantDB is designed for real-time data synchronization with built-in offline support, allowing changes to be queued locally and [synced](./replication.md) when the user reconnects. While it offers seamless [optimistic updates](./articles/optimistic-ui.md) and rollback capabilities, its offline-first design is not as mature or comprehensive as RxDB's - the [offline data](./articles/offline-database.md) is more of a cache, not a full-database sync. The query language used is Datalog, and the backend sync service is written in Clojure. InstantDB is focused more on simplicity and real-time collaboration, with fewer customization options for storage or conflict resolution compared to RxDB, which supports various storage adapters and advanced conflict handling via CRDTs.
### Yjs
Yjs is a [CRDT-based](./crdt.md) (Conflict-free Replicated Data Type) library focused on enabling real-time collaboration - particularly for text editing, although it can handle other data types as well. While it provides powerful conflict resolution and peer-to-peer synchronization out of the box, Yjs itself is not a full-fledged database. Instead, you typically combine Yjs with other storage or networking layers to achieve a [local-first architecture](./offline-first.md). This flexibility allows for sophisticated [real-time](./articles/realtime-database.md) features, but also means you must handle indexing, queries, and persistence on your own if you need them. Compared to RxDB, Yjs does not offer built-in replication adapters or a query system, so developers who require a more complete solution for conflict resolution, data persistence, and offline-first capabilities may find RxDB more convenient.
### ElectricSQL
2024: ElectricSQL is being rewritten in a new Electric-Next branch, which focuses on partial syncing of ("shapes", which makes is basically a NoSQL like document database) of data from a remote Postgres DB to a local clients written in TypeScript/JS or Elixir. The write path is not yet implemented, neither is client-side reactivity. The ElectricSQL backend is written in Elixir.
### SignalDB
SignalDB provides a reactive, in-memory local-first JavaScript database with real-time sync, but it doesn't offer the same level of multi-client replication or flexibility with storage backends that RxDB provides, and through a RxDB persistence adapters you can actually use SignalDB for the front-end reactivity while relying on RxDB for backend sync and persistence.
### PowerSync
PowerSync is a "framework" for implementing local-first solutions. It centralizes business logic and conflict resolution on a central, authoritative server (PostgreSQL or MongoDB), vs RxDB that also supports custom backends. Both RxDB and PowerSync can be used with a variety of storage backends, but PowerSync uses SQLite as the front-end database which has shown to be slow because the WASM-SQLite abstraction increases read and write latency. In terms of client SDKs, PowerSync offers Flutter, Kotlin, and Swift in addition to JS/TypeScript. PowerSync offers many client technologies, PowerSync is under a license that restricts commercial use that competes with PowerSync and the JourneyApps Platform.
# Read further
- [Offline First Database Comparison](https://github.com/pubkey/client-side-databases)
---
## RxDB as a Database in an Angular Application
import {VideoBox} from '@site/src/components/video-box';
# RxDB as a Database in an Angular Application
In modern web development, Angular has emerged as a popular framework for building robust and scalable applications. As Angular applications often require persistent [storage](./browser-storage.md) and efficient data handling, choosing the right database solution is crucial. One such solution is [RxDB](https://rxdb.info/), a reactive JavaScript database for the [browser](./browser-database.md), [node.js](../nodejs-database.md), and [mobile devices](./mobile-database.md). In this article, we will explore the integration of RxDB into an Angular application and examine its various features and techniques.
## Angular Web Applications
Angular is a powerful JavaScript framework developed and maintained by Google. It enables developers to build single-page applications (SPAs) with a modular and component-based approach. Angular provides a comprehensive set of tools and features for creating dynamic and responsive web applications.
## Importance of Databases in Angular Applications
Databases play a vital role in Angular applications by providing a structured and efficient way to store, retrieve, and manage data. Whether it's handling user authentication, caching data, or persisting application state, a robust database solution is essential for ensuring optimal performance and user experience.
## Introducing RxDB as a Database Solution
RxDB stands for Reactive Database and is built on the principles of reactive programming. It combines the best features of [NoSQL databases](./in-memory-nosql-database.md) with the power of reactive programming to provide a scalable and efficient database solution. RxDB offers seamless integration with Angular applications and brings several unique features that make it an attractive choice for developers.
## Getting Started with RxDB
To begin our journey with RxDB, let's understand its key concepts and features.
### What is RxDB?
[RxDB](https://rxdb.info/) is a client-side database that follows the principles of reactive programming. It is built on top of IndexedDB, the [native browser database](./browser-database.md), and leverages the RxJS library for reactive data handling. RxDB provides a simple and intuitive API for managing data and offers features like data replication, multi-tab support, and efficient query handling.
### Reactive Data Handling
At the core of RxDB is the concept of reactive data handling. RxDB leverages observables and reactive streams to enable real-time updates and data synchronization. With RxDB, you can easily subscribe to data changes and react to them in a reactive and efficient manner.
### Offline-First Approach
One of the standout features of RxDB is its offline-first approach. It allows you to build applications that can work seamlessly in offline scenarios. RxDB stores data locally and automatically synchronizes changes with the server when the network becomes available. This capability is particularly useful for applications that need to function in low-connectivity or unreliable network environments.
### Data Replication
RxDB provides built-in support for data replication between clients and servers. This means you can synchronize data across multiple devices or instances of your application effortlessly. RxDB handles conflict resolution and ensures that data remains consistent across all connected clients.
### Observable Queries
RxDB offers a powerful querying mechanism with support for [observable queries](../rx-query.md). This allows you to create dynamic queries that automatically update when the underlying data changes. By leveraging RxDB's observable queries, you can build reactive UI components that respond to data changes in real-time.
### Multi-Tab Support
RxDB provides out-of-the-box support for multi-tab scenarios. This means that if your Angular application is running in multiple browser tabs, RxDB automatically keeps the data in sync across all tabs. It ensures that changes made in one tab are immediately reflected in others, providing a seamless user experience.
### RxDB vs. Other Angular Database Options
While there are other database options available for Angular applications, RxDB stands out with its reactive programming model, offline-first approach, and built-in synchronization capabilities. Unlike traditional SQL databases, RxDB's NoSQL-like structure and observables-based API make it well-suited for real-time applications and complex data scenarios.
## Using RxDB in an Angular Application
Now that we have a good understanding of RxDB and its features, let's explore how to integrate it into an Angular application.
### Installing RxDB in an Angular App
To use RxDB in an Angular application, we first need to install the necessary dependencies. You can install RxDB using npm or yarn by running the following command:
```bash
npm install rxdb --save
```
Once installed, you can import RxDB into your Angular application and start using its API to create and manage databases.
### Patch Change Detection with zone.js
Angular uses change detection to detect and update UI elements when data changes. However, RxDB's data handling is based on observables, which can sometimes bypass Angular's change detection mechanism. To ensure that changes made in RxDB are detected by Angular, we need to patch the change detection mechanism using zone.js. Zone.js is a library that intercepts and tracks asynchronous operations, including observables. By patching zone.js, we can make sure that Angular is aware of changes happening in RxDB.
:::warning
RxDB creates rxjs observables outside of angulars zone
So you have to import the rxjs patch to ensure the [angular change detection](https://angular.io/guide/change-detection) works correctly.
[link](https://www.bennadel.com/blog/3448-binding-rxjs-observable-sources-outside-of-the-ngzone-in-angular-6-0-2.htm)
```ts
//> app.component.ts
import 'zone.js/plugins/zone-patch-rxjs';
```
:::
### Use the Angular async pipe to observe an RxDB Query
Angular provides the async pipe, which is a convenient way to subscribe to observables and handle the subscription lifecycle automatically. When working with RxDB, you can use the async pipe to observe an RxDB query and bind the results directly to your Angular template. This ensures that the UI stays in sync with the data changes emitted by the RxDB query.
```ts
constructor(
private dbService: DatabaseService,
private dialog: MatDialog
) {
this.heroes$ = this.dbService
.db.hero // collection
.find({ // query
selector: {},
sort: [{ name: 'asc' }]
})
.$;
}
```
```html
{{hero.name}}
```
### Different RxStorage layers for RxDB
RxDB supports multiple storage layers for persisting data. Some of the available storage options include:
- [LocalStorage RxStorage](../rx-storage-localstorage.md): Uses the [LocalStorage API](./localstorage.md) without any third party plugins.
- [IndexedDB RxStorage](../rx-storage-indexeddb.md): RxDB directly supports IndexedDB as a storage layer. IndexedDB is a low-level browser database that offers good performance and reliability.
- [OPFS RxStorage](../rx-storage-opfs.md): The OPFS [RxStorage](../rx-storage.md) for RxDB is built on top of the [File System Access API](https://webkit.org/blog/12257/the-file-system-access-api-with-origin-private-file-system/) which is available in [all modern browsers](https://caniuse.com/native-filesystem-api). It provides an API to access a sandboxed private file system to persistently store and retrieve data.
Compared to other persistent storage options in the browser (like [IndexedDB](../rx-storage-indexeddb.md)), the OPFS API has a **way better performance**.
- [Memory RxStorage](../rx-storage-memory.md): In addition to persistent storage options, RxDB also provides a memory-based storage layer. This is useful for testing or scenarios where you don't need long-term data persistence.
You can choose the storage layer that best suits your application's requirements and configure RxDB accordingly.
## Synchronizing Data with RxDB between Clients and Servers
Data replication between an Angular application and a server is a common requirement. RxDB simplifies this process and provides built-in support for data synchronization. Let's explore how to replicate data between an Angular application and a server using RxDB.
### Offline-First Approach
One of the key strengths of RxDB is its [offline-first approach](../offline-first.md). It allows Angular applications to function seamlessly even in offline scenarios. RxDB stores data locally and automatically synchronizes changes with the server when the network becomes available. This capability is particularly useful for applications that need to operate in low-connectivity or unreliable network environments.
### Conflict Resolution
In a distributed system, conflicts can arise when multiple clients modify the same data simultaneously. RxDB offers conflict resolution mechanisms to handle such scenarios. You can define conflict resolution strategies based on your application's requirements. RxDB provides hooks and events to detect conflicts and resolve them in a consistent manner.
### Bidirectional Synchronization
RxDB supports bidirectional data synchronization, allowing updates from both the client and server to be replicated seamlessly. This ensures that data remains consistent across all connected clients and the server. RxDB handles conflicts and resolves them based on the defined conflict resolution strategies.
### Real-Time Updates
RxDB provides real-time updates by leveraging reactive programming principles. Changes made to the data are automatically propagated to all connected clients in real-time. Angular applications can subscribe to these updates and update the user interface accordingly. This real-time capability enables collaborative features and enhances the overall user experience.
## Advanced RxDB Features and Techniques
RxDB offers several advanced features and techniques that can further enhance your Angular application.
### Indexing and Performance Optimization
To improve query performance, RxDB allows you to define indexes on specific fields of your documents. Indexing enables faster data retrieval and query execution, especially when working with large datasets. By strategically creating indexes, you can optimize the performance of your Angular application.
### Encryption of Local Data
RxDB provides built-in support for [encrypting](../encryption.md) local data using the Web Crypto API. With encryption, you can protect sensitive data stored in the client-side database. RxDB transparently encrypts the data, ensuring that it remains secure even if the underlying storage is compromised.
### Change Streams and Event Handling
RxDB exposes change streams, which allow you to listen for data changes at a database or collection level. By subscribing to change streams, you can react to data modifications and perform specific actions, such as updating the UI or triggering notifications. Change streams enable real-time event handling in your Angular application.
### JSON Key Compression
To reduce the storage footprint and improve performance, RxDB supports [JSON key compression](../key-compression.md). With key compression, RxDB replaces long keys with shorter aliases, reducing the overall storage size. This optimization is particularly useful when working with large datasets or frequently updating data.
## Best Practices for Using RxDB in Angular Applications
To make the most of RxDB in your Angular application, consider the following best practices:
### Use Async Pipe for Subscriptions so you do not have to unsubscribe
Angular's `async` pipe is a powerful tool for handling observables in templates. By using the async pipe, you can avoid the need to manually subscribe and unsubscribe from RxDB observables. Angular takes care of the subscription lifecycle, ensuring that resources are released when they are no longer needed. Instead of manually subscribing to Observables, you should always prefer the `async` pipe.
```ts
// WRONG:
let amount;
this.dbService
.db.hero
.find({
selector: {},
sort: [{ name: 'asc' }]
})
.$.subscribe(docs => {
amount = 0;
docs.forEach(d => amount = d.points);
});
// RIGHT:
this.amount$ = this.dbService
.db.hero
.find({
selector: {},
sort: [{ name: 'asc' }]
})
.$.pipe(
map(docs => {
let amount = 0;
docs.forEach(d => amount = d.points);
return amount;
})
);
```
### Use custom reactivity to have signals instead of rxjs observables
RxDB supports adding custom reactivity factories that allow you to get angular signals out of the database instead of rxjs observables. [read more](../reactivity.md).
### Use Angular Services for Database creation
To ensure proper separation of concerns and maintain a clean codebase, it is recommended to create an Angular service responsible for managing the RxDB database instance. This service can handle database creation, initialization, and provide methods for interacting with the database throughout your application.
### Efficient Data Handling
RxDB provides various mechanisms for efficient data handling, such as batching updates, debouncing, and throttling. Leveraging these techniques can help optimize performance and reduce unnecessary UI updates. Consider the specific data handling requirements of your application and choose the appropriate strategies provided by RxDB.
### Data Synchronization Strategies
When working with data synchronization between clients and servers, it's important to consider strategies for conflict resolution and handling network failures. RxDB provides plugins and hooks that allow you to customize the replication behavior and implement specific synchronization strategies tailored to your application's needs.
## Conclusion
RxDB is a powerful database solution for Angular applications, offering reactive data handling, offline-first capabilities, and seamless data synchronization. By integrating RxDB into your Angular application, you can build responsive and scalable web applications that provide a rich user experience. Whether you're building real-time collaborative apps, progressive web applications, or offline-capable applications, RxDB's features and techniques make it a valuable addition to your Angular development toolkit.
## Follow Up
To explore more about RxDB and leverage its capabilities for browser database development, check out the following resources:
- [RxDB GitHub Repository](https://github.com/pubkey/rxdb): Visit the official GitHub repository of RxDB to access the source code, documentation, and community support.
- [RxDB Quickstart](../quickstart.md): Get started quickly with RxDB by following the provided quickstart guide, which provides step-by-step instructions for setting up and using RxDB in your projects.
- [RxDB Angular Example at GitHub](https://github.com/pubkey/rxdb/tree/master/examples/angular)
---
## Build Smarter Offline-First Angular Apps - How RxDB Beats IndexedDB Alone
import {Tabs} from '@site/src/components/tabs';
import {Steps} from '@site/src/components/steps';
# Build Smarter Offline-First Angular Apps: How RxDB Beats IndexedDB Alone
In modern web applications, offline capabilities and fast interactions are crucial. IndexedDB, the [browser](./browser-database.md)'s built-in database, allows you to store data locally, making your Angular application more robust and responsive. However, IndexedDB can be cumbersome to work with directly. That's where RxDB (Reactive Database) shines. In this article, we'll walk you through how to utilize IndexedDB in your Angular project using [RxDB](https://rxdb.info/) as a convenient abstraction layer.
## What Is IndexedDB?
[IndexedDB](https://developer.mozilla.org/en-US/docs/Web/API/IndexedDB_API) is a low-level JavaScript API for client-side storage of large amounts of structured data. It allows you to create key-value or object store-based data storage right in the user's browser. IndexedDB supports transactions and indexing but lacks a robust query API and can be complex to use due to its callback-based nature.
## Why Use IndexedDB in Angular
- [Offline-First](../offline-first.md)/[Local-First](./local-first-future.md): If your app needs to function with limited or no internet connectivity, IndexedDB provides a reliable local storage layer. Users can continue using the application offline, and data can sync when the connection is restored.
- **Performance**: Local data access comes with [near-zero latency](./zero-latency-local-first.md), removing the need for constant server requests and eliminating most loading spinners.
- **Easier to Implement**: By replicating all necessary data to the client once, you avoid implementing numerous backend endpoints for each user interaction.
- **Scalability**: Local data queries remove processing load from your servers and reduce bandwidth usage by handling queries on the client side.
## Why Using Plain IndexedDB is a Problem
Despite the advantages, directly working with IndexedDB has several drawbacks:
- **Callback-Based**: IndexedDB was originally designed around a callback-based API, which can be unwieldy compared to modern Promise or RxJS-based flows.
- **Difficult to Implement**: IndexedDB is often described as a "low-level" API. It's more suitable for library authors rather than application developers who simply need a robust local store.
- **Rudimentary Query API**: Complex or dynamic queries are cumbersome with IndexedDB's basic get/put approach and limited indexes.
- **TypeScript Support**: Maintaining strong TypeScript types for all document structures is not straightforward with IndexedDB's untyped object stores.
- **No Observable API**: IndexedDB cannot directly emit live data changes. With RxDB, you can subscribe to changes on a collection or even a single document field.
- **Cross-Tab Synchronization**: Handling concurrent data changes across multiple browser tabs is difficult in IndexedDB. RxDB has built-in multi-tab support that keeps all tabs in sync.
- **Advanced Features Missing**: IndexedDB lacks built-in support for encryption, compression, or other advanced data management features.
- **Browser-Only**: IndexedDB works in the browser but not in environments like [React Native](../react-native-database.md) or [Electron](../electron-database.md). RxDB offers storage adapters to seamlessly reuse the same code on different platforms.
## Set Up RxDB in Angular
### Installing RxDB
You can [install RxDB](../install.md) into your Angular application via npm:
```bash
npm install rxdb --save
```
### Patch Change Detection with zone.js
RxDB creates RxJS observables outside of Angular's zone, meaning Angular won't automatically trigger change detection when new data arrives. You must patch RxJS with zone.js:
```ts
//> app.component.ts
/**
* IMPORTANT: RxDB creates rxjs observables outside of Angular's zone
* So you have to import the rxjs patch to ensure change detection works correctly.
* @link https://www.bennadel.com/blog/3448-binding-rxjs-observable-sources-outside-of-the-ngzone-in-angular-6-0-2.htm
*/
import 'zone.js/plugins/zone-patch-rxjs';
```
### Create a Database and Collections
RxDB supports multiple storage options. The free and simple approach is using the [localstorage-based](../rx-storage-localstorage.md) storage. For higher performance, there's a premium plain [IndexedDB storage](../rx-storage-indexeddb.md).
```ts
import { createRxDatabase } from 'rxdb/plugins/core';
// Define your schema
const heroSchema = {
title: 'hero schema',
version: 0,
description: 'Describes a hero in your app',
primaryKey: 'id',
type: 'object',
properties: {
id: {
type: 'string',
maxLength: 100
},
name: {
type: 'string'
},
power: {
type: 'string'
}
},
required: ['id', 'name']
};
```
### Localstorage
```ts
import { getRxStorageLocalstorage } from 'rxdb/plugins/storage-localstorage';
export async function initDB() {
// Create a database
const db = await createRxDatabase({
name: 'heroesdb', // the name of the database
storage: getRxStorageLocalstorage()
});
// Add collections
await db.addCollections({
heroes: {
schema: heroSchema
}
});
return db;
}
```
### IndexedDB
```ts
import { getRxStorageIndexedDB } from 'rxdb-premium/plugins/storage-indexeddb';
export async function initDB() {
// Create a database
const db = await createRxDatabase({
name: 'heroesdb', // the name of the database
storage: getRxStorageIndexedDB()
});
// Add collections
await db.addCollections({
heroes: {
schema: heroSchema
}
});
return db;
}
```
It's recommended to encapsulate database creation logic in an Angular service, such as in a DatabaseService. A full example is available in [RxDB's Angular example](https://github.com/pubkey/rxdb/blob/master/examples/angular/src/app/services/database.service.ts).
### CRUD Operations
Once your database is initialized, you can perform all CRUD operations:
```ts
// insert
await db.heroes.insert({ name: 'Iron Man', power: 'Genius-level intellect' });
// bulk insert
await db.heroes.bulkInsert([
{ name: 'Thor', power: 'God of Thunder' },
{ name: 'Hulk', power: 'Superhuman Strength' }
]);
// find and findOne
const heroes = await db.heroes.find().exec();
const ironMan = await db.heroes.findOne({ selector: { name: 'Iron Man' } }).exec();
// update
const doc = await db.heroes.findOne({ selector: { name: 'Hulk' } }).exec();
await doc.update({ $set: { power: 'Unlimited Strength' } });
// delete
const doc = await db.heroes.findOne({ selector: { name: 'Thor' } }).exec();
await doc.remove();
```
## Reactive Queries and Live Updates
A key benefit of RxDB is reactivity. You can subscribe to changes and have your UI automatically reflect updates in [real time](./realtime-database.md) even across browser tabs.
### With RxJS Observables and Async Pipes
In Angular, you can display this data with the `AsyncPipe`:
```ts
constructor(private dbService: DatabaseService) {
this.heroes$ = this.dbService.db.heroes.find({
selector: {},
sort: [{ name: 'asc' }]
}).$;
}
```
```html
{{ hero.name }}
```
### With Angular Signals
Angular Signals are a newer approach for reactivity. RxDB supports them via a [custom reactivity](../reactivity.md) factory. You can convert RxJS Observables to Signals using Angular's `toSignal`:
```ts
import { RxReactivityFactory } from 'rxdb/plugins/core';
import { Signal, untracked, Injector } from '@angular/core';
import { toSignal } from '@angular/core/rxjs-interop';
export function createReactivityFactory(injector: Injector): RxReactivityFactory> {
return {
fromObservable(observable$, initialValue) {
return untracked(() =>
toSignal(observable$, {
initialValue,
injector,
rejectErrors: true
})
);
}
};
}
```
Pass this factory when creating your [RxDatabase](../rx-database.md):
```ts
import { createRxDatabase } from 'rxdb/plugins/core';
import { getRxStorageLocalstorage } from 'rxdb/plugins/storage-localstorage';
import { inject, Injector } from '@angular/core';
const database = await createRxDatabase({
name: 'mydb',
storage: getRxStorageLocalstorage(),
reactivity: createReactivityFactory(inject(Injector))
});
```
Use the double-dollar sign (`$$`) to get a `Signal` instead of an `Observable`:
```ts
const heroesSignal = database.heroes.find().$$;
```
```html
{{ hero.name }}
```
## Angular IndexedDB Example with RxDB
A comprehensive example of RxDB in an Angular application is available in the [RxDB GitHub repository](https://github.com/pubkey/rxdb/tree/master/examples/angular). It demonstrates [database](./angular-database.md) creation, queries, and Angular integration using best practices.
## Advanced RxDB Features
Beyond simple CRUD and local data storage, RxDB supports:
- **Replication**: Sync your local data with a remote database. Learn more at [RxDB Replication](https://rxdb.info/replication.html).
- **Data Migration on Schema Changes**: RxDB supports automatic or manual schema migrations to manage backward-compatibility and evolve your data structure. See [RxDB Migration](https://rxdb.info/migration-schema.html).
- **Encryption**: Easily encrypt sensitive data at rest. See [RxDB Encryption](https://rxdb.info/encryption.html).
- **Compression**: Reduce storage and bandwidth usage using key compression. Learn more at [RxDB Key Compression](https://rxdb.info/key-compression.html).
## Limitations of IndexedDB
While IndexedDB works well for many use cases, it does have a few constraints:
- **Potentially Slow**: While adequate for most use cases, IndexedDB performance can degrade for very large datasets. More details at RxDB [Slow IndexedDB](../slow-indexeddb.md).
- **Storage Limits**: Browsers may cap the amount of data you can store in IndexedDB. For more info, see [Local Storage Limits of IndexedDB](./indexeddb-max-storage-limit.md).
## Alternatives to IndexedDB
Depending on your needs, you might explore:
- **Origin Private File System (OPFS)**: A newer browser storage mechanism that can offer better performance. RxDB supports [OPFS storage](../rx-storage-opfs.md).
- **SQLite**: When building a mobile or hybrid app (e.g., with [Capacitor](../capacitor-database.md) or [Ionic](./ionic-database.md)), you can use SQLite locally. See [RxDB with SQLite](../rx-storage-sqlite.md).
## Performance comparison with other browser storages
Here is a [performance overview](../rx-storage-performance.md) of the various browser based storage implementation of RxDB:
## Follow Up
Continue your deep dive into RxDB with official quickstart guides and star the repository on GitHub to stay updated.
- **RxDB Quickstart**: Get started quickly with the [RxDB Quickstart](../quickstart.md).
- **RxDB GitHub**: Explore the source, open issues, and star β the project at [RxDB GitHub Repo](https://github.com/pubkey/rxdb).
By combining IndexedDB's local storage with RxDB's powerful features, you can build performant, robust, and offline-capable Angular applications. RxDB takes care of the lower-level complexities, letting you focus on delivering a great user experience-online or off.
---
## Benefits of RxDB & Browser Databases
# RxDB: The benefits of Browser Databases
In the world of web development, efficient data management is a cornerstone of building successful and performant applications. The ability to store data directly in the browser brings numerous advantages, such as caching, offline accessibility, simplified replication of database state, and real-time application development. In this article, we will explore [RxDB](https://rxdb.info/), a powerful browser JavaScript database, and understand why it is an excellent choice for implementing a browser database solution.
## Why you might want to store data in the browser
There are compelling reasons to consider storing data in the browser:
### Use the database for caching
By leveraging a browser database, you can harness the power of caching. Storing frequently accessed data locally enables you to reduce server requests and greatly improve application performance. Caching provides a faster and smoother user experience, enhancing overall user satisfaction.
### Data is offline accessible
Storing data in the browser allows for offline accessibility. Regardless of an active internet connection, users can access and interact with the application, ensuring uninterrupted productivity and user engagement.
### Easier implementation of replicating database state
Browser databases simplify the replication of database state across multiple devices or instances of the application. Compared to complex REST routes, replicating data becomes easier and more streamlined. This capability enables the development of real-time and collaborative applications, where changes are seamlessly synchronized among users.
### Building real-time applications is easier with local data
With a local browser database, building real-time applications becomes more straightforward. The availability of local data allows for reactive data flows and dynamic user interfaces that instantly reflect changes in the underlying data. Real-time features can be seamlessly implemented, providing a rich and interactive user experience.
### Browser databases can scale better
Browser databases distribute the query workload to users' devices, allowing queries to run locally instead of relying solely on server resources. This decentralized approach improves scalability by reducing the burden on the server, resulting in a more efficient and responsive application.
### Running queries locally has low latency
Browser databases offer the advantage of running queries locally, resulting in low latency. Eliminating the need for server round-trips significantly improves query performance, ensuring faster data retrieval and a more responsive application.
### Faster initial application start time
Storing data in the browser reduces the initial application start time. Instead of waiting for data to be fetched from the server, the application can leverage the [local database](./local-database.md), resulting in faster initialization and improved user satisfaction right from the start.
### Easier integration with JavaScript frameworks
Browser databases, including [RxDB](https://rxdb.info/), seamlessly integrate with popular JavaScript frameworks such as [Angular](./angular-database.md), [React.js](./react-database.md), [Vue.js](./vue-database.md), and Svelte. This integration allows developers to leverage the power of a database while working within the familiar environment of their preferred framework, enhancing productivity and ease of development.
### Store local data with encryption
Security is a crucial aspect of data storage, especially when handling sensitive information. Browser databases, like RxDB, offer the capability to store local data with [encryption](../encryption.md), ensuring the confidentiality and protection of sensitive user data.
### Using a local database for state management
Utilizing a local browser database for state management eliminates the need for traditional state management libraries like Redux or NgRx. This approach simplifies the application's architecture by leveraging the database's capabilities to handle state-related operations efficiently.
### Data is portable and always accessible by the user
When data is stored in the browser, it becomes portable and always accessible by the user. This ensures that users have control and ownership of their data, enhancing data privacy and accessibility.
## Why SQL databases like SQLite are not a good fit for the browser
While SQL databases, such as [SQLite](../rx-storage-sqlite.md), excel in server-side scenarios, they are not always the optimal choice for browser-based applications. Here are some reasons why SQL databases may not be the best fit for the browser:
### Push/Pull based vs. reactive
SQL databases typically rely on a push/pull mechanism, where the server pushes updates to the client or the client pulls data from the server. This approach is not inherently reactive and requires additional effort to implement real-time data updates. In contrast, browser databases like [RxDB](https://rxdb.info/) provide built-in reactive mechanisms, allowing the application to react to data changes seamlessly.
### Build size of server-side databases
Server-side databases, designed to handle large-scale applications, often have significant build sizes that are unsuitable for browser applications. In contrast, browser databases are specifically optimized for browser environments and leverage browser APIs like [IndexedDB](../rx-storage-indexeddb.md), [OPFS](../rx-storage-opfs.md), and [Webworker](../rx-storage-worker.md), resulting in smaller build sizes.
### Initialization time and performance
The initialization time and performance of server-side databases can be suboptimal in browser applications. Browser databases, on the other hand, are designed to provide fast initialization and efficient performance within the browser environment, ensuring a smooth user experience.
## Why RxDB is a good fit for the browser
RxDB stands out as an excellent choice for implementing a browser database solution. Here's why RxDB is a perfect fit for browser applications:
### Observable Queries (rxjs) to automatically update the UI on changes
RxDB provides Observable Queries, powered by RxJS, enabling automatic UI updates when data changes occur. This reactive approach eliminates the need for manual data synchronization and ensures a real-time and responsive user interface.
```typescript
const query = myCollection.find({
selector: {
age: {
$gt: 21
}
}
});
const querySub = query.$.subscribe(results => {
console.log('got results: ' + results.length);
});
```
### NoSQL [JSON](./json-database.md) documents are a better fit for UIs
RxDB utilizes NoSQL [JSON documents](./json-database.md), which align naturally with UI development in JavaScript. JavaScript's native handling of JSON objects makes working with NoSQL documents more intuitive, simplifying UI-related operations.
### NoSQL has better TypeScript support compared to SQL
TypeScript is widely used in modern JavaScript development. [NoSQL databases](./in-memory-nosql-database.md), including RxDB, offer excellent TypeScript support, making it easier to build type-safe applications and leverage the benefits of static typing.
### Observable document fields
RxDB allows observing individual document fields, providing granular reactivity. This feature enables efficient tracking of specific data changes and fine-grained UI updates, optimizing performance and responsiveness.
### Made in JavaScript, optimized for JavaScript applications
RxDB is built entirely in JavaScript, optimized for JavaScript applications. This ensures seamless integration with JavaScript codebases and maximizes performance within the browser environment.
### Optimized observed queries with the EventReduce Algorithm
RxDB employs the EventReduce Algorithm to optimize observed queries. This algorithm intelligently reduces unnecessary data transmissions, resulting in efficient query execution and improved performance.
### Built-in multi-tab support
RxDB natively supports multi-tab applications, allowing data synchronization and replication across different tabs or instances of the same application. This feature ensures consistent data across the application and enhances collaboration and real-time experiences.
### Handling of schema changes
RxDB excels in handling schema changes, even when data is stored on multiple client devices. It provides mechanisms to handle schema migrations seamlessly, ensuring data integrity and compatibility as the application evolves.
### Storing documents compressed
To optimize [storage](./browser-storage.md) space, RxDB allows the [compression](../key-compression.md) of documents. Storing compressed documents reduces storage requirements and improves overall performance, especially in scenarios with large data volumes.
### Flexible storage layer for various platforms
RxDB offers a flexible storage layer, enabling code reuse across different platforms, including [Electron.js](../electron-database.md), React Native, hybrid apps (e.g., Capacitor.js), and web browsers. This flexibility streamlines development efforts and ensures consistent data management across multiple platforms.
### Replication Algorithm for compatibility with any backend
RxDB incorporates a [Replication Algorithm](../replication.md) that is open-source and can be made compatible with various backend systems. This compatibility allows seamless data synchronization with different backend architectures, such as own servers, [Firebase](../replication-firestore.md), [CouchDB](../replication-couchdb.md), [NATS](../replication-nats.md) or [WebSocket](../replication-websocket.md).
## Follow Up
To explore more about RxDB and leverage its capabilities for browser database development, check out the following resources:
- [RxDB GitHub Repository](https://github.com/pubkey/rxdb): Visit the official GitHub repository of RxDB to access the source code, documentation, and community support.
- [RxDB Quickstart](../quickstart.md): Get started quickly with RxDB by following the provided quickstart guide, which provides step-by-step instructions for setting up and using RxDB in your projects.
[RxDB](https://rxdb.info/) empowers developers to unlock the power of browser databases, enabling efficient data management, real-time applications, and enhanced user experiences. By leveraging RxDB's features and benefits, you can take your browser-based applications to the next level of performance, scalability, and responsiveness.
---
## Browser Storage - RxDB as a Database for Browsers
**Storing Data in the Browser**
When it comes to building web applications, one essential aspect is the storage of data. Two common methods of storing data directly within the user's web browser are LocalStorage and [IndexedDB](../rx-storage-indexeddb.md). These browser-based storage options serve various purposes and cater to different needs in web development.
### LocalStorage
[LocalStorage](./localstorage.md) is a straightforward way to store small amounts of data in the user's web browser. It operates on a simple key-value basis and is relatively easy to use. While it has limitations, it is suitable for basic data storage requirements.
### IndexedDB
IndexedDB, on the other hand, offers a more robust and structured approach to browser-based data storage. It can handle larger datasets and complex queries, making it a valuable choice for more advanced web applications.
## Why Store Data in the Browser
Now that we've explored the methods of storing data in the browser, let's delve into why this is a beneficial strategy for web developers:
1. **Caching**:
Storing data in the browser allows you to cache frequently used information. This means that your web application can access essential data more quickly because it doesn't need to repeatedly fetch it from a server. This results in a smoother and more responsive user experience.
2. **Offline Access**:
One significant advantage of browser storage is that data becomes portable and remains accessible even when the user is offline. This feature ensures that users can continue to use your application, view their saved information, and make changes, irrespective of their internet connection status.
3. **Faster Real-time Applications**:
For real-time applications, having data stored locally in the browser significantly enhances performance. Local data allows your application to respond faster to user interactions, creating a more seamless and responsive user interface.
4. **Low-Latency Queries**:
When you run queries locally within the browser, you minimize the latency associated with network requests. This results in near-instant access to data, which is particularly crucial for applications that require rapid data retrieval.
5. **Faster Initial Application Start Time**:
By preloading essential data into browser storage, you can reduce the initial load time of your web application. Users can start using your application more swiftly, which is essential for making a positive first impression.
6. **Store Local Data with Encryption**:
For applications that deal with sensitive data, browser storage allows you to implement [encryption](../encryption.md) to secure the stored information. This ensures that even if data is stored on the user's device, it remains confidential and protected.
In summary, storing data in the browser offers several advantages, including improved performance, offline access, and enhanced user experiences. LocalStorage and IndexedDB are two valuable tools that developers can utilize to leverage these benefits and create web applications that are more responsive and user-friendly.
## Browser Storage Limitations
While browser storage, such as LocalStorage and IndexedDB, offers many advantages, it's important to be aware of its limitations:
- **Slower Performance Compared to Native Databases**: Browser-based storage solutions can't match the [performance](../rx-storage-performance.md) of native server-side databases. They may experience slower data retrieval and processing, especially for large datasets or complex operations.
- **Storage Space Limitations**: Browsers [impose restrictions on the amount of data that can be stored locally](./indexeddb-max-storage-limit.md). This limitation can be problematic for applications with extensive data storage requirements, potentially necessitating creative solutions to manage data effectively.
## Why SQL Databases Like SQLite Aren't a Good Fit for the Browser
SQL databases like [SQLite](../rx-storage-sqlite.md), while powerful in server environments, may not be the best choice for browser-based applications due to various reasons:
### Push/Pull Based vs. Reactive
SQL databases often use a push/pull model for data synchronization. This approach is less reactive and may not align well with the real-time nature of web applications, where immediate updates to the user interface are crucial.
### Build Size of Server-Side Databases
Server-side databases like SQLite have a significant build size, which can increase the initial load time of web applications. This can result in a suboptimal user experience, particularly for users with slower internet connections.
### Initialization Time and Performance
SQL databases are optimized for server environments, and their initialization processes and performance characteristics may not align with the needs of web applications. They might not offer the swift performance required for seamless user interactions.
## Why RxDB Is a Good Fit as Browser Storage
RxDB is an excellent choice for browser-based storage due to its numerous features and advantages:
### Flexible Storage Layer for Various Platforms
RxDB offers a flexible storage layer that can seamlessly integrate with different platforms, making it versatile and adaptable to various application needs.
### NoSQL JSON Documents Are a Better Fit for UIs
NoSQL [JSON documents](./json-database.md), used by [RxDB](https://rxdb.info/), are well-suited for user interfaces. They provide a natural and efficient way to structure and display data in web applications.
### NoSQL Has Better TypeScript Support Compared to SQL
RxDB boasts robust TypeScript support, which is beneficial for developers who prefer type safety and code predictability in their projects.
### Observable Document Fields
RxDB enables developers to observe individual document fields, offering fine-grained control over data tracking and updates.
### Made in JavaScript, Optimized for JavaScript Applications
Being built in JavaScript and optimized for JavaScript applications, RxDB seamlessly integrates into web development stacks, minimizing compatibility issues.
### Observable Queries (rxjs) to Automatically Update the UI on Changes
RxDB's support for Observable Queries allows the user interface to update automatically in real-time when data changes. This reactivity enhances the user experience and simplifies UI development.
```typescript
const query = myCollection.find({
selector: {
age: {
$gt: 21
}
}
});
const querySub = query.$.subscribe(results => {
console.log('got results: ' + results.length);
});
```
### Optimized Observed Queries with the EventReduce Algorithm
RxDB's [EventReduce Algorithm](https://github.com/pubkey/event-reduce) ensures efficient data handling and rendering, improving overall performance and responsiveness.
### Handling of Schema Changes
RxDB provides built-in support for [handling schema changes](../migration-schema.md), simplifying database management when updates are required.
### Built-In Multi-Tab Support
For applications requiring multi-tab support, RxDB natively handles data consistency across different browser tabs, streamlining data synchronization.
### Storing Documents Compressed
Efficient data storage is achieved through [document compression](../key-compression.md), reducing storage space requirements and enhancing overall performance.
### Replication Algorithm for Compatibility with Any Backend
RxDB's [Replication Algorithm](../replication.md) facilitates compatibility with various backend systems, ensuring seamless data synchronization between the browser and server.
## Summary
In conclusion, RxDB is a powerful and feature-rich solution for browser-based storage. Its adaptability, real-time capabilities, TypeScript support, and optimization for JavaScript applications make it an ideal choice for modern web development projects, addressing the limitations of traditional SQL databases in the browser. Developers can harness RxDB to create efficient, responsive, and user-friendly web applications that leverage the full potential of browser storage.
## Follow Up
To explore more about RxDB and leverage its capabilities for browser storage, check out the following resources:
- [RxDB GitHub Repository](https://github.com/pubkey/rxdb): Visit the official GitHub repository of RxDB to access the source code, documentation, and community support.
- [RxDB Quickstart](../quickstart.md): Get started quickly with RxDB by following the provided quickstart guide, which provides step-by-step instructions for setting up and using RxDB in your projects.
---
## Empower Web Apps with Reactive RxDB Data-base
# RxDB as a data base: Empowering Web Applications with Reactive Data Handling
In the world of web applications, efficient data management plays a crucial role in delivering a seamless user experience. As mobile applications continue to dominate the digital landscape, the importance of robust data bases becomes evident. In this article, we will explore RxDB as a powerful data base solution for web applications. We will delve into its features, advantages, and advanced techniques, highlighting its ability to handle reactive data and enable an offline-first approach.
## Overview of Web Applications that can benefit from RxDB
Before diving into the specifics of RxDB, let's take a moment to understand the scope of web applications that can leverage its capabilities. Any web application that requires real-time data updates, offline functionality, and synchronization between clients and servers can greatly benefit from RxDB. Whether it's a collaborative document editing tool, a task management app, or a chat application, RxDB offers a robust foundation for building these types of applications.
## Importance of data bases in Mobile Applications
Mobile applications have become an integral part of our lives, providing us with instant access to information and services. Behind the scenes, data bases play a pivotal role in storing and managing the data that powers these applications. data bases enable efficient data retrieval, updates, and synchronization, ensuring a smooth user experience even in challenging network conditions.
## Introducing RxDB as a data base Solution
RxDB, short for Reactive data base, is a client-side data base solution designed specifically for web and mobile applications. Built on the principles of reactive programming, RxDB brings the power of observables and event-driven architecture to data management. With RxDB, developers can create applications that are responsive, offline-ready, and capable of seamless data synchronization between clients and servers.
## Getting Started with RxDB
### What is RxDB?
RxDB is an open-source JavaScript data base that leverages reactive programming and provides a seamless API for handling data. It is built on top of existing popular data base technologies, such as [IndexedDB](../rx-storage-indexeddb.md), and adds a layer of reactive features to enable real-time data updates and synchronization.
### Reactive Data Handling
One of the standout features of RxDB is its reactive data handling. It utilizes observables to provide a stream of data that automatically updates whenever a change occurs. This reactive approach allows developers to build applications that respond instantly to data changes, ensuring a highly interactive and real-time user experience.
### Offline-First Approach
RxDB embraces an offline-first approach, enabling applications to work seamlessly even when there is no internet connectivity. It achieves this by caching data locally on the client-side and synchronizing it with the server when the connection is available. This ensures that users can continue working with the application and have their data automatically synchronized when they come back online.
### Data Replication
RxDB simplifies the process of data replication between clients and servers. It provides replication plugins that handle the synchronization of data in real-time. These plugins allow applications to keep data consistent across multiple clients, enabling collaborative features and ensuring that each client has the most up-to-date information.
### Observable Queries
RxDB introduces the concept of observable queries, which are powerful tools for efficiently querying data. With observable queries, developers can subscribe to specific data queries and receive automatic updates whenever the underlying data changes. This eliminates the need for manual polling and ensures that applications always have access to the latest data.
### Multi-Tab support
RxDB offers multi-tab support, allowing applications to function seamlessly across multiple [browser](./browser-database.md) tabs. This feature ensures that data changes in one tab are immediately reflected in all other open tabs, enabling a consistent user experience across different browser windows.
### RxDB vs. Other data base Options
When considering data base options for web applications, developers often encounter choices like IndexedDB, [OPFS](../rx-storage-opfs.md), and Memory-based solutions. RxDB, while built on top of IndexedDB, stands out due to its reactive data handling capabilities and advanced synchronization features. Compared to other options, RxDB offers a more streamlined and powerful approach to managing data in web applications.
### Different RxStorage layers for RxDB
RxDB provides various [storage layers](../rx-storage.md), known as RxStorage, that serve as interfaces to different underlying [storage](./browser-storage.md) technologies. These layers include:
- [LocalStorage RxStorage](../rx-storage-localstorage.md): Built on top of the browsers [localStorage API](./localstorage.md).
- [IndexedDB RxStorage](../rx-storage-indexeddb.md): This layer directly utilizes IndexedDB as its backend, providing a robust and widely supported storage option.
- [OPFS RxStorage](../rx-storage-opfs.md): OPFS (Operational Transformation File System) is a file system-like storage layer that allows for efficient conflict resolution and real-time collaboration.
- Memory RxStorage: Primarily used for testing and development, this storage layer keeps data in memory without persisting it to disk.
Each RxStorage layer has its strengths and is suited for different scenarios, enabling developers to choose the most appropriate option for their specific use case.
## Synchronizing Data with RxDB between Clients and Servers
### Offline-First Approach
As mentioned earlier, RxDB adopts an offline-first approach, allowing applications to function seamlessly in disconnected environments. By caching data locally, applications can continue to operate and make updates even without an internet connection. Once the connection is restored, RxDB's replication plugins take care of synchronizing the data with the server, ensuring consistency across all clients.
### RxDB Replication Plugins
RxDB provides a range of replication plugins that simplify the process of synchronizing data between clients and servers. These plugins enable real-time replication using various protocols, such as WebSocket or HTTP, and handle conflict resolution strategies to ensure data integrity. By leveraging these replication plugins, developers can easily implement robust and scalable synchronization capabilities in their applications.
### Advanced RxDB Features and Techniques
Indexing and Performance Optimization
To achieve optimal performance, RxDB offers indexing capabilities. Indexing allows for efficient data retrieval and faster query execution. By strategically defining indexes on frequently accessed fields, developers can significantly enhance the overall performance of their RxDB-powered applications.
### Encryption of Local Data
In scenarios where data security is paramount, RxDB provides options for encrypting local data. By encrypting the data base contents, developers can ensure that sensitive information remains secure even if the underlying storage is compromised. RxDB integrates seamlessly with encryption libraries, making it easy to implement end-to-end encryption in applications.
### Change Streams and Event Handling
RxDB offers change streams and event handling mechanisms, enabling developers to react to data changes in real-time. With change streams, applications can listen to specific collections or documents and trigger custom logic whenever a change occurs. This capability opens up possibilities for building real-time collaboration features, notifications, or other reactive behaviors.
### JSON Key Compression
In scenarios where storage size is a concern, RxDB provides JSON [key compression](../key-compression.md). By applying compression techniques to JSON keys, developers can significantly reduce the storage footprint of their data bases. This feature is particularly beneficial for applications dealing with large datasets or [limited storage capacities](./indexeddb-max-storage-limit.md).
## Conclusion
RxDB provides an exceptional data base solution for web and mobile applications, empowering developers to create reactive, offline-ready, and synchronized applications. With its reactive data handling, offline-first approach, and replication plugins, RxDB simplifies the challenges of building real-time applications with data synchronization requirements. By embracing advanced features like indexing, encryption, change streams, and JSON key compression, developers can optimize performance, enhance security, and reduce storage requirements. As web and [mobile applications](./mobile-database.md) continue to evolve, RxDB proves to be a reliable and powerful
---
## Embedded Database, Real-time Speed - RxDB
# Using RxDB as an Embedded Database
In modern UI applications, efficient data storage is a crucial aspect for seamless user experiences. One powerful solution for achieving this is by utilizing an embedded database. In this article, we will explore the concept of an embedded database and delve into the benefits of using [RxDB](https://rxdb.info/) as an embedded database in UI applications. We will also discuss why RxDB stands out as a robust choice for real-time applications with embedded database functionality.
## What is an Embedded Database?
An embedded database refers to a client-side database system that is integrated directly within an application. It is designed to operate within the client environment, such as a web browser or a [mobile](./mobile-database.md) app. This approach eliminates the need for a separate database server and allows the database to run locally on the client device.
## Embedded Database in UI Applications
In the context of UI applications, an embedded database serves as a local data storage solution. It enables applications to efficiently manage data, facilitate real-time updates, and enhance performance. Let's explore some of the benefits of using an embedded database compared to a traditional server database:
- Replicating the database state becomes easier: Implementing real-time data synchronization and replication is simpler with an embedded database compared to complex REST routes. The embedded nature allows for efficient replication of the database state across multiple instances of the application.
- Using the database for caching: An embedded database can be utilized for caching frequently accessed data. This caching mechanism enhances performance and reduces the need for repeated network requests, resulting in faster data retrieval.
- Building real-time applications is easier with local data: By leveraging local data storage, real-time applications can easily update the user interface in response to data changes. This approach simplifies the development of real-time features and enhances the responsiveness of the application.
- Store local data with [encryption](../encryption.md): Embedded databases, like RxDB, offer the ability to store local data with encryption. This ensures that sensitive information remains protected even when stored locally on the client device.
- Data is accessible offline: With an embedded database, data remains accessible even when the application is offline. Users can continue to interact with the application and access their data seamlessly, irrespective of their internet connectivity.
- Faster initial application start time: Since the data is already stored locally, there is no need for initial data fetching from a remote server. This significantly reduces the application's startup time and allows users to engage with the application more quickly.
- Improved scalability with local queries: Embedded databases, such as RxDB, perform queries locally on the client device instead of relying on server round-trips. This reduces latency and enhances scalability, particularly when dealing with large datasets or high query volumes.
- Seamless integration with JavaScript frameworks: Embedded databases, including RxDB, integrate seamlessly with popular JavaScript frameworks like Angular, React.js, [Vue.js](./vue-database.md), and Svelte. This compatibility allows developers to leverage the capabilities of these frameworks while benefiting from embedded database functionality.
- Running queries locally has low latency: With an embedded database, queries are executed locally on the client device, resulting in minimal latency. This improves the overall performance and responsiveness of the application.
- Data is portable and always accessible by the user: Embedded databases enable data portability, allowing users to seamlessly transition between devices while maintaining their data and application state. This ensures that data is always accessible and available to the user.
- Using a [local database](./local-database.md) for state management: Instead of relying on additional state management libraries like Redux or NgRx, an embedded database can be used for local state management. This simplifies state management and ensures data consistency within the application.
## Why RxDB as an Embedded Database for Real-time Applications
RxDB is a JavaScript-based embedded database that offers numerous advantages for building real-time applications. Let's explore why RxDB is a compelling choice:
- [Observable Queries](../rx-query.md) (RxJS): RxDB leverages the power of Observables through RxJS, enabling developers to create queries that automatically update the user interface on data changes. This reactive approach simplifies UI updates and ensures real-time synchronization of data.
- [NoSQL JSON Documents](./json-database.md) for UIs: RxDB utilizes NoSQL (JSON) documents as its data model, aligning seamlessly with the requirements of modern UI development. JavaScript's native support for JSON objects makes NoSQL documents a natural fit for UI-driven applications.
- Better TypeScript Support Compared to SQL: RxDB's NoSQL approach provides excellent TypeScript support. The flexibility of working with JSON objects enables robust typing and enhanced development experiences, ensuring type safety and reducing runtime errors.
- [Observable Document Fields](../rx-document.md): RxDB allows developers to observe individual fields within documents. This granularity enables efficient tracking of specific data changes and facilitates targeted UI updates, enhancing performance and responsiveness.
- Made in JavaScript, Optimized for JavaScript Applications: Being built entirely in JavaScript, RxDB is optimized for JavaScript applications. It leverages JavaScript's capabilities and integrates seamlessly with JavaScript frameworks and libraries, making it a natural choice for JavaScript developers.
- Optimized Observed Queries with the [EventReduce Algorithm](https://github.com/pubkey/event-reduce): RxDB incorporates the EventReduce algorithm to optimize observed queries. This algorithm reduces the number of emitted events during query execution, resulting in enhanced query performance and reduced overhead.
- Built-in Multi-tab Support: RxDB provides built-in multi-tab support, allowing multiple instances of an application to share and synchronize data seamlessly. This feature enables collaborative and real-time scenarios across multiple browser tabs or windows.
- Handling of Schema Changes across Multiple Client Devices: With RxDB, handling schema changes across multiple client devices becomes straightforward. RxDB's schema [migration capabilities](../migration-schema.md) ensure that applications can seamlessly adapt to evolving data structures, providing a consistent experience across different devices.
- Storing Documents Compressed: RxDB offers the ability to store documents in a compressed format. This reduces the storage footprint and improves performance, especially when dealing with large datasets.
- Flexible Storage Layer and Cross-Platform Compatibility: RxDB provides a flexible storage layer that can be reused across various platforms, including [Electron.js](../electron-database.md), [React Native](../react-native-database.md), hybrid apps (via Capacitor.js), and browsers. This cross-platform compatibility simplifies development and enables code reuse across different environments.
- Replication Algorithm for Backend Compatibility: RxDB's replication algorithm is open-source and can be made compatible with various backend solutions, such as self-hosted servers, Firebase, [CouchDB](../replication-couchdb.md), NATS, WebSockets, and more. This flexibility allows developers to choose their preferred backend infrastructure while benefiting from RxDB's embedded database capabilities.
## Follow Up
To further explore [RxDB](https://rxdb.info/) and leverage its capabilities as an embedded database, the following resources can be helpful:
- [RxDB GitHub Repository](https://github.com/pubkey/rxdb): Visit the official GitHub repository of RxDB to access the source code, documentation, and community support.
- [RxDB Quickstart](../quickstart.md): Get started quickly with RxDB by following the provided quickstart guide, which offers step-by-step instructions for setting up and using RxDB in your projects.
By utilizing [RxDB](https://rxdb.info/) as an embedded database in UI applications, developers can harness the power of efficient data management, real-time updates, and enhanced user experiences. RxDB's features and benefits make it a compelling choice for building modern, responsive, and scalable applications.
---
## RxDB - Firebase Realtime Database Alternative to Sync With Your Own Backend
# RxDB - The Firebase Realtime Database Alternative That Can Sync With Your Own Backend
Are you on the lookout for a **Firebase Realtime Database alternative** that gives you greater freedom, deeper offline capabilities, and allows you to seamlessly integrate with any backend? **RxDB** (Reactive Database) might be the perfect choice. This [local-first](./local-first-future.md), NoSQL data store runs entirely on the client while supporting real-time updates and robust syncing with any server environmentβmaking it a strong contender against Firebase Realtime Database's limitations and potential vendor lock-in.
## Why RxDB Is an Excellent Firebase Realtime Database Alternative
### 1. Complete Offline-First Experience
Unlike Firebase Realtime Database, which relies on central infrastructure to process data, RxDB is fully embedded within your client application (including [browsers](./browser-database.md), [Node.js](../nodejs-database.md), [Electron](../electron-database.md), and [React Native](../react-native-database.md)). This design means your app stays completely functional offline, since all data reads and writes happen locally. When connectivity is restored, RxDB's syncing framework automatically reconciles local changes with your remote backend.
### 2. Freedom to Use Any Server or Cloud
While Firebase Realtime Database ties you into Google's ecosystem, RxDB allows you to choose any hosting environment. You can:
- Host your data on your own servers or private cloud.
- Integrate with relational databases like [PostgreSQL](../replication-http.md) or other NoSQL options such as [CouchDB](../replication-couchdb.md).
- Build custom endpoints using [REST](../replication-http.md), [GraphQL](../replication-graphql.md), or any other protocol.
This flexibility ensures you're not locked into a single vendor and can adapt your backend strategy as your project evolves.
### 3. Advanced Conflict Handling
Firebase Realtime Database typically updates data with a simple last-in-wins approach. RxDB, on the other hand, lets you implement more sophisticated conflict resolution logic. Using [revisions and conflict handlers](../transactions-conflicts-revisions.md#custom-conflict-handler), RxDB can merge concurrent edits or preserve multiple versionsβensuring your application remains consistent even when multiple clients modify the same data at the same time.
### 4. Lower Cloud Costs for Read-Heavy Apps
When you rely on Firebase Realtime Database, each query or listener can translate into ongoing reads, potentially running up your monthly bill. With RxDB, all queries are performed [locally](../offline-first.md). Your app only communicates with the backend to sync document changes, significantly reducing bandwidth and hosting expenses for applications that frequently read data.
### 5. Powerful Local Queries
If you've hit Firebase Realtime Database's querying limits, RxDB offers a far more robust approach to data retrieval. You can:
- Define custom indexes for faster local lookups.
- Perform sophisticated filters, joins, or full-text searches right on the client.
- Subscribe to real-time data updates through RxDB's [reactive query engine](../reactivity.md).
Because these operations happen locally, your [UI updates](./optimistic-ui.md) instantly, providing a snappy user experience.
### 6. True Offline Initialization
While Firebase offers some offline caching, it often requires an initial connection for authentication or to seed local data. RxDB, however, is built to handle an **offline-start** scenario. Users can begin working with the application immediately, regardless of connectivity, and any modifications they make will sync once the network is available again.
### 7. Works Everywhere JavaScript Runs
One of RxDB's core strengths is its ability to run in **any JavaScript environment**. Whether you're building a web app that uses IndexedDB in the browser, an [Electron](../electron-database.md) desktop program, or a [React Native](../react-native-database.md) mobile application, RxDB's **swappable storage** adapts to your runtime of choice. This consistency makes code-sharing and cross-platform development far simpler than being tied to a single backend system.
---
## How RxDB's Syncing Mechanism Operates
RxDB employs its own [Sync Engine](../replication.md) to manage data flow between your client and remote [servers](../rx-server.md). Replication revolves around:
1. **Pull**: Retrieving updated or newly created documents from the server.
2. **Push**: Sending local changes to the backend for persistence.
3. **Live Updates**: Continuously streaming changes to and from the backend for real-time synchronization.
## Sample Code: Sync RxDB With a Custom Endpoint
```ts
import { createRxDatabase } from 'rxdb/plugins/core';
import { getRxStorageLocalstorage } from 'rxdb/plugins/storage-localstorage';
import { replicateRxCollection } from 'rxdb/plugins/replication';
async function initDB() {
const db = await createRxDatabase({
name: 'localdb',
storage: getRxStorageLocalstorage(),
multiInstance: true,
eventReduce: true
});
await db.addCollections({
tasks: {
schema: {
title: 'task schema',
version: 0,
type: 'object',
primaryKey: 'id',
properties: {
id: { type: 'string', maxLength: 100 },
title: { type: 'string' },
complete: { type: 'boolean' }
}
}
}
});
// Start a custom replication
replicateRxCollection({
collection: db.tasks,
replicationIdentifier: 'custom-tasks-api',
push: {
handler: async (docs) => {
// post local changes to your server
const resp = await fetch('https://yourapi.com/tasks/push', {
method: 'POST',
body: JSON.stringify({ changes: docs })
});
return await resp.json(); // return conflicting documents if any
}
},
pull: {
handler: async (lastCheckpoint, batchSize) => {
// fetch new/updated items from your server
const response = await fetch(
`https://yourapi.com/tasks/pull?checkpoint=${JSON.stringify(
lastCheckpoint
)}&limit=${batchSize}`
);
return await response.json();
}
},
live: true
});
return db;
}
```
### Setting Up P2P Replication Over WebRTC
In addition to using a centralized backend, RxDB supports peer-to-peer synchronization through WebRTC, enabling devices to share data directly.
```ts
import {
replicateWebRTC,
getConnectionHandlerSimplePeer
} from 'rxdb/plugins/replication-webrtc';
const webrtcPool = await replicateWebRTC({
collection: db.tasks,
topic: 'p2p-topic-123',
connectionHandlerCreator: getConnectionHandlerSimplePeer({
signalingServerUrl: 'wss://signaling.rxdb.info/',
wrtc: require('node-datachannel/polyfill'),
webSocketConstructor: require('ws').WebSocket
})
});
webrtcPool.error$.subscribe((error) => {
console.error('P2P error:', error);
});
```
Here, any client that joins the same topic communicates changes to other peers, all without requiring a traditional client-server model.
## Quick Steps to Get Started
1. Install RxDB
```bash
npm install rxdb rxjs
```
2. Create a Local Database
```ts
import { createRxDatabase } from 'rxdb/plugins/core';
import { getRxStorageLocalstorage } from 'rxdb/plugins/storage-localstorage';
const db = await createRxDatabase({
name: 'myLocalDB',
storage: getRxStorageLocalstorage()
});
Add a Collection
ts
Kopieren
await db.addCollections({
notes: {
schema: {
title: 'notes schema',
version: 0,
primaryKey: 'id',
type: 'object',
properties: {
id: { type: 'string', maxLenght: 100 },
content: { type: 'string' }
}
}
}
});
```
3. Synchronize
Use one of the [Replication Plugins](../replication.md) to connect with your preferred backend.
### Is RxDB the Right Solution for You?
- **Long Offline Use**: If your users need to work without an internet connection, RxDB's built-in offline-first design stands out compared to Firebase Realtime Database's partial offline approach.
- **Custom or Complex Queries**: RxDB lets you perform your [queries](../rx-query.md) locally, define [indexing](../rx-schema.md#indexes), and handle even complex [transformations](../rx-pipeline.md) locally - no extra call to an external API.
- **Avoid Vendor Lock-In**: If you anticipate needing to move or adapt your backend later, you can do so without rewriting how your client manages its data.
- **Peer-to-Peer Collaboration**: Whether you need quick demos or real production use, [WebRTC replication](../replication-webrtc.md) can link your users directly without central coordination of data storage.
---
## RxDB - Firestore Alternative to Sync with Your Own Backend
# RxDB - The Firestore Alternative That Can Sync with Your Own Backend
If you're seeking a **Firestore alternative**, you're likely looking for a way to:
- **Avoid vendor lock-in** while still enjoying real-time replication.
- **Reduce cloud usage costs** by reading data locally instead of constantly fetching from the server.
- **Customize** how you store, query, and secure your data.
- **Implement advanced conflict resolution** strategies beyond Firestore's last-write-wins approach.
Enter **RxDB** (Reactive Database) - a [local-first](./local-first-future.md), NoSQL database for JavaScript applications that can sync in real time with **any** backend of your choice. Whether you're tired of the limitations and fees associated with Firebase Cloud Firestore or simply need more flexibility, RxDB might be the Firestore alternative you've been searching for.
## What Makes RxDB a Great Firestore Alternative?
Firestore is convenient for many projects, but it does lock you into Google's ecosystem. Below are some of the key advantages you gain by choosing RxDB:
### 1. Fully Offline-First
RxDB runs directly in your client application ([browser](./browser-database.md), [Node.js](../nodejs-database.md), [Electron](../electron-database.md), [React Native](../react-native-database.md), etc.). Data is stored locally, so your application **remains fully functional even when offline**. When the device returns online, RxDB's flexible replication protocol synchronizes your local changes with any remote endpoint.
### 2. Freedom to Use Any Backend
Unlike Firestore, RxDB doesn't require a proprietary hosting service. You can:
- Host your data on your own server (Node.js, Go, Python, etc.).
- Use existing databases like [PostgreSQL](../replication-http.md), [CouchDB](../replication-couchdb.md), or [MongoDB with custom endpoints](../replication.md).
- Implement a [custom GraphQL](../replication-graphql.md) or [REST-based](../replication-http.md) API for syncing.
This **backend-agnostic** approach protects you from vendor lock-in. Your application's client-side data storage remains consistent; only your replication logic (or plugin) changes if you switch servers.
### 3. Advanced Conflict Resolution
Firestore enforces a [last-write-wins](https://stackoverflow.com/a/47781502/3443137) conflict resolution strategy. This might cause issues if multiple users or devices update the same data in complex ways.
RxDB lets you:
- Implement **custom conflict resolution** via [revisions](../transactions-conflicts-revisions.md#custom-conflict-handler).
- Store partial merges, track versions, or preserve multiple user edits.
- Fine-tune how your data merges to ensure consistency across distributed systems.
### 4. Reduced Cloud Costs
Firestore queries often count as billable reads. With RxDB, queries run **locally** against your local state - no repeated network calls or extra charges. You pay only for the data actually synced, not every read. For **read-heavy** apps, using RxDB as a Firestore alternative can significantly reduce costs.
### 5. No Limits on Query Features
Firestore's query engine is limited by certain constraints (e.g., no advanced joins, limited indexing). With RxDB:
- **NoSQL** data is stored locally, and you can define any indexes you need.
- Perform [complex queries](../rx-query.md), run [full-text search](../fulltext-search.md), or do aggregated transformations or even [vector search](./javascript-vector-database.md).
- Use [RxDB's reactivity](../rx-query.md#observe) to subscribe to query results in real time.
### 6. True Offline-Start Support
While Firestore does have offline caching, it often requires an online check at app initialization for authentication. RxDB is [truly offline-first](../offline-first.md); you can launch the app and write data even if the device never goes online initially. It's ready whenever the user is.
### 7. Cross-Platform: Any JavaScript Runtime
RxDB is designed to run in **any environment** that can execute JavaScript. Whether youβre building a web app in the browser, an [Electron](../electron-database.md) desktop application, a [React Native](../react-native-database.md) mobile app, or a command-line tool with [Node.js](../nodejs-database.md), RxDBβs storage layer is swappable to fit your runtimeβs capabilities.
- In the **browser**, store data in [IndexedDB](../rx-storage-indexeddb.md) or [OPFS](../rx-storage-opfs.md).
- In [Node.js](../nodejs-database.md), use LevelDB or other supported storages.
- In [React Native](../react-native-database.md), pick from a range of adapters suited for mobile devices.
- In [Electron](../electron-database.md), rely on fast local storage with zero changes to your application code.
---
## How Does RxDB's Sync Work?
RxDB replication is powered by its own [Sync Engine](../replication.md). This simple yet robust protocol enables:
1. **Pull**: Fetch new or updated documents from the server.
2. **Push**: Send local changes back to the server.
3. **Live Real-Time**: Once you're caught up, you can opt for event-based streaming instead of continuous polling.
Code Example: Sync RxDB with a Custom Backend
```ts
import { createRxDatabase } from 'rxdb/plugins/core';
import { getRxStorageLocalstorage } from 'rxdb/plugins/storage-localstorage';
import { replicateRxCollection } from 'rxdb/plugins/replication';
async function initDB() {
const db = await createRxDatabase({
name: 'mydb',
storage: getRxStorageLocalstorage(),
multiInstance: true,
eventReduce: true
});
await db.addCollections({
tasks: {
schema: {
title: 'task schema',
version: 0,
type: 'object',
primaryKey: 'id',
properties: {
id: { type: 'string', maxLength: 100 },
title: { type: 'string' },
done: { type: 'boolean' }
}
}
}
});
// Start a custom REST-based replication
replicateRxCollection({
collection: db.tasks,
replicationIdentifier: 'my-tasks-rest-api',
push: {
handler: async (documents) => {
// Send docs to your REST endpoint
const res = await fetch('https://myapi.com/push', {
method: 'POST',
body: JSON.stringify({ docs: documents })
});
// Return conflicts if any
return await res.json();
}
},
pull: {
handler: async (lastCheckpoint, batchSize) => {
// Fetch from your REST endpoint
const res = await fetch(`https://myapi.com/pull?checkpoint=${JSON.stringify(lastCheckpoint)}&limit=${batchSize}`);
return await res.json();
}
},
live: true // keep watching for changes
});
return db;
}
```
By swapping out the handler implementations or using an official plugin (e.g., [GraphQL](../replication-graphql.md), [CouchDB](../replication-couchdb.md), [Firestore replication](../replication-firestore.md), etc.), you can adapt to any backend or data source. RxDB thus becomes a flexible alternative to Firestore while maintaining [real-time capabilities](./realtime-database.md).
## Getting Started with RxDB as a Firestore Alternative
### Install RxDB:
```bash
npm install rxdb rxjs
```
### Create a Database:
```ts
import { createRxDatabase } from 'rxdb/plugins/core';
import { getRxStorageLocalstorage } from 'rxdb/plugins/storage-localstorage';
const db = await createRxDatabase({
name: 'mydb',
storage: getRxStorageLocalstorage()
});
```
### Define Collections:
```ts
await db.addCollections({
items: {
schema: {
title: 'items schema',
version: 0,
primaryKey: 'id',
type: 'object',
properties: {
id: { type: 'string', maxLength: 100 },
text: { type: 'string' }
}
}
}
});
```
### Sync
Use a [Replication Plugin](../replication.md) to connect with a custom backend or existing database.
For a Firestore-specific approach, RxDB [Firestore Replication](../replication-firestore.md) also exists if you want to combine local indexing and advanced queries with a Cloud Firestore backend. But if you really want to replace Firestore entirely - just point RxDB to your new backend.
### Example: Start a WebRTC P2P Replication
In addition to syncing with a central server, RxDB also supports pure peer-to-peer replication using [WebRTC](../replication-webrtc.md). This can be invaluable for scenarios where clients need to sync data directly without a master server.
```ts
import {
replicateWebRTC,
getConnectionHandlerSimplePeer
} from 'rxdb/plugins/replication-webrtc';
const replicationPool = await replicateWebRTC({
collection: db.tasks,
topic: 'my-p2p-room', // Clients with the same topic will sync with each other.
connectionHandlerCreator: getConnectionHandlerSimplePeer({
// Use your own or the official RxDB signaling server
signalingServerUrl: 'wss://signaling.rxdb.info/',
// Node.js requires a polyfill for WebRTC & WebSocket
wrtc: require('node-datachannel/polyfill'),
webSocketConstructor: require('ws').WebSocket
}),
pull: {}, // optional pull config
push: {} // optional push config
});
// The replicationPool manages all connected peers
replicationPool.error$.subscribe(err => {
console.error('P2P Sync Error:', err);
});
```
This example sets up a live **P2P replication** where any new peers joining the same topic automatically sync local data with each other, eliminating the need for a dedicated central server for the actual data exchange.
## Is RxDB Right for Your Project?
- **You want offline-first**: If you need an offline-first app that starts offline, RxDB's local database approach and sync protocol excel at this.
- **Your project is read-heavy**: Reading from Firestore for every query can get expensive. With RxDB, reads are free and local; you only pay for writes or sync overhead.
- **You need advanced queries**: Firestore's query constraints may not suit complex data. With RxDB, you can define your own indexing logic or run arbitrary queries locally.
- **You want no vendor lock-in**: Easily transition from Firestore to your own server or another vendor - just change the replication layer.
## Follow Up
If you've been searching for a Firestore alternative that gives you the freedom to sync your data with any backend, offers robust offline-first capabilities, and supports truly customizable conflict resolution and queries, RxDB is worth exploring. You can adopt it seamlessly, ensure local reads, reduce costs, and stay in complete control of your data layer.
Ready to dive in? Check out the RxDB Quickstart Guide, join our Discord community, and experience how RxDB can be the perfect local-first, real-time database solution for your next project.
More resources:
- [RxDB Sync Engine](../replication.md)
- [Firestore Replication Plugin](../replication-firestore.md)
- [Custom Conflict Resolution](../transactions-conflicts-revisions.md)
- [RxDB GitHub Repository](/code/)
---
## Supercharge Flutter Apps with the RxDB Database
# RxDB as a Database in a Flutter Application
In the world of mobile application development, Flutter has gained significant popularity due to its cross-platform capabilities and rich UI framework. When it comes to building feature-rich Flutter applications, the choice of a robust and efficient database is crucial. In this article, we will explore [RxDB](https://rxdb.info/) as a database solution for Flutter applications. We'll delve into the core features of RxDB, its benefits over other database options, and how to integrate it into a Flutter app.
:::note
You can find the source code for an example RxDB Flutter Application [at the github repo](https://github.com/pubkey/rxdb/tree/master/examples/flutter)
:::
### Overview of Flutter Mobile Applications
Flutter is an open-source UI software development kit created by Google that allows developers to build high-performance [mobile](./mobile-database.md) applications for iOS and Android platforms using a single codebase. Flutter's framework provides a wide range of widgets and tools that enable developers to create visually appealing and responsive applications.
### Importance of Databases in Flutter Applications
Databases play a vital role in Flutter applications by providing a persistent and reliable storage solution for storing and retrieving data. Whether it's user profiles, app settings, or complex data structures, a database helps in efficiently managing and organizing the application's data. Choosing the right database for a Flutter application can significantly impact the performance, scalability, and user experience of the app.
### Introducing RxDB as a Database Solution
RxDB is a powerful NoSQL database solution that is designed to work seamlessly with JavaScript-based frameworks, such as Flutter. It stands for Reactive Database and offers a variety of features that make it an excellent choice for building Flutter applications. RxDB combines the simplicity of JavaScript's document-based database model with the reactive programming paradigm, enabling developers to build real-time and offline-first applications with ease.
## Getting Started with RxDB
To understand how RxDB can be utilized in a Flutter application, let's explore its core features and advantages.
### What is RxDB?
[RxDB](https://rxdb.info/) is a client-side database built on top of [IndexedDB](../rx-storage-indexeddb.md), which is a low-level [browser-based database](./browser-database.md) API. It provides a simple and intuitive API for performing CRUD operations (Create, Read, Update, Delete) on documents. RxDB's underlying architecture allows for efficient handling of data synchronization between multiple clients and servers.
### Reactive Data Handling
One of the key strengths of RxDB is its reactive data handling. It leverages the power of Observables, a concept from reactive programming, to automatically update the UI in response to data changes. With RxDB, developers can define queries and subscribe to their results, ensuring that the UI is always in sync with the database.
### Offline-First Approach
RxDB follows an offline-first approach, making it ideal for building Flutter applications that need to function even without an internet connection. It allows data to be stored locally and seamlessly synchronizes it with the server when a connection is available. This ensures that users can access and interact with their data regardless of network availability.
### Data Replication
Data replication is a critical aspect of building distributed applications. RxDB provides robust replication capabilities that enable synchronization of data between different clients and servers. With its replication plugins, RxDB simplifies the process of setting up real-time data synchronization, ensuring consistency across all connected devices.
### [Observable Queries](../rx-query.md)
RxDB introduces the concept of observable queries, which are queries that automatically update when the underlying data changes. This feature is particularly useful for keeping the UI up to date with the latest data. By subscribing to an observable query, developers can receive real-time updates and reflect them in the user interface without manual intervention.
### RxDB vs. Other Flutter Database Options
When considering database options for Flutter applications, developers often come across alternatives such as SQLite or LokiJS. While these databases have their merits, RxDB offers several advantages over them. RxDB's seamless integration with Flutter, its offline-first approach, reactive data handling, and built-in data replication make it a compelling choice for building feature-rich and scalable Flutter applications.
## Using RxDB in a Flutter Application
Now that we understand the core features of RxDB, let's explore how to integrate it into a Flutter application.
## How RxDB can run in Flutter
RxDB is written in TypeScript and compiled to JavaScript. To run it in a Flutter application, the `flutter_qjs` library is used to spawn a QuickJS JavaScript runtime. RxDB itself runs in that runtime and communicates with the flutter dart runtime. To store data persistent, the [LokiJS RxStorage](../rx-storage-lokijs.md) is used together with a custom storage adapter that persists the database inside of the `shared_preferences` data.
To use RxDB, you have to create a compatible JavaScript file that creates your RxDatabase and starts some connectors which are used by Flutter to communicate with the JavaScript RxDB database via setFlutterRxDatabaseConnector().
```javascript
import {
createRxDatabase
} from 'rxdb';
import {
getRxStorageLoki
} from 'rxdb/plugins/storage-lokijs';
import {
setFlutterRxDatabaseConnector,
getLokijsAdapterFlutter
} from 'rxdb/plugins/flutter';
// do all database creation stuff in this method.
async function createDB(databaseName) {
// create the RxDatabase
const db = await createRxDatabase({
// the database.name is variable so we can change it on the flutter side
name: databaseName,
storage: getRxStorageLoki({
adapter: getLokijsAdapterFlutter()
}),
multiInstance: false
});
await db.addCollections({
heroes: {
schema: {
version: 0,
primaryKey: 'id',
type: 'object',
properties: {
id: {
type: 'string',
maxLength: 100
},
name: {
type: 'string',
maxLength: 100
},
color: {
type: 'string',
maxLength: 30
}
},
indexes: ['name'],
required: ['id', 'name', 'color']
}
}
});
return db;
}
// start the connector so that flutter can communicate with the JavaScript process
setFlutterRxDatabaseConnector(
createDB
);
```
Before you can use the JavaScript code, you have to bundle it into a single .js file. In this example we do that with webpack in a npm script here which bundles everything into the `javascript/dist/index.js` file.
To allow Flutter to access that file during runtime, add it to the assets inside of your pubspec.yaml:
```yaml
flutter:
assets:
- javascript/dist/index.js
```
Also you need to install RxDB in your flutter part of the application. First you have to use the rxdb dart package as a flutter dependency. Currently the package is not published at the dart pub.dev. Instead you have to install it from the local filesystem inside of your RxDB npm installation.
```yaml
# inside of pubspec.yaml
dependencies:
rxdb:
path: path/to/your/node_modules/rxdb/src/plugins/flutter/dart
```
Afterwards you can import the rxdb library in your dart code and connect to the JavaScript process from there. For reference, check out the lib/main.dart file.
```dart
import 'package:rxdb/rxdb.dart';
// start the javascript process and connect to the database
RxDatabase database = await getRxDatabase("javascript/dist/index.js", databaseName);
// get a collection
RxCollection collection = database.getCollection('heroes');
// insert a document
RxDocument document = await collection.insert({
"id": "zflutter-${DateTime.now()}",
"name": nameController.text,
"color": colorController.text
});
// create a query
RxQuery query = RxDatabaseState.collection.find();
// create list to store query results
List> documents = [];
// subscribe to a query
query.$().listen((results) {
setState(() {
documents = results;
});
});
```
### Different RxStorage layers for RxDB
RxDB offers multiple storage options, known as RxStorage layers, to store data locally. These options include:
- [LokiJS RxStorage](../rx-storage-lokijs.md): LokiJS is an in-memory database that can be used as a [storage](./browser-storage.md) layer for RxDB. It provides fast and efficient in-memory data management capabilities.
- [SQLite RxStorage](../rx-storage-sqlite.md): SQLite is a popular and widely used [embedded database](./embedded-database.md) that offers robust storage capabilities. RxDB utilizes SQLite as a storage layer to persist data on the device.
- [Memory RxStorage](../rx-storage-memory.md): As the name suggests, Memory RxStorage stores data [in memory](./in-memory-nosql-database.md). While this option does not provide persistence, it can be useful for temporary or cache-based data storage.
By choosing the appropriate RxStorage layer based on the specific requirements of the application, developers can optimize performance and storage efficiency.
## Synchronizing Data with RxDB between Clients and Servers
One of the key strengths of RxDB is its ability to synchronize data between multiple clients and servers seamlessly. Let's explore how this synchronization can be achieved.
### Offline-First Approach
RxDB's offline-first approach ensures that data can be accessed and modified even when there is no internet connection. Changes made offline are automatically synchronized with the server once a connection is reestablished. This ensures data consistency across all devices, providing a seamless user experience.
### RxDB Replication Plugins
RxDB provides replication plugins that simplify the process of setting up data [synchronization between clients and servers](../replication.md). These plugins offer various synchronization strategies, such as one-way replication, two-way replication, and conflict resolution mechanisms. By configuring the appropriate replication plugin, developers can easily establish real-time data synchronization in their Flutter applications.
## Advanced RxDB Features and Techniques
RxDB offers a range of advanced features and techniques that enhance its functionality and performance. Let's explore a few of these features:
### Indexing and Performance Optimization
Indexing is a technique used to optimize query performance by creating indexes on specific fields. RxDB allows developers to define indexes on document fields, improving the efficiency of queries and data retrieval.
### Encryption of Local Data
To ensure data privacy and security, RxDB supports [encryption of local data](../encryption.md). By encrypting the data stored on the device, developers can protect sensitive information and prevent unauthorized access.
### Change Streams and Event Handling
RxDB provides change streams, which emit events whenever data changes occur. By leveraging change streams, developers can implement custom event handling logic, such as updating the UI or triggering background processes, in response to specific data changes.
### JSON Key Compression
To minimize storage requirements and optimize performance, RxDB offers [JSON key compression](../key-compression.md). This feature reduces the size of keys used in the database, resulting in more efficient storage and improved query performance.
## Conclusion
RxDB offers a powerful and flexible database solution for Flutter applications. With its offline-first approach, real-time data synchronization, and reactive data handling capabilities, RxDB simplifies the development of feature-rich and scalable Flutter applications. By integrating RxDB into your Flutter projects, you can leverage its advanced features and techniques to build responsive and data-driven applications that provide an exceptional user experience.
## FAQ
What is the best local-first database for Flutter apps?
RxDB provides the best local-first database for Flutter applications. You gain full reactive data handling where observable queries automatically update your Flutter UI. The system stores data locally to ensure complete application functionality without an internet connection. Replication plugins handle background synchronization with your server effortlessly. You eliminate complex state management while maintaining consistent data across platforms.
:::note
You can find the source code for an example RxDB Flutter Application [at the github repo](https://github.com/pubkey/rxdb/tree/master/examples/flutter)
:::
---
## RxDB - The Ultimate JS Frontend Database
# RxDB JavaScript Frontend Database: Efficient Data Storage in Frontend Applications
In modern web development, managing data on the front end has become increasingly important. Storing data in the frontend offers numerous advantages, such as offline accessibility, caching, faster application startup, and improved state management. Traditional SQL databases, although widely used on the server-side, are not always the best fit for frontend applications. This is where [RxDB](https://rxdb.info/), a frontend JavaScript database, emerges as a powerful solution. In this article, we will explore why storing data in the frontend is beneficial, the limitations of SQL databases in the frontend, and how [RxDB](https://rxdb.info/) addresses these challenges to become an excellent choice for frontend data storage.
## Why you might want to store data in the frontend
### Offline accessibility
One compelling reason to store data in the frontend is to enable [offline accessibility](../offline-first.md). By leveraging a frontend database, applications can cache essential data locally, allowing users to continue using the application even when an internet connection is unavailable. This feature is particularly useful for [mobile](./mobile-database.md) applications or web apps with limited or intermittent connectivity.
### Caching
Frontend databases also serve as efficient caching mechanisms. By storing frequently accessed data locally, applications can minimize network requests and reduce latency, resulting in faster and more responsive user experiences. Caching is particularly beneficial for applications that heavily rely on remote data or perform computationally intensive operations.
### Decreased initial application start time
Storing data in the frontend decreases the initial application start time because the data is already present locally. By eliminating the need to fetch data from a server during startup, applications can quickly render the UI and provide users with an immediate interactive experience. This is especially advantageous for applications with large datasets or complex data retrieval processes.
### Password encryption for local data
Security is a crucial aspect of data storage. With a front end database, developers can [encrypt](../encryption.md) sensitive local data, such as user credentials or personal information, using encryption algorithms. This ensures that even if the device is compromised, the data remains securely stored and protected.
### Local database for state management
Frontend databases provide an alternative to traditional state management libraries like Redux or NgRx. By utilizing a local database, developers can store and manage application state directly in the frontend, eliminating the need for additional libraries. This approach simplifies the codebase, reduces complexity, and provides a more straightforward data flow within the application.
### Low-latency local queries
Frontend databases enable low-latency queries that run entirely on the client's device. Instead of relying on server round-trips for each query, the database executes queries locally, resulting in faster response times. This is particularly beneficial for applications that require real-time updates or frequent data retrieval.
### Building realtime applications with local data
Realtime applications often require immediate updates based on data changes. By storing data locally and utilizing a frontend database, developers can build [realtime applications](./realtime-database.md) more easily. The database can observe data changes and automatically update the UI, providing a seamless and responsive user experience.
### Easier integration with JavaScript frameworks
Frontend databases, including RxDB, are designed to integrate seamlessly with popular JavaScript frameworks such as [Angular](./angular-database.md), [React.js](./react-database.md), [Vue.js](./vue-database.md), and Svelte. These databases offer well-defined APIs and support that align with the specific requirements of these frameworks, enabling developers to leverage the full potential of the frontend database within their preferred development environment.
### Simplified replication of database state
Replicating database state between the frontend and backend can be challenging, especially when dealing with complex REST routes. Frontend databases, however, provide simple mechanisms for replicating database state. They offer intuitive replication algorithms that facilitate data synchronization between the frontend and backend, reducing the complexity and potential pitfalls associated with complex REST-based replication.
### Improved scalability
Frontend databases offer improved scalability compared to traditional SQL databases. By leveraging the computational capabilities of client devices, the burden on server resources is reduced. Queries and operations are performed locally, minimizing the need for server round-trips and enabling applications to scale more efficiently.
## Why SQL databases are not a good fit for the front end of an application
While SQL databases excel in server-side scenarios, they pose limitations when used on the frontend. Here are some reasons why SQL databases are not well-suited for frontend applications:
### Push/Pull based vs. reactive
SQL databases typically rely on a push/pull model, where the server pushes data to the client upon request. This approach is not inherently reactive, as it requires explicit requests for data updates. In contrast, frontend applications often require reactive data flows, where changes in data trigger automatic updates in the UI. Frontend databases, like [RxDB](https://rxdb.info/), provide reactive capabilities that seamlessly integrate with the dynamic nature of frontend development.
### Initialization time and performance
SQL databases designed for server-side usage tend to have larger build sizes and initialization times, making them less efficient for [browser-based](./browser-database.md) applications. Frontend databases, on the other hand, directly leverage browser APIs like [IndexedDB](../rx-storage-indexeddb.md), [OPFS](../rx-storage-opfs.md), and [WebWorker](../rx-storage-worker.md), resulting in leaner builds and faster initialization times. Often the queries are such fast, that it is not even necessary to implement a loading spinner.
### Build size considerations
Server-side SQL databases typically come with a significant build size, which can be impractical for browser applications where code size optimization is crucial. Frontend databases, on the other hand, are specifically designed to operate within the constraints of browser environments, ensuring efficient resource utilization and smaller build sizes.
For example the [SQLite](../rx-storage-sqlite.md) Webassembly file alone has a size of over 0.8 Megabyte with an additional 0.2 Megabyte in JavaScript code for connection.
## Why RxDB is a good fit for the frontend
RxDB is a powerful frontend JavaScript database that addresses the limitations of SQL databases and provides an optimal solution for frontend [data storage](./browser-storage.md). Let's explore why RxDB is an excellent fit for frontend applications:
### Made in JavaScript, optimized for JavaScript applications
RxDB is designed and optimized for JavaScript applications. Built using JavaScript itself, RxDB offers seamless integration with JavaScript frameworks and libraries, allowing developers to leverage their existing JavaScript knowledge and skills.
### NoSQL (JSON) documents for UIs
RxDB adopts a [NoSQL approach](./in-memory-nosql-database.md), using [JSON documents as its primary data structure](./json-database.md). This aligns well with the JavaScript ecosystem, as JavaScript natively works with JSON objects. By using NoSQL documents, RxDB provides a more natural and intuitive data model for UI-centric applications.
### Better TypeScript support compared to SQL
TypeScript has become increasingly popular for building frontend applications. RxDB provides excellent [TypeScript support](../tutorials/typescript.md), allowing developers to leverage static typing and benefit from enhanced code quality and tooling. This is particularly advantageous when compared to SQL databases, which often have limited TypeScript support.
### [Observable Queries](../rx-query.md) for automatic UI updates
RxDB introduces the concept of observable queries, powered by RxJS. Observable queries automatically update the UI whenever there are changes in the underlying data. This reactive approach eliminates the need for manual UI updates and ensures that the frontend remains synchronized with the database state.
### Optimized observed queries with the EventReduce Algorithm
RxDB optimizes observed queries with its EventReduce Algorithm. This algorithm intelligently reduces redundant events and ensures that UI updates are performed efficiently. By minimizing unnecessary re-renders, RxDB significantly improves performance and responsiveness in frontend applications.
```typescript
const query = myCollection.find({
selector: {
age: {
$gt: 21
}
}
});
const querySub = query.$.subscribe(results => {
console.log('got results: ' + results.length);
});
```
### Observable document fields
RxDB supports observable document fields, enabling developers to track changes at a granular level within documents. By observing specific fields, developers can reactively update the UI when those fields change, ensuring a responsive and synchronized frontend interface.
```typescript
myDocument.firstName$.subscribe(newName => console.log('name is: ' + newName));
```
### Storing Documents Compressed
RxDB provides the option to store documents in a [compressed format](../key-compression.md), reducing storage requirements and improving overall database performance. Compressed storage offers benefits such as reduced disk space usage, faster data read/write operations, and improved network transfer speeds, making it an essential feature for efficient frontend data storage.
### Built-in Multi-tab support
RxDB offers built-in multi-tab support, allowing data synchronization and state management across multiple browser tabs. This feature ensures consistent data access and synchronization, enabling users to work seamlessly across different tabs without conflicts or data inconsistencies.
### Replication Algorithm can be made compatible with any backend
RxDB's [realtime replication algorithm](../replication.md) is designed to be flexible and compatible with various backend systems. Whether you're using your own servers, [Firebase](../replication-firestore.md), [CouchDB](../replication-couchdb.md), [NATS](../replication-nats.md), [WebSocket](../replication-websocket.md), or any other backend, RxDB can be seamlessly integrated and synchronized with the backend system of your choice.
### Flexible storage layer for code reuse
RxDB provides a [flexible storage layer](../rx-storage.md) that enables code reuse across different platforms. Whether you're building applications with [Electron.js](../electron-database.md), [React Native](../react-native-database.md), hybrid apps using [Capacitor.js](../capacitor-database.md), or traditional web browsers, RxDB allows you to reuse the same codebase and leverage the power of a frontend database across different environments.
### Handling schema changes in distributed environments
In distributed environments where data is stored on multiple client devices, handling schema changes can be challenging. RxDB tackles this challenge by providing robust mechanisms for [handling schema changes](../migration-schema.md). It ensures that schema updates propagate smoothly across devices, maintaining data integrity and enabling seamless schema evolution.
## Follow Up
To further explore RxDB and get started with using it in your frontend applications, consider the following resources:
- [RxDB Quickstart](../quickstart.md): A step-by-step guide to quickly set up RxDB in your project and start leveraging its features.
- [RxDB GitHub Repository](https://github.com/pubkey/rxdb): The official repository for RxDB, where you can find the code, examples, and community support.
By adopting [RxDB](https://rxdb.info/) as your frontend database, you can unlock the full potential of frontend data storage and empower your applications with offline accessibility, caching, improved performance, and seamless data synchronization. RxDB's JavaScript-centric approach and powerful features make it an ideal choice for frontend developers seeking efficient and scalable data storage solutions.
---
## ideas for articles
- storing and searching through 1mio emails in a browser database
- Finding the optimal way to shorten vector embeddings
- Performance and quality of vector comparison functions (euclideanDistance etc)
- performance and quality of vector indexing methods
- What is new in IndexedDB 3.0
- how progressive syncing beats client-server architecture
## Seo keywords:
X- "optimistic ui"
X- "local database" (rddt done)
X- "react-native encryption"
X- "vue database" (rddt done)
X- "jquery database"
X- "vue indexeddb"
X- "firebase realtime database alternative" (rddt done)
X- "firestore alternative" (rddt done)
X- "ionic storage" (rddt done)
X- "local database"
X- "offline database"
X- "zero local first"
X- "webrtc p2p" - 390 http://localhost:3000/replication-webrtc.html
X- "indexeddb storage limit" - 590 https://rxdb.info/articles/indexeddb-max-storage-limit.html
X- "indexeddb size limit" - 260 https://rxdb.info/articles/indexeddb-max-storage-limit.html
X- "indexeddb max size" - 590 https://rxdb.info/articles/indexeddb-max-storage-limit.html
X- "indexeddb limits" - 170 https://rxdb.info/articles/indexeddb-max-storage-limit.html
X- "json based database"
X- "json vs database"
X- "reactjs storage"
## Seo
- "supabase alternative"
- "store local storage"
- "react localstorage"
- "react-native storage"
- "supabase offline" - 260
- "store array in localstorage", "localStorage array of objects"
- "real time web apps" - 170
- "reactive database" - 210
- "electron sqlite"
- "in browser database" - 90
- "offline first app" - 260
- "react native sql" - 110
- "sqlite electron"
- "localstorage vs indexeddb"
- "react native nosql database" - 30
- "indexeddb library" - 260
- "indexeddb encryption" - 90
- "client side database" - 140
- "webtransport vs websocket"
- "local first development" - 210
- "local storage examples"
- "local vector database" - 590
- "mobile app database" - 590
- "web based database"
- "livequery" - 210
- "expo database" - 390
- "database sync" - 8100
- "p2p database" - 170
- "reactive app" - 260
- "offline web app" - 320
- "offline sync" - 320
- "react native encrypted storage" - 1000
- "firestore vs firebase" - 1300
- "ionic alternatives" - 480
- "react native backend" - 720
- "react native alternative" - 1000
- "react native sqlite" - 1900
- "flutter vs react native" - 5400
- "react native redux" - 3600
- "redux alternative" - 1300
- "Awesome local first" - 10
- "tauri database" - 170
- "capacitor embedded database"
- "Node.js embedded database"
- "sqlite javascript" - 2900
- "sqlite typescript" - 260
- "sync engine" - 390
- "indexeddb alternative" - 70
## Non Seo
- "Local-First Partial Sync with RxDB"
- "why the indexeddb API is almost perfekt"
- "how to do auth with RxDB"
- "Where to store that JWT token?"
---
## RxDB In-Memory NoSQL - Supercharge Real-Time Apps
# RxDB as In-memory NoSQL Database: Empowering Real-Time Applications
Real-time applications have become increasingly popular in today's digital landscape. From instant messaging to collaborative editing tools, the demand for responsive and interactive software is on the rise. To meet these requirements, developers need powerful and efficient database solutions that can handle large amounts of data in real-time. [RxDB](https://rxdb.info/), an javascript NoSQL database, is revolutionizing the way developers build and scale their applications by offering exceptional speed, flexibility, and scalability.
## Speed and Performance Benefits
One of the key advantages of using RxDB as an in-memory NoSQL database is its ability to leverage in-memory storage for faster database operations. By storing data directly in memory, database operations can be performed significantly faster compared to traditional disk-based databases. This is especially important for real-time applications where every millisecond counts. With RxDB, developers can achieve near-instantaneous data access and manipulation, enabling highly responsive user experiences.
Additionally, RxDB eliminates disk I/O bottlenecks that are typically associated with traditional databases. In traditional databases, disk reads and writes can become a bottleneck as the amount of data grows. In contrast, an in-memory database like RxDB keeps the entire dataset in RAM, eliminating disk access overhead. This makes it an excellent choice for applications dealing with real-time analytics, high-throughput data processing, and caching.
## Persistence Options
While RxDB offers an [in-memory](../rx-storage-memory.md) storage adapter, it also offers [persistence storages](../rx-storage.md). Adapters such as [IndexedDB](../rx-storage-indexeddb.md), [SQLite](../rx-storage-sqlite.md), and [OPFS](../rx-storage-opfs.md) enable developers to persist data locally in the browser, making applications accessible even when [offline](../offline-first.md). This hybrid approach combines the benefits of in-memory performance with data durability, providing the best of both worlds. Developers can choose the adapter that best suits their needs, balancing the speed of in-memory storage with the long-term data persistence required for certain applications.
```javascript
import {
createRxDatabase
} from 'rxdb';
import {
getRxStorageMemory
} from 'rxdb/plugins/storage-memory';
const db = await createRxDatabase({
name: 'exampledb',
storage: getRxStorageMemory()
});
```
Also the [memory mapped RxStorage](../rx-storage-memory-mapped.md) exists as a wrapper around any other RxStorage. The wrapper creates an in-memory storage that is used for query and write operations. This memory instance is replicated with the underlying storage for persistence. The main reason to use this is to improve initial page load and query/write times. This is mostly useful in browser based applications.
## Use Cases for RxDB
RxDB's capabilities make it well-suited for various real-time applications. Some notable use cases include:
- Chat Applications and Real-Time Messaging: RxDB's in-memory performance and real-time synchronization capabilities make it an excellent choice for building chat applications and real-time messaging systems. Developers can ensure that messages are delivered and synchronized across multiple clients in real-time, providing a seamless and responsive chat experience.
- Collaborative Document Editors: RxDB's ability to handle data streams and propagate changes in real-time makes it ideal for collaborative document editing. Multiple users can simultaneously edit a document, and their changes are instantly synchronized, allowing for real-time collaboration and ensuring that everyone has the most up-to-date version of the document.
- Real-Time Analytics Dashboards: RxDB's speed and scalability make it a valuable tool for real-time analytics dashboards. It can handle high volumes of data and perform complex analytics operations in real-time, providing instant insights and visualizations to users.
In conclusion, RxDB serves as a powerful in-memory NoSQL database that empowers developers to build real-time applications with exceptional speed, flexibility, and scalability. Its ability to leverage in-memory storage, eliminate disk I/O bottlenecks, and provide persistence options make it an attractive choice for a wide range of real-time use cases. Whether it's chat applications, collaborative document editors, or real-time analytics dashboards, RxDB provides the foundation for building responsive and interactive software that meets the demands of today's users.
---
## IndexedDB Alternative - Why RxDB is the Better Choice
# IndexedDB Alternatives
IndexedDB is the standard [browser storage](../articles/browser-storage.md) API for storing significant amounts of structured data, including files/blobs. It is available in every modern browser.
However, using the native IndexedDB API is **verbose**, **low-level**, and lacks many features modern applications need.
If you are looking for an **IndexedDB alternative**, you likely want a library that abstracts the complexity away and provides features like **Reactivity**, **Schema Validation**, and **Sync**.
RxDB is the ultimate alternative because it gives you the speed of a [local database](../articles/local-database.md) with the ease of use of a modern [JSON-document store](../rx-collection.md).
## The Problem with Raw IndexedDB
IndexedDB was designed as a low-level building block, not a developer-facing database engine.
Because of that, relying on raw IndexedDB (or thin wrappers) often leads to significant friction:
1. **Callback Hell**: The API heavily relies on event handlers (`onsuccess`, `onerror`), making control flow difficult to read and maintain.
2. **Missing Observability**: Standard IndexedDB provides no way to listen to data changes. You have to build your own event bus to update the UI when data changes.
3. **Complex Transaction Management**: You must explicitly create transactions for every read or write, which is repetitive and error-prone.
4. **No Schema Enforcement**: IndexedDB is schema-less. You can store anything, which sounds good until your app crashes because of inconsistent data structures.
5. **Limited Querying**: You can only query by simple key ranges. Complex queries (like "find users older than 18 and sort by name") require manually iterating over cursors, which is slow and code-heavy. See [Slow IndexedDB](../slow-indexeddb.md).
RxDB solves all of these problems while maintaining the benefits of a local database.
## Why RxDB is the Best Alternative
RxDB is a NoSQL database for JavaScript applications. It uses [IndexedDB](../rx-storage-indexeddb.md) (or faster alternatives) under the hood but provides a rich, feature-complete API on top.
### 1. Developer Experience
RxDB offers a promise-based API that feels intuitive for JavaScript developers. It uses [JSON Schema](../rx-schema.md) to define your data structure, ensuring you never store invalid data.
**Raw IndexedDB:**
```js
const request = indexedDB.open('myDatabase', 1);
request.onupgradeneeded = (event) => { /* Handle versions */ };
request.onsuccess = (event) => {
const db = event.target.result;
const transaction = db.transaction(['users'], 'readonly');
const store = transaction.objectStore('users');
const getRequest = store.get('user1');
getRequest.onsuccess = () => {
console.log(getRequest.result); // Finally got the data
};
};
```
**RxDB:**
```js
const db = await createRxDatabase({
name: 'myDatabase',
storage: getRxStorageDexie() // Uses IndexedDB under the hood
});
// Define collection once
await db.addCollections({
users: { schema: myJsonSchema }
});
// Query data
const user = await db.users.findOne('user1').exec();
```
### 2. Reactivity (The "Rx" in RxDB)
Modern UIs ([React](../articles/react-database.md), [Vue](../articles/vue-database.md), [Angular](../articles/angular-database.md), Svelte) need to be reactive. When data changes, the view should update.
RxDB is built on **[RxJS](https://rxjs.dev/)**. Every query, document, or field can be observed.
```js
// Subscribe to a query -> UI updates automatically on change
db.users.find({
selector: { age: { $gt: 18 } }
}).$.subscribe(users => {
updateUI(users);
});
```
This works even across multiple browser tabs. If a user changes data in Tab A, Tab B updates instantly. Implementing this with raw IndexedDB and `BroadcastChannel` manually is a massive undertaking. See [Reactivity](../reactivity.md).
### 3. Advanced Query Engine
Searching for data in raw IndexedDB requires opening cursors and iterating over records manually, which is slow and complex.
RxDB includes [Mango Query](../rx-query.md) syntax (like MongoDB). You can filter, sort, and limit data with a simple JSON object.
```js
const results = await db.users.find({
selector: {
age: { $gt: 18 },
role: { $in: ['admin', 'moderator'] }
},
sort: [{ name: 'asc' }],
limit: 10
}).exec();
```
### 4. Synchronization
Raw IndexedDB is purely local. Usage in real-world apps usually requires syncing data with a backend.
Building a robust sync protocol (handling [offline](../offline-first.md) changes, conflict resolution, delta updates) is one of the hardest problems in software engineering.
RxDB solves this out of the box. It has a robust [replication protocol](../replication.md) that supports:
- **[Real-time sync](../articles/realtime-database.md)**: Changes are pushed/pulled immediately.
- **Conflict Resolution**: Strategies to handle concurrent edits.
- **Multi-Backend**: Plugins for [CouchDB](../replication-couchdb.md), [GraphQL](../replication-graphql.md), [Firestore](../replication-firestore.md), [Supabase](../replication-supabase.md), and generic [HTTP](../replication-http.md).
### 5. Performance & Storage Engines
While IndexedDB is fast enough for simple tasks, it can be the bottleneck for high-performance apps due to serialization overhead and browser implementation details. See [RxStorage Performance](../rx-storage-performance.md).
RxDB abstracts the storage layer. You can start with IndexedDB and switch to unparalleled performance engines later without changing your application code:
- **[Dexie.js RxStorage](../rx-storage-dexie.md)**: A optimized wrapper around IndexedDB.
- **[OPFS RxStorage](../rx-storage-opfs.md)**: Uses the Origin Private File System, which is significantly faster than IDB for many operations.
- **[SQLite RxStorage](../rx-storage-sqlite.md)**: Uses SQLite via WebAssembly (or Native in [React Native](../react-native-database.md)/[Electron](../electron-database.md)) for robust SQL-based storage.
- **[Memory RxStorage](../rx-storage-memory.md)**: For ephemeral data or testing.
### 6. TypeScript Support
IndexedDB API is loosely typed. You often cast `any` or struggle with correct event types.
RxDB is written in TypeScript and provides first-class type safety. Your database schema generates TypeScript types, so you get autocomplete for every field in your documents and queries.
```ts
// TypeScript knows that 'age' is a number
const user = await db.users.findOne().exec();
console.log(user.age.toFixed(2));
```
### 7. Encryption & Compression
Storing sensitive data? RxDB has [Encryption](../encryption.md) built-in. You provide a password, and the data is stored encrypted at rest.
Storing lots of data? The [Key-Compression](../key-compression.md) plugin shrinks your JSON keys to minimize storage usage, often reducing database size by 40%+.
## Other Alternatives
There are other ways to store data in the browser, but they all have significant limitations compared to IndexedDB (and RxDB).
### LocalStorage
`localStorage` is a synchronous key-value store. See [Using localStorage](../articles/localstorage.md).
- **Why it fails**: It blocks the main thread (UI freezes on large reads/writes). It is capped at ~5MB. It only supports strings, so you must constantly `JSON.parse` and `JSON.stringify`.
- **Use case**: Simple settings like "dark mode: on".
### Cookies
Cookies are small pieces of data sent with every HTTP request.
- **Why it fails**: Extremely limited size (4KB). Wastes bandwidth by sending data to the server on every request.
- **Use case**: Session tokens, authentication.
### WebSQL
WebSQL was a wrapper around SQLite but is **deprecated** and removed from non-Google browsers.
- **Why it fails**: It is a dead standard. Do not use it.
- **Use case**: Legacy apps only.
### OPFS (Origin Private File System)
OPFS is a new high-performance file system API for the web.
- **Why it fails**: It is a file system, not a database. It has no indexing, no querying, and no document structure. It is extremely low-level.
- **Note**: RxDB *uses* OPFS in its [OPFS RxStorage](../rx-storage-opfs.md) to give you the performance of OPFS with the features of a real database.
## Comparison
| Feature | Raw IndexedDB | **RxDB** |
| :--- | :--- | :--- |
| **Api Style** | Event-based / Callback | Promise / Observable |
| **Reactivity** | β None | β [Observables](../rx-query.md) / [Signals](../reactivity.md) |
| **Sync** | β Manual Implementation | β Built-in & Backend Agnostic |
| **Query Engine** | β Basic Key-Range | β [MongoDB-style (Mango)](../rx-query.md) |
| **Transactions** | β Manual | β Automatic |
| **Schema** | β None | β [JSON Schema](../rx-schema.md) |
| **Migrations** | β οΈ Manual | β [Declarative](../migration-schema.md) |
| **Multi-Tab Sync**| β Manual | β Automatic |
| **Encryption** | β None | β [Built-in](../encryption.md) |
| **TypeScript** | β οΈ Partial | β Full Support |
## Conclusion
If you are building a toy project, `localStorage` or a simple wrappers like `idb-keyval` might suffice.
But if you are building a **production application** that needs to be fast, reliable, and maintainable, relying on raw IndexedDB is a premature optimization that costs you development time.
**RxDB** is the "Battery Included" alternative that handles the hard parts of local data (sync, reactivity, queries) so you can focus on building your product. For further reading, check out [Why Local-First Software Is the Future](../articles/local-first-future.md) or [RxDB as a Database for Browsers](../articles/browser-database.md).
---
## IndexedDB Max Storage Size Limit - Detailed Best Practices
import {VideoBox} from '@site/src/components/video-box';
# IndexedDB Max Storage Size Limit
IndexedDB is widely known as the primary browser-based storage API for large client-side data, particularly valuable for modern [offline-first](../offline-first.md) applications. These apps aim to keep everything functional and interactive even without an internet connection, which naturally demands substantial local storage. However, IndexedDB has various size limits depending on the browser, disk space, and user settings. Being aware of these constraints is crucial so you can avoid quota errors and deliver a seamless user experience without unexpected data loss.
Offline-first apps have grown in popularity because they provide immediate feedback, zero-latency interactions, and resilience in poor network conditions. Storing big data sets, or even entire data models, in IndexedDB has become far more common than in the era of small localStorage or cookie usage. But all this local data is subject to quotas, and thatβs exactly what this guide will help you understand and manage.
## Why IndexedDB Has a Storage Limit
Browsers need a way to curb runaway disk usage and safeguard user resources. This is accomplished through **quota management** policies, which can vary among Chrome, Firefox, Safari, Edge, and others. Some browsers use a percentage of your total disk space, while others rely on a fixed maximum or dynamic approach per origin. These policies are designed to prevent malicious or poorly optimized web pages from consuming an unreasonable amount of user storage.
Chrome (and Chromium-based browsers) typically allow you to use a percentage of the userβs free disk space, whereas Firefox historically prompts users to allow more than 5 MB in mobile or 50 MB in desktop. Safari often sets tighter maximum caps, especially on iOS devices. Edge aligns closely with Chromeβs rules but can also include enterprise or corporate policy overrides. Understanding these default or dynamic limits prepares you to plan your appβs storage needs appropriately.
## Browser-Specific IndexedDB Limits
IndexedDB size quotas differ significantly across browsers and platforms. While there isnβt a universal rule, the following table summarizes approximate limits and any notes or caveats you should be aware of:
| Browser | Approx. Limit | Notes |
|----------------|---------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------|
| Chrome/Chromium | Up to ~80% of free disk, per origin cap | Often cited as 60 GB on a 100 GB drive. Shared pool approach. Quota usage can prompt partial or extended user approvals. |
| Firefox | ~2 GB (desktop) or ~5 MB initial for mobile | Older versions asked permission at 50 MB for desktop. Ephemeral/incognito sessions may require repeated user prompts. |
| Safari (iOS) | ~1 GB per origin (variable) | Historically stricter. iOS devices limit quotas further. Behavior can differ between iOS Safari versions or iPadOS. |
| Edge | Similar to Chromeβs 80% of free space | Can be influenced by Windows enterprise policies. Generally aligned with Chromium approach. |
| iOS Safari | Typically 1 GB, can be less on older iOS | Early iOS versions were known for more aggressive quotas and data eviction on low space. |
| Android Chrome | Similar to desktop Chrome | May exhibit warnings in especially low-storage devices. The same 80% free space logic generally applies. |
Historically, these limits have evolved. For instance, older Firefox versions included `dom.indexedDB.warningQuota`, showing a 50 MB prompt on desktop or a 5 MB prompt on mobileβmany developers wrote about these notifications on Stack Overflow. Since around 2015, Firefox has changed its quota approach significantly. Likewise, Safari used to limit data more aggressively on older iOS versions. Some older tutorials suggest comparing IndexedDB to localStorage, but modern browsers allow far larger and more flexible storage with IndexedDB than the old localStorage or cookie-based setups.
---
## Checking Your Current IndexedDB Usage
To assess where your app stands relative to these storage limits, you can use the **Storage Estimation API**. The snippet below shows how to estimate both your used storage and the total space allocated to your origin:
```js
const quota = await navigator.storage.estimate();
const totalSpace = quota.quota;
const usedSpace = quota.usage;
console.log('Approx total allocated space:', totalSpace);
console.log('Approx used space:', usedSpace);
```
[Some browsers (all modern ones)](https://developer.mozilla.org/en-US/docs/Web/API/StorageManager/persist#browser_compatibility) also provide a `navigator.storage.persist()` method to request persistent storage, preventing the browser from automatically clearing your data if the userβs device runs low on space. Note that users might deny such requests, or the request might fail silently on stricter environments. Always handle these outcomes gracefully and design your app to degrade if persistent storage is unavailable.
## Testing Your Appβs IndexedDB Quotas
The best way to handle real-world usage is to test for low storage conditions and large data sets in different environments. You can fill up the space manually by writing repetitive test data or running scripts that bulk-insert documents until an error occurs.
Real-time usage monitors or dashboards can keep track of your `navigator.storage.estimate()` results, letting you see how close you are to the max limit in production. Developer tools in Chrome or Firefox can simulate limited storage situations, which is crucial for QA:
This short tutorial shows how you can artificially reduce available storage in Google Chromeβs dev tools to see how your app behaves when nearing or exceeding the quota.
## Handling Errors When Limits Are Reached
When the userβs device is too full or your app exceeds the allotted quota, most browsers will throw a **QuotaExceededError** (or similarly named exception) when trying to store additional data. Often, the request to IndexedDB simply fails with an error event. Handling this gracefully is essential to avoid crashes or data corruption.
A typical approach is to wrap your write operations in try/catch blocks or in `onsuccess` / `onerror` event callbacks. If you detect a quota error, you can prompt the user to clear out old items or reduce the scope of offline data. Some apps implement a fallback system that removes less critical documents to free space and then retries the write.
```js
try {
const tx = db.transaction('largeStore', 'readwrite');
const store = tx.objectStore('largeStore');
await store.add(hugeData, someKey);
await tx.done;
} catch (error) {
if (error.name === 'QuotaExceededError') {
console.warn('IndexedDB quota exceeded. Cleanup or prompt user to free space.');
// Optionally remove older data or show a UI hint:
// removeOldDocuments();
// displayStorageFullDialog();
} else {
// handle other errors
console.error('IndexedDB write error:', error);
}
}
```
## Tricks to Exceed the Storage Size Limitation
Even if you plan well, your app might need more storage than a single origin typically allows. There are a few advanced tactics you can use:
If you store binary data such as images or videos, consider compressing them via the Compression Streams API. For textual or [JSON data](./json-based-database.md), a library like [RxDB](/) supports built-in [key-compression](../key-compression.md) to shorten field names or entire documents. This can be extremely helpful when storing large sets of objects:
```ts
// Example: How key-compression can transform your documents internally
const uncompressed = {
"firstName": "Corrine",
"lastName": "Ziemann",
"shoppingCartItems": [
{
"productNumber": 29857,
"amount": 1
},
{
"productNumber": 53409,
"amount": 6
}
]
};
const compressed = {
"|e": "Corrine",
"|g": "Ziemann",
"|i": [
{
"|h": 29857,
"|b": 1
},
{
"|h": 53409,
"|b": 6
}
]
};
```
Sharding data across multiple subdomains or iframes is another trick, though it complicates communication. When you need truly massive offline data, you might store part of the data under `sub1.yoursite.com` and another chunk under `sub2.yoursite.com`, using `postMessage()` to coordinate. This can circumvent single-origin limitations, but it introduces extra complexity. Another effective method is to let data expire automaticallyβperhaps older records are removed if they havenβt been accessed for a certain period.
## IndexedDB Max Size of a Single Object
There is no explicit cap on how large an individual object or record in IndexedDB can be, other than the overall disk quota. If you attempt to store one extremely large object, you will eventually hit browser memory constraints or the global storage quota. In practice, youβll encounter out-of-memory issues in JavaScript before IndexedDB itself refuses a single large write. A helpful test can be seen in [this JSFiddle experiment](https://jsfiddle.net/sdrqf8om/2/) where you see browsers can crash when creating massive in-memory objects.
## Is There a Time Limit for Data Stored in IndexedDB?
IndexedDB data can remain indefinitely as long as the user does not clear the browserβs data or the origin does not run afoul of automated eviction policies (e.g., Safari or Android might remove large caches for sites unused over a long period when space is needed). Typically, there is no βtime limit,β but ephemeral modes or incognito sessions have their own rules. If you rely on permanent offline data, request persistent storage and handle the possibility that the user or the OS could still remove your data under extreme conditions. Especially Safari is known to be very fast in deleting local data.
## Follow Up
Learn more by checking the [IndexedDB official docs](https://developer.mozilla.org/en-US/docs/Web/API/IndexedDB_API), which detail store design, error handling, and quota usage. If you need a straightforward way to manage large offline data with compression and conflict resolution, explore the [RxDB Quickstart](../quickstart.md). You can also join the community on [GitHub](/code/) to share tips on overcoming the **IndexedDB max storage size limit** in production environments.
---
## RxDB - The Perfect Ionic Database
# Ionic Storage - RxDB as database for hybrid apps
In the fast-paced world of mobile app development, **hybrid applications** have emerged as a versatile solution, offering the best of both worlds - the web and native app experiences. One key challenge these apps face is efficiently storing and querying data on the **client's device**. Enter [RxDB](https://rxdb.info/), a powerful client-side database tailored for ionic hybrid applications. In this article, we'll explore how RxDB addresses the requirements of storing and querying data in ionic apps, and why it stands out as a preferred choice.
## What are Ionic Hybrid Apps?
Ionic (aka Ionic 2 ) hybrid apps combine the strengths of web technologies (HTML, CSS, JavaScript) with native app development to deliver cross-platform applications. They are built using web technologies and then wrapped in a native container to be deployed on various platforms like iOS, Android, and the web. These apps provide a consistent user experience across devices while benefiting from the efficiency and familiarity of web development.
## Storing and Querying Data in an Ionic App
Storing and querying data is a fundamental aspect of any application, including hybrid apps. These apps often need to operate offline, store user-generated content, and provide responsive user interfaces. Therefore, having a reliable and efficient way to manage data on the client's device is crucial.
## Introducing RxDB as a Client-Side Database for Ionic Apps
RxDB steps in as a powerful solution to address the data management needs of ionic hybrid apps. It's a NoSQL client-side database that offers exceptional performance and features tailored to the unique requirements of client-side applications. Let's delve into the key features of RxDB that make it a great fit for these apps.
### Getting Started with RxDB
### What is RxDB?
At its core, [RxDB](https://rxdb.info/) is a **NoSQL** database that operates with a [local-first](../offline-first.md) approach. This means that your app's data is stored and processed primarily on the client's device, reducing the dependency on constant network connectivity. By doing so, RxDB ensures your app remains responsive and functional, even when offline.
### Local-First Approach
The [local-first](../offline-first.md) approach adopted by RxDB is a game-changer for hybrid applications. Storing data locally allows your app to function seamlessly without an internet connection, providing users with uninterrupted access to their data. When connectivity is restored, RxDB handles the synchronization of data, ensuring that any changes made offline are appropriately propagated.
### Observable Queries
One of RxDB's standout features is its implementation of [observable queries](../rx-query.md). This concept allows your app's user interface to be dynamically updated in real time as data changes within the database. RxDB's observables create a bridge between your database and user interface, keeping them in sync effortlessly.
### NoSQL Query Engine
RxDB's NoSQL query engine empowers you to perform powerful queries on your app's data, without the constraints imposed by traditional relational databases. This flexibility is particularly valuable when dealing with unstructured or semi-structured data. With the NoSQL query engine, you can retrieve, filter, and manipulate data according to your app's unique requirements.
```ts
const foundDocuments = await myDatabase.todos.find({
selector: {
done: {
$eq: false
}
}
}).exec();
```
### Great Observe Performance with EventReduce
RxDB introduces a concept called [EventReduce](https://github.com/pubkey/event-reduce), which optimizes the observation process. Instead of overwhelming your app's UI with every data change, EventReduce filters and batches these changes to provide a smooth and efficient experience. This leads to enhanced app performance, lower resource usage, and ultimately, happier users.
## Why NoSQL is a Better Fit for Client-Side Applications Compared to relational databases like SQLite
When it comes to choosing the right database solution for your client-side applications, NoSQL RxDB presents compelling advantages over traditional options like [SQLite](../rx-storage-sqlite.md). Let's delve into the key reasons why NoSQL RxDB is a superior fit for your ionic hybrid app development.
### Easier Document-Based Replication
NoSQL databases, like RxDB, inherently embrace a document-based approach to [data storage](./ionic-storage.md). This design choice simplifies data [replication](../replication.md) between clients and servers. With documents representing discrete units of data, you can easily synchronize individual pieces of information without the complexity that can arise when dealing with rows and tables in a relational database like SQLite. This document-centric replication model streamlines the synchronization process and ensures that your app's data remains consistent across devices.
### Offline Capable
One of the defining features of client-side applications is the ability to function even when offline. NoSQL RxDB excels in this area by supporting a local-first approach. Data is cached on the client's device, enabling the app to remain fully functional even without an internet connection. As connectivity is restored, RxDB handles data synchronization with the server seamlessly. This offline capability ensures a smooth user experience, critical for ionic hybrid apps catering to users in various network conditions.
### NoSQL Has Better TypeScript Support
TypeScript, a popular superset of JavaScript, is renowned for its static typing and enhanced developer experience. NoSQL databases like RxDB are inherently flexible, making them well-suited for TypeScript integration. With well-defined data structures and clear typings, NoSQL RxDB offers [improved type safety](../tutorials/typescript.md) and easier development when compared to traditional SQL databases like SQLite. This results in reduced debugging time and increased code reliability.
### Easier [Schema Migration](../migration-schema.md) with NoSQL Documents
Schema changes are a common occurrence in application development, and dealing with them can be challenging. NoSQL databases, including RxDB, are more forgiving in this aspect. Since documents in NoSQL databases don't enforce a rigid structure like tables in relational databases, schema changes are often simpler to manage. This flexibility makes it easier to evolve your app's data structure over time without the need for complex migration scripts, a notable advantage when compared to SQLite.
## Great Performance
RxDB's [excellent performance](../rx-storage-performance.md) stems from its advanced indexing capabilities, which streamline data retrieval and ensure swift query execution. Additionally, the [JSON key compression](../key-compression.md) employed by RxDB minimizes storage overhead, enabling efficient data transfer and quicker loading times. The incorporation of real-time updates through change streams and the **EventReduce mechanism** further enhances RxDB's performance, delivering a responsive user experience even as data changes are propagated seamlessly.
## Using RxDB in an Ionic Hybrid App
RxDB's integration into your ionic hybrid app opens up a world of possibilities for efficient data management. Let's explore how to set up RxDB, use it with popular JavaScript frameworks, and take advantage of its diverse storage options.
### Setup RxDB
Getting started with RxDB is a straightforward process. By including the RxDB library in your project, you can quickly start harnessing its capabilities. Begin by installing the [RxDB package](https://www.npmjs.com/package/rxdb) from the npm registry. Then, configure your database instance to suit your app's needs. This setup process paves the way for seamless data management in your ionic hybrid app.
For a full instruction, follow the [RxDB Quickstart](https://rxdb.info/quickstart.html).
### Using RxDB in Frameworks (React, Angular, Vue.js)
RxDB seamlessly integrates with various JavaScript frameworks, ensuring compatibility with your preferred development environment. Whether you're building your ionic hybrid app with [React](./react-database.md), [Angular](./angular-database.md), or [Vue.js](./vue-database.md), RxDB offers bindings and tools that enable you to leverage its features effortlessly. This compatibility allows you to stay within the comfort zone of your chosen framework while benefiting from RxDB's powerful data management capabilities.
### Different RxStorage Layers for RxDB
RxDB doesn't limit you to a single storage solution. Instead, it provides a range of RxStorage layers to accommodate diverse use cases. These storage layers offer flexibility and customization, enabling you to tailor your data management strategy to match your app's requirements. Let's explore some of the available RxStorage options:
- [LocalStorage RxStorage](../rx-storage-localstorage.md): Based on the browsers [localStorage](./localstorage.md). Easy to set up and fast for small datasets.
- [IndexedDB RxStorage](../rx-storage-indexeddb.md): Leveraging the native browser storage, IndexedDB RxStorage offers reliable data persistence. This storage option is suitable for a wide range of scenarios and is supported by most modern browsers.
- [OPFS RxStorage](../rx-storage-opfs.md): Operating within the browser's file system, OPFS RxStorage is a unique choice that can handle larger data volumes efficiently. It's particularly useful for applications that require substantial data storage.
- [Memory RxStorage](../rx-storage-memory.md): Memory RxStorage is perfect for temporary or cache-like data storage. It keeps data in memory, which can result in rapid data access but doesn't provide long-term persistence.
- [SQLite RxStorage](../rx-storage-sqlite.md): SQLite is the goto database for mobile applications. It is build in on android and iOS devices. The SQLite RxDB storage layer is build upon SQLite and offers the best performance on hybrid apps, like ionic.
## Replication of Data with RxDB between Clients and Servers
Efficient data replication between clients and servers is the backbone of modern application development, ensuring that data remains consistent and up-to-date across various devices and platforms. RxDB provides a suite of replication methods that facilitate seamless communication between clients and servers, ensuring that your data is always in sync.
### RxDB Replication Algorithm
At the heart of RxDB's replication capabilities lies a sophisticated [algorithm](../replication.md) designed to manage data synchronization between clients and servers. This algorithm intelligently handles data changes, conflict resolution, and network connectivity fluctuations, resulting in reliable and efficient data replication. With the RxDB replication algorithm, your application can maintain data consistency across devices without unnecessary complexities.
- [CouchDB Replication](../replication-couchdb.md):
RxDB's integration with CouchDB replication presents a powerful way to synchronize data between clients and servers. CouchDB, a well-established NoSQL database, excels at distributed and decentralized data scenarios. By utilizing RxDB's CouchDB replication, you can establish bidirectional synchronization between your RxDB-powered client and a CouchDB server. This synchronization ensures that data updates made on either end are seamlessly propagated to the other, facilitating collaboration and data sharing.
- [Firestore Replication](../replication-firestore.md):
Firestore, Google's cloud-hosted NoSQL database, offers another avenue for data replication in RxDB. With Firestore replication, you can establish a connection between your RxDB-powered app and Firestore's cloud infrastructure. This integration provides real-time updates to data across multiple instances of your application, ensuring that users always have access to the latest information. RxDB's support for Firestore replication empowers you to build dynamic and responsive applications that thrive in today's fast-paced digital landscape.
- [WebRTC Replication](../replication-webrtc.md):
Peer-to-peer (P2P) replication via WebRTC introduces a cutting-edge approach to data synchronization in RxDB. P2P replication allows devices to communicate directly with each other, bypassing the need for a central server. This method proves invaluable in scenarios where network connectivity is limited or unreliable. With WebRTC replication, devices can exchange data directly, enabling collaboration and information sharing even in challenging network conditions.
## RxDB as an Alternative for Ionic Secure Storage
When it comes to securing sensitive data in your Ionic applications, RxDB emerges as a powerful alternative to traditional secure storage solutions. Let's delve into why RxDB is an exceptional choice for safeguarding your data while providing additional benefits.
### RxDB On-Device Encryption Plugin
RxDB offers an [on-device encryption plugin](https://rxdb.info/encryption.html), adding an extra layer of security to your app's data. This means that data stored within the RxDB database can be encrypted, ensuring that even if the device falls into the wrong hands, the sensitive information remains inaccessible without the proper decryption key. This level of data protection is crucial for applications that deal with personal or confidential information. Encryption runs either with `AES` on `crypto-js` or with the [Web Crypto API](https://developer.mozilla.org/en-US/docs/Web/API/Web_Crypto_API) which is faster and more secure.
### Works Offline
Security should never compromise functionality. RxDB excels in this area by allowing your application to operate seamlessly even when offline. The locally stored encrypted data remains accessible and functional, enabling users to interact with the app's features even without an active internet connection. This offline capability ensures that user data is secure, while the app continues to deliver a responsive and uninterrupted experience.
### Easy-to-Setup Replication with Your Backend
Ensuring data consistency between your client-side application and backend is a key concern for developers. RxDB simplifies this process with its straightforward replication setup. You can effortlessly configure data synchronization between your local RxDB instance and your backend server. This replication capability ensures that encrypted data remains up-to-date and aligned with the central database, enhancing data integrity and security.
### Compression of Client-Side Stored Data
In addition to security and offline capabilities, RxDB also offers [data compression](https://rxdb.info/key-compression.html). This means that the data stored on the client's device is efficiently compressed, reducing storage requirements and improving overall app performance. This compression ensures that your app remains responsive and efficient, even as data volumes grow.
### Cost-Effective Solution
In addition to its security features, RxDB offers cost-effective benefits. RxDB is [priced more affordably](/premium/) compared to some other secure storage solutions, making it an attractive option for developers seeking robust security without breaking the bank. For many users, the free version of RxDB provides ample features to meet their application's security and data management needs.
## FAQ
What are the best peer-to-peer syncing databases for mobile applications?
RxDB excels as a peer-to-peer syncing database for mobile applications. You build mobile applications using local storage on the user device. RxDB synchronizes changes directly between multiple clients utilizing WebRTC data channels. You connect devices locally without depending on a central server. This approach minimizes latency and ensures continuous data sharing even across isolated network environments.
## Follow Up
- Try out the [RxDB ionic example project](https://github.com/pubkey/rxdb/tree/master/examples/ionic2)
- Try out the [RxDB Quickstart](https://rxdb.info/quickstart.html)
- Join the [RxDB Chat](https://rxdb.info/chat/)
---
## RxDB - Local Ionic Storage with Encryption, Compression & Sync
When building **Ionic** apps, developers face the challenge of choosing a robust **Ionic storage** mechanism that supports:
- **Offline-First** usage
- **Data Encryption** to protect sensitive content
- **Compression** to reduce storage usage and improve performance
- **Seamless Sync** with any backend for real-time updates
[RxDB](https://rxdb.info/) (Reactive Database) offers all these features in a single, [local-first](./local-first-future.md) database solution tailored to **Ionic** and other hybrid frameworks. Keep reading to learn how RxDB solves the most common storage pitfalls in hybrid app development while providing unmatched flexibility.
## Why RxDB for Ionic Storage?
### 1. Offline-Ready NoSQL Storage
[Offline functionality](../offline-first.md) is crucial for modern mobile applications, particularly when devices encounter unreliable or slow networks. RxDB stores all data **locally** so your Ionic app can run seamlessly without needing a continuous internet connection. When a network is available again, RxDB automatically synchronizes changes with your backend - no extra code required.
### 2. Powerful Encryption
Securing on-device data is paramount when handling sensitive information. RxDB includes [encryption plugins](../encryption.html) that let you:
- **Encrypt** data fields at rest with AES
- Invalidate data access by simply withholding the password
- Keep your users' data confidential, even if the device is stolen
This built-in encryption sets RxDB apart from many other Ionic storage options that lack integrated security.
### 3. Built-In Data Compression
Large or repetitive data can significantly slow down devices with minimal memory. RxDB's [key-compression](../key-compression.md) feature decreases document size stored on the device, improving overall performance by:
- Reducing disk usage
- Accelerating queries
- Minimizing network overhead when syncing
### 4. Real-Time Sync & Conflict Handling
In addition to functioning fully offline, RxDB supports advanced [replication](../replication.md) options. Your Ionic app can instantly sync updates with any backend ([CouchDB](../replication-couchdb.md), [Firestore](../replication-firestore.md), [GraphQL](../replication-graphql.md), or [custom REST](../replication-http.md)), maintaining a [real-time](./realtime-database.md) user experience. Plus, RxDB handles [conflicts](../transactions-conflicts-revisions.md) gracefully - meaning less worry about clashing user edits.
### 5. Easy to Adopt and Extend
RxDB runs with a **NoSQL** approach and integrates seamlessly into [Ionic Angular](https://ionicframework.com/docs/angular/overview) or other frameworks you might use with Ionic. You can extend or replace storage backends, add encryption, or build advanced offline-first features with minimal overhead.
## Quick Start: Implementing RxDB with LocalSTorage Storage
For a simple proof-of-concept or testing environment in [Ionic](./ionic-database.md), you can use [localstorage](../rx-storage-localstorage.md) as your underlying storage. Later, if you need better native performance, you can **switch to the SQLite storage** offered by the [RxDB Premium plugins](https://rxdb.info/premium/).
1. **Install RxDB**
```bash
npm install rxdb rxjs
```
2. **Initialize the Database**
```ts
import { createRxDatabase } from 'rxdb/plugins/core';
import { getRxStorageLocalstorage } from 'rxdb/plugins/storage-localstorage';
async function initDB() {
const db = await createRxDatabase({
name: 'myionicdb',
storage: getRxStorageLocalstorage(),
multiInstance: false // or true if you plan multi-tab usage
// Note: If you need encryption, set `password` here
});
await db.addCollections({
notes: {
schema: {
title: 'notes schema',
version: 0,
type: 'object',
primaryKey: 'id',
properties: {
id: { type: 'string' },
content: { type: 'string' },
timestamp: { type: 'number' }
},
required: ['id']
}
}
});
return db;
}
```
3. **Ready to Upgrade Later?**
When you need the best performance on mobile devices, purchase the RxDB [Premium](/premium/) [SQLite Storage](../rx-storage-sqlite.md) and replace `getRxStorageLocalstorage()` with `getRxStorageSQLite()` - your app logic remains largely the same. You only have to change the configuration.
## Encryption Example
To secure local data, add the crypto-js [encryption plugin](../encryption.md) (free version) or the [premium](/premium/) web-crypto plugin. Below is an example using the free crypto-js plugin:
```ts
import { wrappedKeyEncryptionCryptoJsStorage } from 'rxdb/plugins/encryption-crypto-js';
import { getRxStorageLocalstorage } from 'rxdb/plugins/storage-localstorage';
import { createRxDatabase } from 'rxdb/plugins/core';
async function initEncryptedDB() {
const encryptedStorage = wrappedKeyEncryptionCryptoJsStorage({
storage: getRxStorageLocalstorage()
});
const db = await createRxDatabase({
name: 'secureIonicDB',
storage: encryptedStorage,
password: 'myS3cretP4ssw0rd'
});
await db.addCollections({
secrets: {
schema: {
title: 'secret schema',
version: 0,
type: 'object',
primaryKey: 'id',
properties: {
id: { type: 'string' },
text: { type: 'string' }
},
required: ['id'],
// all fields in this array will be stored encrypted:
encrypted: ['text']
}
}
});
return db;
}
```
With encryption enabled:
- `text` is automatically encrypted at rest.
- [Queries](../rx-query.md) on encrypted fields are not directly possible (since data is encrypted), but once a document is loaded, RxDB decrypts it for normal usage.
## Compression Example
To minimize the storage footprint, RxDB offers a [key-compression](../key-compression.md) feature. You can enable it in your schema:
```ts
await db.addCollections({
logs: {
schema: {
title: 'logs schema',
version: 0,
keyCompression: true, // enable compression
type: 'object',
primaryKey: 'id',
properties: {
id: { type: 'string' },
message: { type: 'string' },
createdAt: { type: 'string', format: 'date-time' }
}
}
}
});
```
With `keyCompression: true`, RxDB shortens field names internally, significantly reducing document size. This helps both stored data and network transport during replication.
## RxDB vs. Other Ionic Storage Options
**Ionic Native Storage** or **Capacitor-based** key-value stores may handle small amounts of data but lack advanced features like:
- Complex queries
- Full NoSQL document model
- [Offline-first](../offline-first.md) [sync](../replication.md)
- Encryption & key compression out of the box
- RxDB stands out by delivering all these capabilities in a unified library.
## Follow Up
For Ionic storage that supports offline-first operations, built-in encryption, optional data compression, and live syncing with any backend, RxDB provides a powerful solution. Start quickly with [localstorage](../rx-storage-localstorage.md) for local development and testing - then scale up to the premium SQLite storage for optimal performance on production mobile devices.
Ready to learn more?
- Explore the [RxDB Quickstart Guide](../quickstart.md)
- Check out [RxDB Encryption](../encryption.md) to protect user data
- Learn about [SQLite Storage](../rx-storage-sqlite.md) in [RxDB Premium](/premium/) for top [performance](../rx-storage-performance.md) on mobile.
- Join our community on the [RxDB Chat](/chat/)
**RxDB** - The ultimate toolkit for Ionic developers seeking offline-first, secure, and compressed local data, with real-time sync to any server.
---
## Local JavaScript Vector Database that works offline
# Local Vector Database with RxDB and transformers.js in JavaScript
The [local-first](../offline-first.md) revolution is here, changing the way we build apps! Imagine a world where your app's data lives right on the user's device, always available, even when there's no internet. That's the magic of local-first apps. Not only do they bring faster performance and limitless scalability, but they also empower users to work offline without missing a beat. And leading the charge in this space are local database solutions, like [RxDB](https://rxdb.info/).
But here's where things get even more exciting: when building [local-first](./local-first-future.md) apps, traditional databases often fall short. They're great at searching for exact matches, like `numbers` or `strings`, but what if you want to search by **meaning**, like sifting through emails to find a specific topic? Sure, you could use **RegExp**, but to truly unlock the power of semantic search and similarity-based queries, you need something more cutting-edge. Something that really understands the content of the data.
Enter **vector databases**, the game-changers for searching data by meaning! They have unlocked these new possibilities for storing and querying data, especially in tasks requiring **semantic search** and **similarity-based** queries. With the help of a **machine learning model**, data is transformed into a vector representation that can be stored, queried and compared in a database.
But unfortunately, most vector databases are designed for server-side use, typically running in large cloud clusters, not to run on a users device. To fix that, in this article, we will combine **RxDB** and **transformers.js** to create a local vector database running in the **browser** with **JavaScript**. It stores data in **[IndexedDB](../rx-storage-indexeddb.md)**, and uses a machine learning model with **WebAssembly** locally, without the need for external servers.
- [transformers.js](https://github.com/xenova/transformers.js) is a powerful framework that allows machine learning models to run directly within JavaScript using WebAssembly or WebGPU.
- [RxDB](https://rxdb.info/) is a [NoSQL](./in-memory-nosql-database.md), local-first database with a flexible storage layer that can run on any JavaScript runtime, including browsers and mobile environments. (You are reading this article on the RxDB docs).
A local vector database offers several key benefits:
- **Zero network latency**: Data is processed locally on the user's device, ensuring near-instant responses.
- **Offline functionality**: Data can be queried even without an internet connection.
- **Enhanced privacy**: Sensitive information remains on the device, never needing to leave for external processing.
- **Simple setup**: No backend servers are required, making deployment straightforward.
- **Cost savings**: By running everything locally, you avoid fees for API access or cloud services for large language models.
:::note
In this article only the important source code parts are shown. You can find the full open-source vector database implementation at the [github repository](https://github.com/pubkey/javascript-vector-database).
:::
## What is a Vector Database?
A vector database is a specialized database optimized for storing and querying data in the form of **high-dimensional** vectors, often referred to as **embeddings**. These embeddings are numerical representations of data, such as text, images, or audio, created by machine learning models like [MiniLM](https://huggingface.co/Xenova/all-MiniLM-L6-v2). Unlike traditional databases that work with exact matches on predefined fields, vector databases focus on **semantic similarity**, allowing you to query data based on meaning rather than exact values.
> A vector, or embedding, is essentially an array of numbers, like `[0.56, 0.12, -0.34, -0.90]`.
For example, instead of asking "Which document has the word 'database'?", you can query "Which documents discuss similar topics to this one?" The vector database compares embeddings and returns results based on how similar the vectors are to each other.
Vector databases handle multiple types of data beyond **text**, including **images**, **videos**, and **audio** files, all transformed into embeddings for efficient querying. Mostly you would not train a model by yourself and instead use one of the public available [transformer models](https://huggingface.co/models?pipeline_tag=feature-extraction&library=transformers.js).
Vector databases are highly effective in various types of applications:
- **Similarity Search**: Finds the closest matches to a query, even when the query doesn't contain the exact terms.
- **Clustering**: Groups similar items based on the proximity of their vector representations.
- **Recommendations**: Suggests items based on shared characteristics.
- **Anomaly Detection**: Identifies outliers that differ from the norm.
- **Classification**: Assigns categories to data based on its vector's nearest neighbors.
In this tutorial, we will build a vector database designed as a **Similarity Search** for **text**. For other use cases, the setup can be adapted accordingly. This flexibility is why [RxDB](https://rxdb.info/) doesn't provide a dedicated vector-database plugin, but rather offers utility functions to help you build your own vector search system.
## Generating Embeddings Locally in a Browser
For the first step to build a local-first vector database we need to compute embeddings directly on the user's device. This is where [transformers.js](https://github.com/xenova/transformers.js) from [huggingface](https://huggingface.co/docs/transformers.js/index) comes in, allowing us to run machine learning models in the browser with **WebAssembly**. Below is an implementation of a `getEmbeddingFromText()` function, which takes a piece of text and transforms it into an embedding using the [Xenova/all-MiniLM-L6-v2](https://huggingface.co/Xenova/all-MiniLM-L6-v2) model:
```js
import { pipeline } from "@xenova/transformers";
const pipePromise = pipeline('feature-extraction', 'Xenova/all-MiniLM-L6-v2');
async function getEmbeddingFromText(text) {
const pipe = await pipePromise;
const output = await pipe(text, {
pooling: "mean",
normalize: true,
});
return Array.from(output.data);
}
```
This function creates an embedding by running the text through a pre-trained model and returning it in the form of an array of numbers, which can then be stored and further processed locally.
:::note
Vector embeddings from different machine learning models or versions are not compatible with each other. When you change your model, you have to recreate all embeddings for your data.
:::
## Storing the Embeddings in RxDB
To store the embeddings, first we have to create our [RxDB Database](../rx-database.md) with the [localstorage storage](../rx-storage-localstorage.md) that stores data in the browsers [localstorage](./localstorage.md). For more advanced projects, you can use any other [RxStorage](../rx-storage.md).
```ts
import { createRxDatabase } from 'rxdb';
import { getRxStorageLocalstorage } from 'rxdb/plugins/storage-localstorage';
const db = await createRxDatabase({
name: 'mydatabase',
storage: getRxStorageLocalstorage()
});
```
Then we add a `items` collection that stores our documents with the `text` field that stores the content.
```ts
await db.addCollections({
items: {
schema: {
version: 0,
primaryKey: 'id',
type: 'object',
properties: {
id: {
type: 'string',
maxLength: 20
},
text: {
type: 'string'
}
},
required: ['id', 'text']
}
}
});
const itemsCollection = db.items;
```
In our [example repo](https://github.com/pubkey/javascript-vector-database), we use the [Wiki Embeddings](https://huggingface.co/datasets/Supabase/wikipedia-en-embeddings) dataset from supabase which was transformed and used to fill up the `items` collection with test data.
```ts
const imported = await itemsCollection.count().exec();
const response = await fetch('./files/items.json');
const items = await response.json();
const insertResult = await itemsCollection.bulkInsert(
items
);
```
Also we need a `vector` collection that stores our embeddings.
RxDB, as a NoSQL database, allows for the storage of flexible data structures, such as embeddings, within documents. To achieve this, we need to define a [schema](../rx-schema.md) that specifies how the embeddings will be stored alongside each document. The schema includes fields for an `id` and the `embedding` array itself.
```ts
await db.addCollections({
vector: {
schema: {
version: 0,
primaryKey: 'id',
type: 'object',
properties: {
id: {
type: 'string',
maxLength: 20
},
embedding: {
type: 'array',
items: {
type: 'string'
}
}
},
required: ['id', 'embedding']
}
}
});
const vectorCollection = db.vector;
```
When storing documents in the database, we need to ensure that the embeddings for these documents are generated and stored automatically. This requires a handler that runs during every document write, calling the machine learning model to generate the embeddings and storing them in a separate vector collection.
Since our app runs in a browser, it's essential to avoid duplicate work when **multiple browser tabs** are open and ensure efficient use of resources. Furthermore, we want the app to resume processing documents from where it left off if it's closed or interrupted. To achieve this, RxDB provides a [pipeline plugin](../rx-pipeline.md), which allows us to set up a workflow that processes items and stores their embeddings. In our example, a pipeline takes batches of 10 documents, generates embeddings, and stores them in a separate vector collection.
```ts
const pipeline = await itemsCollection.addPipeline({
identifier: 'my-embeddings-pipeline',
destination: vectorCollection,
batchSize: 10,
handler: async (docs) => {
await Promise.all(docs.map(async(doc) => {
const embedding = await getVectorFromText(doc.text);
await vectorCollection.upsert({
id: doc.primary,
embedding
});
}));
}
});
```
However, processing data locally presents performance challenges. Running the handler with a batch size of 10 takes around **2-4 seconds per batch**, meaning processing 10k documents would take up to an hour. To improve performance, we can do parallel processing using [WebWorkers](https://developer.mozilla.org/en-US/docs/Web/API/Web_Workers_API/Using_web_workers). A WebWorker runs on a different JavaScript process and we can start and run many of them in parallel.
Our worker listens for messages and performance the embedding generation on each request. It then sends the result embedding back to the main thread.
```ts
// worker.js
import { getVectorFromText } from './vector.js';
onmessage = async (e) => {
const embedding = await getVectorFromText(e.data.text);
postMessage({
id: e.data.id,
embedding
});
};
```
On the main thread we spawn one worker per core and send the tasks to the worker instead of processing them on the main thread.
```ts
// create one WebWorker per core
const workers = new Array(navigator.hardwareConcurrency)
.fill(0)
.map(() => new Worker(new URL("worker.js", import.meta.url)));
```
```ts
let lastWorkerId = 0;
let lastId = 0;
export async function getVectorFromTextWithWorker(text: string): Promise {
let worker = workers[lastWorkerId++];
if(!worker) {
lastWorkerId = 0;
worker = workers[lastWorkerId++];
}
const id = (lastId++) + '';
return new Promise(res => {
const listener = (ev: any) => {
if (ev.data.id === id) {
res(ev.data.embedding);
worker.removeEventListener('message', listener);
}
};
worker.addEventListener('message', listener);
worker.postMessage({
id,
text
});
});
}
const pipeline = await itemsCollection.addPipeline({
identifier: 'my-embeddings-pipeline',
destination: vectorCollection,
batchSize: navigator.hardwareConcurrency, // one per CPU core
handler: async (docs) => {
await Promise.all(docs.map(async (doc, i) => {
const embedding = await getVectorFromTextWithWorker(doc.body);
/* ... */
});
}
});
```
This setup allows us to utilize the full hardware capacity of the client's machine. By setting the batch size to match the number of logical processors available (using the [navigator.hardwareConcurrency](https://developer.mozilla.org/en-US/docs/Web/API/Navigator/hardwareConcurrency) API) and running one worker per processor, we can reduce the processing time for 10k embeddings to **about 5 minutes** on my developer laptop with 32 CPU cores.
## Comparing Vectors by calculating the distance
Now that we have stored our embeddings in the database, the next step is to compare these vectors to each other. Various methods are available to measure the similarity or difference between two vectors, such as [Euclidean distance](https://en.wikipedia.org/wiki/Euclidean_distance), [Manhattan distance](https://www.singlestore.com/blog/distance-metrics-in-machine-learning-simplfied/), [Cosine similarity](https://tomhazledine.com/cosine-similarity/), and **Jaccard similarity** (and more). RxDB provides utility functions for each of these methods, making it easy to choose the most suitable method for your application. In this tutorial, we will use **Euclidean distance** to compare vectors. However, the ideal algorithm may vary depending on your data's distribution and the specific type of query you are performing. To find the optimal method for your app, it is up to you to try out all of these and compare the results.
Each method gets two vectors as input and returns a single number. Here's how to calculate the Euclidean distance between two embeddings with the vector utilities from RxDB:
```ts
import { euclideanDistance } from 'rxdb/plugins/vector';
const distance = euclideanDistance(embedding1, embedding2);
console.log(distance); // 25.20443
```
With this we can sort multiple embeddings by how good they match our search query vector.
## Searching the Vector database with a full table scan
To find out if our embeddings have been stored correctly and that our vector comparison works as should, let's run a basic query to ensure everything functions as expected. In this query, we aim to find documents similar to a given user input text. The process involves calculating the embedding from the input text, fetching all documents, calculating the distance between their embeddings and the query embedding, and then sorting them based on their similarity.
```ts
import { euclideanDistance } from 'rxdb/plugins/vector';
import { sortByObjectNumberProperty } from 'rxdb/plugins/core';
const userInput = 'new york people';
const queryVector = await getEmbeddingFromText(userInput);
const candidates = await vectorCollection.find().exec();
const withDistance = candidates.map(doc => ({
doc,
distance: euclideanDistance(queryVector, doc.embedding)
}));
const queryResult = withDistance.sort(sortByObjectNumberProperty('distance')).reverse();
console.dir(queryResult);
```
:::note
For **distance**-based comparisons, sorting should be in ascending order (smallest first), while for **similarity**-based algorithms, the sorting should be in descending order (largest first).
:::
If we inspect the results, we can see that the documents returned are ordered by relevance, with the most similar document at the top:
:::note
This demo page can be [run online here](https://pubkey.github.io/javascript-vector-database/).
:::
However our full-scan method presents a significant challenge: it does not scale well. As the number of stored documents increases, the time taken to fetch and compare embeddings grows proportionally. For example, retrieving embeddings from our [test dataset](https://huggingface.co/datasets/Supabase/wikipedia-en-embeddings) of 10k documents takes around **700 milliseconds**. If we scale up to 100k documents, this delay would rise to approximately **7 seconds**, making the search process inefficient for larger datasets.
## Indexing the Embeddings for Better Performance
To address the scalability issue, we need to store embeddings in a way that allows us to avoid fetching all of them from storage during a query. In traditional databases, you can sort documents by an **index field**, allowing efficient queries that retrieve only the necessary documents. An index organizes data in a structured, sortable manner, much **like a phone book**. However, with vector embeddings we are not dealing with simple, single values. Instead, we have large **lists of numbers**, which makes indexing more complex because we have more than one dimension.
### Vector Indexing Methods
Various methods exist for indexing these vectors to improve query efficiency and performance:
- [Locality Sensitive Hashing (LSH)](https://www.youtube.com/watch?v=Arni-zkqMBA): LSH hashes data so that similar items are likely to fall into the same bucket, optimizing approximate nearest neighbor searches in high-dimensional spaces by reducing the number of comparisons.
- [Hierarchical Small World](https://www.youtube.com/watch?v=77QH0Y2PYKg): HSW is a graph structure designed for efficient navigation, allowing quick jumps across the graph while maintaining short paths between nodes, forming the basis for HNSW's optimization.
- [Hierarchical Navigable Small Worlds (HNSW)](https://www.youtube.com/watch?v=77QH0Y2PYKg): HNSW builds a hierarchical graph for fast approximate nearest neighbor search. It uses multiple layers where higher layers represent fewer, more connected nodes, improving search efficiency in large datasetsβ.
- **Distance to samples**: While testing different indexing strategies, [I](https://github.com/pubkey) found out that using the distance to a sample set of items is a good way to index embeddings. You pick like 5 random items of your data and get the embeddings for them out of the model. These are your 5 index vectors. For each embedding stored in the vector database, we calculate the distance to our 5 index vectors and store that `number` as an index value. This seems to work good because similar things have similar distances to other things. For example the words "shoe" and "socks" have a similar distance to "boat" and therefore should have roughly the same index value.
When building **local-first** applications, performance is often a challenge, especially in JavaScript. With **IndexedDB**, certain operations, like many sequential `get by id` calls, [are slow](../slow-indexeddb.md), while bulk operations, such as `get by index range`, are fast. Therefore, it's essential to use an indexing method that allows embeddings to be stored in a sortable way, like **Locality Sensitive Hashing** or **Distance to Samples**. In this article, we'll use **Distance to Samples**, because for [me](https://github.com/pubkey) it provides the best default behavior for the sample dataset.
### Storing indexed embeddings in RxDB
The optimal way to store index values alongside embeddings in RxDB is to place them within the same [RxCollection](../rx-collection.md). To ensure that the index values are both sortable and precise, we convert them into strings with a fixed length of `10` characters. This standardization helps in managing values with many decimals and ensures proper sorting in the database.
Here's is our schema example schema where each document contains an embedding and corresponding index fields:
```ts
const indexSchema = {
type: 'string',
maxLength: 10
};
const schema = {
"version": 0,
"primaryKey": "id",
"type": "object",
"properties": {
"id": {
"type": "string",
"maxLength": 100
},
"embedding": {
"type": "array",
"items": {
"type": "number"
}
},
// index fields
"idx0": indexSchema,
"idx1": indexSchema,
"idx2": indexSchema,
"idx3": indexSchema,
"idx4": indexSchema
},
"required": [
"id",
"embedding",
"idx0",
"idx1",
"idx2",
"idx3",
"idx4"
],
"indexes": [
"idx0",
"idx1",
"idx2",
"idx3",
"idx4"
]
}
```
To populate these index fields, we modify the [RxPipeline](../rx-pipeline.md) handler accordingly to the **Distance to samples** method. We calculate the distance between the document's embedding and our set of `5` index vectors. The calculated distances are converted to `string` and stored in the appropriate index fields:
```ts
import { euclideanDistance } from 'rxdb/plugins/vector';
const sampleVectors: number[][] = [/* the index vectors */];
const pipeline = await itemsCollection.addPipeline({
handler: async (docs) => {
await Promise.all(docs.map(async(doc) => {
const embedding = await getEmbedding(doc.text);
const docData = { id: doc.primary, embedding };
// calculate the distance to all samples and store them in the index fields
new Array(5).fill(0).map((_, idx) => {
const indexValue = euclideanDistance(sampleVectors[idx], embedding);
docData['idx' + idx] = indexNrToString(indexValue);
});
await vectorCollection.upsert(docData);
}));
}
});
```
## Searching the Vector database with utilization of the indexes
Once our embeddings are stored in an indexed format, we can perform searches much **more efficiently** than through a full table scan. While this indexing method boosts performance, it comes with a tradeoff: a slight loss in precision, meaning that the result set may not always be the optimal one. However, this is generally acceptable for **similarity search** use cases.
There are multiple ways to leverage indexes for faster queries. Here are two effective methods:
1. **Query for Index Similarity in Both Directions**: For each index vector, calculate the distance to the search embedding and fetch all relevant embeddings in both directions (sorted before and after) from that value.
```ts
async function vectorSearchIndexSimilarity(searchEmbedding: number[]) {
const docsPerIndexSide = 100;
const candidates = new Set();
await Promise.all(
new Array(5).fill(0).map(async (_, i) => {
const distanceToIndex = euclideanDistance(sampleVectors[i], searchEmbedding);
const [docsBefore, docsAfter] = await Promise.all([
vectorCollection.find({
selector: {
['idx' + i]: {
$lt: indexNrToString(distanceToIndex)
}
},
sort: [{ ['idx' + i]: 'desc' }],
limit: docsPerIndexSide
}).exec(),
vectorCollection.find({
selector: {
['idx' + i]: {
$gt: indexNrToString(distanceToIndex)
}
},
sort: [{ ['idx' + i]: 'asc' }],
limit: docsPerIndexSide
}).exec()
]);
docsBefore.map(d => candidates.add(d));
docsAfter.map(d => candidates.add(d));
})
);
const docsWithDistance = Array.from(candidates).map(doc => {
const distance = euclideanDistance((doc as any).embedding, searchEmbedding);
return {
distance,
doc
};
});
const sorted = docsWithDistance.sort(sortByObjectNumberProperty('distance')).reverse();
return {
result: sorted.slice(0, 10),
docReads
};
}
```
2. **Query for an Index Range with a Defined Distance**: Set an `indexDistance` and retrieve all embeddings within a specified range from the index vector to the search embedding.
```ts
async function vectorSearchIndexRange(searchEmbedding: number[]) {
await pipeline.awaitIdle();
const indexDistance = 0.003;
const candidates = new Set();
let docReads = 0;
await Promise.all(
new Array(5).fill(0).map(async (_, i) => {
const distanceToIndex = euclideanDistance(sampleVectors[i], searchEmbedding);
const range = distanceToIndex * indexDistance;
const docs = await vectorCollection.find({
selector: {
['idx' + i]: {
$gt: indexNrToString(distanceToIndex - range),
$lt: indexNrToString(distanceToIndex + range)
}
},
sort: [{ ['idx' + i]: 'asc' }],
}).exec();
docs.map(d => candidates.add(d));
docReads = docReads + docs.length;
})
);
const docsWithDistance = Array.from(candidates).map(doc => {
const distance = euclideanDistance((doc as any).embedding, searchEmbedding);
return {
distance,
doc
};
});
const sorted = docsWithDistance.sort(sortByObjectNumberProperty('distance')).reverse();
return {
result: sorted.slice(0, 10),
docReads
};
};
```
Both methods allow you to limit the number of embeddings fetched from storage while still ensuring a reasonably precise search result. However, they differ in how many embeddings are read and how precise the results are, with trade-offs between performance and accuracy. The first method has a known embedding read amount of `docsPerIndexSide * 2 * [amount of indexes]`. The second method reads out an unknown amount of embeddings, depending on the sparsity of the dataset and the value of `indexDistance`.
And that's it for the implementation. We now have a local first vector database that is able to store and query vector data.
## Performance benchmarks
In server-side databases, performance can be improved by scaling hardware or adding more servers. However, [local-first](../offline-first.md) apps face the unique challenge that the hardware is determined by the end user, making performance unpredictable. Some users may have **high-end gaming PCs**, while others might be using **outdated smartphones in power-saving mode**. Therefore, when building a local-first app that processes more than a few documents, performance becomes a critical factor and should be thoroughly tested upfront.
Let's run performance benchmarks on my **high-end gaming PC** to give you a sense of how long different operations take and what's achievable.
### Performance of the Query Methods
| Query Method | Time in milliseconds | Docs read from storage |
| ---------------- | -------------------- | ---------------------- |
| Full Scan | 765 | 10000 |
| Index Similarity | 1647 | 934 |
| Index Range | 88 | 2187 |
As shown, the **index similarity** query method takes significantly longer compared to others. This is due to the need for descending sort orders in some queries `sort: [{ ['idx' + i]: 'desc' }]`. While RxDB supports descending sorts, performance suffers because IndexedDB does not efficiently handle [reverse indexed bulk operations](https://github.com/w3c/IndexedDB/issues/130). As a result, the **index range method** performs much better for this use case and should be used instead. With its query time of only `88` milliseconds it is fast enough for all most things and likely such fast that you do not even need to show a loading spinner. Also it is faster compared to fetching the query result from a server-side vector database over the internet.
### Performance of the Models
Let's also look at the time taken to calculate a single embedding across various models from the [huggingface transformers list](https://huggingface.co/models?pipeline_tag=feature-extraction&library=transformers.js):
| Model Name | Time per Embedding in (ms) | Vector Size | Model Size (MB) |
| -------------------------------------------- | -------------------------- | ----------- | --------------- |
| Xenova/all-MiniLM-L6-v2 | 173 | 384 | 23 |
| Supabase/gte-small | 341 | 384 | 34 |
| Xenova/paraphrase-multilingual-mpnet-base-v2 | 1000 | 768 | 279 |
| jinaai/jina-embeddings-v2-base-de | 1291 | 768 | 162 |
| jinaai/jina-embeddings-v2-base-zh | 1437 | 768 | 162 |
| jinaai/jina-embeddings-v2-base-code | 1769 | 768 | 162 |
| mixedbread-ai/mxbai-embed-large-v1 | 3359 | 1024 | 337 |
| WhereIsAI/UAE-Large-V1 | 3499 | 1024 | 337 |
| Xenova/multilingual-e5-large | 4215 | 1024 | 562 |
From these benchmarks, it's evident that models with larger vector outputs **take longer to process**. Additionally, the model size significantly affects performance, with larger models requiring more time to compute embeddings. This trade-off between model complexity and performance must be considered when choosing the right model for your use case.
## Potential Performance Optimizations
There are multiple other techniques to improve the performance of your local vector database:
- **Shorten embeddings**: The storing and retrieval of embeddings can be improved by "shortening" the embedding. To do that, you just strip away numbers from your vector. For example `[0.56, 0.12, -0.34, 0.78, -0.90]` becomes `[0.56, 0.12]`. That's it, you now have a smaller embedding that is faster to read out of the storage and calculating distances is faster because it has to process less numbers. The downside is that you loose precision in your search results. Sometimes shortening the embeddings makes more sense as a pre-query step where you first compare the shortened vectors and later fetch the "real" vectors for the 10 most matching documents to improve their sort order.
- **Optimize the variables in our Setup**: In this examples we picked our variables in a non-optimal way. You can get huge performance improvements by setting different values:
- We picked 5 indexes for the embeddings. Using less indexes improves your query performance with the cost of less good results.
- For queries that search by fetching a specific embedding distance we used the `indexDistance` value of `0.003`. Using a lower value means we read less document from the storage. This is faster but reduces the precision of the results which means we will get a less optimal result compared to a full table scan.
- For queries that search by fetching a given amount of documents per index side, we set the value `docsPerIndexSide` to `100`. Increasing this value means you fetch more data from the storage but also get a better precision in the search results. Decreasing it can improve query performance with worse precision.
- **Use faster models**: There are many ways to improve performance of machine learning models. If your embedding calculation is too slow, try other models. **Smaller** mostly means **faster**. The model `Xenova/all-MiniLM-L6-v2` which is used in this tutorial is about [1 year old](https://huggingface.co/Xenova/all-MiniLM-L6-v2/tree/main). There exist better, more modern models to use. Huggingface makes these convenient to use. You only have to switch out the model name with any other model from [that site](https://huggingface.co/models?pipeline_tag=feature-extraction&library=transformers.js).
- **Narrow down the search space**: By utilizing other "normal" filter operators to your query, you can narrow down the search space and optimize performance. For example in an email search you could additionally use a operator that limits the results to all emails that are not older than one year.
- **Dimensionality Reduction** with an [autoencoder](https://www.youtube.com/watch?v=D16rii8Azuw): An autoencoder encodes vector data with minimal loss which can improve the performance by having to store and compare less numbers in an embedding.
- **Different RxDB Plugins**: RxDB has different storages and plugins that can improve the performance like the [IndexedDB RxStorage](../rx-storage-indexeddb.md), the [OPFS RxStorage](../rx-storage-opfs.md), the [sharding](../rx-storage-sharding.md) plugin and the [Worker](../rx-storage-worker.md) and [SharedWorker](../rx-storage-shared-worker.md) storages.
## Migrating Data on Model/Index Changes
When you change the index parameter or even update the whole model which was used to create the embeddings, you have to migrate the data that is already stored on your users devices. RxDB offers the [Schema Migration Plugin](../migration-schema.md) for that.
When the app is reloaded and the updated source code is started, RxDB detects changes in your [schema version](../rx-schema.md#version) and runs the [migration strategy](../migration-schema.md#providing-strategies) accordingly. So to update the stored data, increase the schema version and define a handler:
```ts
const schemaV1 = {
"version": 1, // <- increase schema version by 1
"primaryKey": "id",
"properties": {
/* ... */
},
/* ... */
};
```
In the migration handler we recreate the new embeddings and index values.
```ts
await myDatabase.addCollections({
vectors: {
schema: schemaV1,
migrationStrategies: {
1: function(docData){
const embedding = await getEmbedding(docData.body);
new Array(5).fill(0).map((_, idx) => {
docData['idx' + idx] = euclideanDistance(mySampleVectors[idx], embedding);
});
return docData;
},
}
}
});
```
## Possible Future Improvements to Local-First Vector Databases
For now our vector database works and we are good to go. However there are some things to consider for the future:
- **WebGPU** is [not fully supported](https://caniuse.com/webgpu) yet. When this changes, creating embeddings in the browser have the potential to become faster. You can check if your current chrome supports WebGPU by opening `chrome://gpu/`. Notice that WebGPU has been reported to sometimes be [even slower](https://github.com/xenova/transformers.js/issues/894#issuecomment-2323897485) compared to WASM but likely it will be faster in the long term.
- **Cross-Modal AI Models**: While progress is being made, AI models that can understand and integrate multiple modalities are still in development. For example you could query for an **image** together with a **text** prompt to get a more detailed output.
- **Multi-Step queries**: In this article we only talked about having a single query as input and an ordered list of outputs. But there is big potential in chaining models or queries together where you take the results of one query and input them into a different model with different embeddings or outputs.
## Follow Up
- Shared/Like my [announcement tweet](https://x.com/rxdbjs/status/1833429569434427494)
- Read the source code that belongs to this article [at github](https://github.com/pubkey/javascript-vector-database)
- Learn how to use RxDB with the [RxDB Quickstart](../quickstart.md)
- Check out the [RxDB github repo](https://github.com/pubkey/rxdb) and leave a star β
---
## RxDB as a Database in a jQuery Application
import {VideoBox} from '@site/src/components/video-box';
# RxDB as a Database in a jQuery Application
In the early days of dynamic web development, **jQuery** emerged as a popular library that simplified DOM manipulation and AJAX requests. Despite the rise of modern frameworks, many developers still maintain or extend existing jQuery projects, or leverage jQuery in specific contexts. As jQuery applications grow in complexity, they often require efficient data handling, offline support, and synchronization capabilities. This is where [RxDB](https://rxdb.info/), a reactive JavaScript database for the browser, node.js, and [mobile devices](./mobile-database.md), steps in.
## jQuery Web Applications
jQuery provides a simple API for DOM manipulation, event handling, and AJAX calls. It has been widely adopted due to its ease of use and strong community support. Many projects continue to rely on jQuery for handling client-side functionality, UI interactions, and animations. As these applications evolve, the need for a robust database solution that can manage data locally (and offline) becomes increasingly important.
## Importance of Databases in jQuery Applications
Modern, data-driven jQuery applications often need to:
- **Store and retrieve data locally** for quick and responsive user experiences.
- **Synchronize data** between clients or with a [central server](../rx-server.md).
- **Handle offline scenarios** seamlessly.
- **Handle large or complex data structures** without repeatedly hitting the server.
Relying solely on server endpoints or basic browser storage (like `localStorage`) can quickly become unwieldy for larger or more complex use cases. Enter RxDB, a dedicated solution that manages data on the client side while offering real-time synchronization and offline-first capabilities.
## Introducing RxDB as a Database Solution
RxDB (short for Reactive Database) is built on top of [IndexedDB](./browser-database.md) and leverages [RxJS](https://rxjs.dev/) to provide a modern, reactive approach to handling data in the browser. With RxDB, you can store documents locally, query them in real-time, and synchronize changes with a remote server whenever an internet connection is available.
### Key Features
- **Reactive Data Handling**: RxDB emits real-time updates whenever your data changes, allowing you to instantly reflect these changes in the DOM with jQuery.
- **Offline-First Approach**: Keep your application usable even when the user's network is unavailable. Data is automatically synchronized once connectivity is restored.
- **Data Replication**: Enable multi-device or multi-tab synchronization with minimal effort.
- **[Observable Queries](../rx-query.md)**: Reduce code complexity by subscribing to queries instead of constantly polling for changes.
- **Multi-Tab Support**: If a user opens your jQuery application in multiple tabs, RxDB keeps data in sync across all sessions.
## Getting Started with RxDB
### What is RxDB?
[RxDB](https://rxdb.info/) is a client-side NoSQL database that stores data in the browser (or [node.js](../nodejs-database.md)) and synchronizes changes with other instances or servers. Its design embraces reactive programming principles, making it well-suited for real-time applications, offline scenarios, and multi-tab use cases.
### Reactive Data Handling
RxDB's use of observables enables an event-driven architecture where data mutations automatically trigger UI updates. In a jQuery application, you can subscribe to these changes and update DOM elements as soon as data changes occur - no need for manual refresh or complicated change detection logic.
### Offline-First Approach
One of RxDB's distinguishing traits is its emphasis on offline-first design. This means your jQuery application continues to function, display, and update data even when there's no network connection. When connectivity is restored, RxDB synchronizes updates with the server or other peers, ensuring consistency across all instances.
### Data Replication
RxDB supports real-time data replication with different backends. By enabling replication, you ensure that multiple clients - be they multiple [browser](./browser-database.md) tabs or separate devices - stay in sync. RxDB's conflict resolution strategies help keep the data consistent even when multiple users make changes simultaneously.
### Observable Queries
Instead of static queries, RxDB provides observable queries. Whenever data relevant to a query changes, RxDB re-emits the new result set. You can subscribe to these updates within your jQuery code and instantly reflect them in the UI.
### Multi-Tab Support
Running your jQuery app in multiple tabs? RxDB automatically synchronizes changes between those tabs. Users can freely switch windows without missing real-time updates.
### RxDB vs. Other jQuery Database Options
Historically, jQuery developers might use `localStorage` or raw `IndexedDB` for storing data. However, these solutions can require significant boilerplate, lack reactivity, and offer no built-in sync or conflict resolution. RxDB fills these gaps with an out-of-the-box solution, abstracting away low-level database complexities and providing an event-driven, offline-capable approach.
## Using RxDB in a jQuery Application
### Installing RxDB
Install RxDB (and `rxjs`) via npm or yarn:
```bash
npm install rxdb rxjs
```
If your project isn't set up with a build process, you can still use bundlers like Webpack or Rollup, or serve RxDB as a UMD bundle. Once included, you'll have access to RxDB globally or via import statements.
## Creating and Configuring a Database
Below is a minimal example of how to create an RxDB instance and collection. You can call this when your page initializes, then store the `db` object for later use:
```js
import { createRxDatabase } from 'rxdb';
import { getRxStorageLocalstorage } from 'rxdb/plugins/storage-localstorage';
async function initDatabase() {
const db = await createRxDatabase({
name: 'heroesdb',
storage: getRxStorageLocalstorage(),
password: 'myPassword', // optional encryption password
multiInstance: true, // multi-tab support
eventReduce: true // optimizes event handling
});
await db.addCollections({
hero: {
schema: {
title: 'hero schema',
version: 0,
primaryKey: 'id',
type: 'object',
properties: {
id: { type: 'string' },
name: { type: 'string' },
points: { type: 'number' }
}
}
}
});
return db;
}
```
## Updating the DOM with jQuery
Once you have your RxDB instance, you can query data reactively and use jQuery to manipulate the DOM:
```js
// Example: Displaying heroes using jQuery
$(document).ready(async function () {
const db = await initDatabase();
// Subscribing to all hero documents
db.hero
.find()
.$ // the observable
.subscribe((heroes) => {
// Clear the list
$('#heroList').empty();
// Append each hero to the DOM
heroes.forEach((hero) => {
$('#heroList').append(`
${hero.name} - Points: ${hero.points}
`);
});
});
// Example of adding a new hero
$('#addHeroBtn').on('click', async () => {
const heroName = $('#heroName').val();
const heroPoints = parseInt($('#heroPoints').val(), 10);
await db.hero.insert({
id: Date.now().toString(),
name: heroName,
points: heroPoints
});
});
});
```
With this approach, any time data in the `hero` collection changes - like when a new hero is added - your jQuery code re-renders the list of heroes automatically.
## Different RxStorage layers for RxDB
RxDB supports multiple storage backends (RxStorage layers). Some popular ones:
- [LocalStorage.js RxStorage](../rx-storage-localstorage.md): Uses the browsers [localstorage](./localstorage.md). Fast and easy to set up.
- [IndexedDB RxStorage](../rx-storage-indexeddb.md): Direct IndexedDB usage, suitable for modern browsers.
- [OPFS RxStorage](../rx-storage-opfs.md): Uses the File System Access API for better performance in supported browsers.
- [Memory RxStorage](../rx-storage-memory.md): Stores data in memory, handy for tests or ephemeral data.
- [SQLite RxStorage](../rx-storage-sqlite.md): Uses SQLite (potentially via WebAssembly). In typical browser-based scenarios, localstorage or IndexedDB storage is usually more straightforward.
## Synchronizing Data with RxDB between Clients and Servers
### Offline-First Approach
RxDB's [offline-first](../offline-first.md) approach allows your jQuery application to store and query data locally. Users can continue interacting, even offline. When connectivity returns, RxDB syncs to the server.
### Conflict Resolution
Should multiple clients update the same document, RxDB offers [conflict handling strategies](../transactions-conflicts-revisions.md). You decide how to resolve conflicts - like keeping the latest edit or merging changes - ensuring data integrity across distributed systems.
### Bidirectional Synchronization
With RxDB, data changes flow both ways: from client to server and from server to client. This real-time synchronization ensures that all users or tabs see consistent, up-to-date data.
## Advanced RxDB Features and Techniques
### Indexing and Performance Optimization
Create indexes on frequently queried fields to speed up performance. For large data sets, indexing can drastically improve query times, keeping your jQuery UI snappy.
### Encryption of Local Data
RxDB supports [encryption to secure data stored in the browser](../encryption.md). This is crucial if your application handles sensitive user information.
### Change Streams and Event Handling
Use change streams to listen for data modifications at the database or collection level. This can trigger [real-time](./realtime-database.md) [UI updates](./optimistic-ui.md), notifications, or custom logic whenever the data changes.
### JSON Key Compression
If your data model has large or repetitive field names, [JSON key compression](../key-compression.md) can minimize stored document size and potentially boost performance.
## Best Practices for Using RxDB in jQuery Applications
- Centralize Your Database: Initialize and configure RxDB in one place. Expose the instance where needed or store it globally to avoid re-creating it on every script.
- Leverage Observables: Instead of polling or manually refreshing data, rely on RxDB's reactivity. Subscribe to queries and let RxDB inform you when data changes.
- Handle Subscriptions: If you create subscriptions in a single-page context, ensure you don't re-subscribe endlessly or create memory leaks. Clean them up if you're navigating away or removing DOM elements.
- Offline Testing: Thoroughly test how your jQuery app behaves without a network connection. Simulate offline states in your browser's dev tools or with flight mode to ensure the user experience remains smooth.
- Performance Profiling: For large data sets or frequent data updates, add indexes and carefully measure query performance. Optimize only where needed.
## Follow Up
To explore more about RxDB and leverage its capabilities for browser database development, check out the following resources:
- [RxDB GitHub Repository](/code/): Visit the official GitHub repository of RxDB to access the source code, documentation, and community support.
- [RxDB Quickstart](../quickstart.md): Get started quickly with RxDB by following the provided quickstart guide, which offers step-by-step instructions for setting up and using RxDB in your projects.
- [RxDB Examples](https://github.com/pubkey/rxdb/tree/master/examples): Browse official examples to see RxDB in action and learn best practices you can apply to your own project - even if jQuery isn't explicitly featured, the patterns are similar.
---
## JSON-Based Databases - Why NoSQL and RxDB Simplify App Development
# JSON-Based Databases: Why NoSQL and RxDB Simplify App Development
Modern applications handle highly dynamic, often deeply nested data structuresβcommonly represented in **JSON**. Whether you're building a real-time dashboard or a fully offline mobile app, storing and querying data in a JSON-friendly way can reduce overhead and coding complexity. This is where **JSON-based databases** (often part of the **NoSQL** family) come into play, letting you store objects in the same format they're used in your code, eliminating the schema wrangling that can come with a strict relational design.
Below, we explore why JSON-based databases naturally align with **NoSQL** principles, how relational engines (like PostgreSQL or SQLite) handle JSON columns, the pitfalls of storing data in a single plain JSON text file, and the ways [RxDB](https://rxdb.info/) stands out as an offline-first JSON solution for JavaScript developersβcomplete with advanced features like JSON-Schema and JSON-key-compression.
## Why JSON-Based Databases Are Typically NoSQL
### Document-Oriented by Nature
When your data is stored as JSON, each record or document can hold nested arrays and sub-objects with no forced table schema. NoSQL solutions such as [MongoDB](../rx-storage-mongodb.md), [CouchDB](../replication-couchdb.md), [Firebase](../replication-firestore.md), and **RxDB** store and retrieve these documents in their βrawβ JSON form. This model integrates smoothly with how front-end applications already handle data, minimizing transformations and improving developer productivity.
### Flexible, Schema-Agnostic
Traditional SQL tables enforce rigid column definitions and demand explicit schema migrations when you add or rename a field. By contrast, NoSQL solutions accept more dynamic data structures, allowing changes on the fly. This means a front-end developer can add a new field to a JSON objectβperhaps for a new featureβwithout the friction of redefining or migrating a database schema. While this is possible, it is often not recommended.
### Aligned With Evolving User Interfaces
As modern UIs frequently manipulate deeply nested or changing data, developers find it easier to store whole objects directly, saving time that might otherwise be spent performing complex joins or normalizing data. For instance, frameworks like [React](./react-database.md), [Vue](./vue-database.md), or [Angular](./angular-database.md) are inherently comfortable with nested JSON structures, which map more directly toNoSQLβs βdocumentβ approach than to relational tables.
## Is NoSQL βBetterβ Than SQL?
It depends on your application. **SQL** remains exceptional for complex aggregations, enforced relationships, and sophisticated transaction handling. But **NoSQL** is often more intuitive and easier to maintain for βdocument-firstβ applications that:
- Thrive on flexible or rapidly evolving data models.
- Rely on hierarchical or nested JSON objects.
- Avoid multi-table joins.
- Require easy horizontal scaling for large sets of documents.
Relational databases are still a top choice for many enterprise back-ends, especially when advanced analytics or strongly enforced referential integrity is needed. But if your application is predominantly storing and manipulating JSON documents (e.g., user profiles, real-time chat logs, embedded items), a JSON-based or document-oriented approach can greatly reduce friction during development.
## When to Prefer SQL Instead of JSON/NoSQL
NoSQL solutionsβparticularly JSON-based document storesβprovide a natural fit for flexible, nested data in UI-heavy applications. However, certain scenarios may benefit more from a **SQL** solution:
1. **Complex Relationships**: If your data demands intricate joins across multiple entities (e.g., many-to-many relationships that canβt easily be embedded in a single document), a well-structured relational schema can simplify queries.
2. **Strong Integrity and Constraints**: SQL excels at enforcing constraints such as foreign keys, unique constraints, and advanced triggers. If your system needs strict data validation and complex business logic within the database, SQL might prove more robust.
3. **High-End Analytical Queries**: Relational databases can handle sophisticated aggregations, groupings, and joins more efficiently. If your app frequently runs advanced SQL queries, a NoSQL approach may complicate or slow down analytics.
4. **Legacy Integration**: Many enterprise systems are built around existing relational schemas. A purely NoSQL approach might mean rewriting or bridging systems that are heavily reliant on SQL constraints and transformations.
5. **Transaction Handling**: While many NoSQL solutions have improved transaction support, it can still lag behind well-established SQL transaction models. If ACID properties and multi-operation atomicity are paramount, you might prefer a tried-and-true relational engine.
In short, if you prioritize advanced relational queries, robust constraints, or complex business rules at the database level, SQL remains a powerful, and possibly superior, choice. For user-centric, fast-evolving JSON data, though, NoSQL or JSON-based solutions often reduce the friction of frequent schema changes.
## Storing JSON in Traditional SQL Databases
### JSON Columns in PostgreSQL or MySQL
To accommodate the demand for flexible data, several SQL engines (notably **PostgreSQL** and MySQL) introduced support for **JSON** columns. PostgreSQL offers the `JSON` and `JSONB` types, enabling developers to store raw JSON in a column. You can also index specific paths within the JSON to speed lookups on nested fields:
```sql
CREATE TABLE products (
id SERIAL PRIMARY KEY,
name TEXT,
details JSONB
);
-- Insert a record with JSON data
INSERT INTO products (name, details) VALUES
('Laptop', '{"brand": "BrandX", "features": ["Touchscreen", "SSD"]}');
```
Although this approach merges the best of both worlds (SQL queries + flexible JSON fields), it can also create a βsplit personalityβ in your schema. You might store stable data in normal columns, while unpredictable or nested details live inside a JSONB field. Some projects flourish with this hybrid design, others find it a bit unwieldy.
## Storing JSON in SQLite
SQLite also allows storing JSON data, typically as text columns, but with some additional features since **SQLite 3.9** (2015) including the [JSON1 extension](https://www.sqlite.org/json1.html). This extension can parse JSON text, perform queries on JSON fields, and do partial updates. However, storing JSON in SQLite does require you to ensure youβve compiled SQLite with JSON1 support or to rely on a library that bundles it. While possible, you still won't get quite the same schema-agnostic ease as a full document store, but itβs a pragmatic solution for smaller or embedded needs on the server sideβor occasionally in the browser if you run SQLite via WebAssembly. RxDB uses this in its [SQLite storage](../rx-storage-sqlite.md).
## JSON vs. Database - Why a Plain JSON Text File is a Problem
Some developers consider storing everything in a single JSON file, typically read and written directly from disk or local storage. This approach, while seemingly simple, usually does not scale. Key issues include:
- **No Concurrency**: If multiple parts of the application try to write to the same JSON file, you risk overwriting changes.
- **No Indexes**: Finding or filtering items in large JSON text requires scanning everything. This is slow and quickly becomes unmanageable.
- **No Partial Updates**: You often reload the entire file, modify it in memory, then write it back, which is highly inefficient for large data sets.
- **Corruption Risk**: A single corrupted write or partial save might break the entire JSON file, losing all data.
- **High Memory Usage**: The entire file may need to be parsed into memory, even if you only need a fraction of the data.
Databasesβrelational or NoSQLβsolve these issues by handling concurrency, enabling partial reads/writes, establishing indexes, and ensuring transactional integrity so you donβt lose everything if the process is interrupted mid-write.
## RxDB: A JSON-Focused Database for JavaScript Apps
Many NoSQL databases operate on the server, whereas RxDB is built for client-side usageβbrowsers, mobile apps, or [Node.js](../nodejs-database.md). It specializes in JSON documents and embraces an [offline-first](../offline-first.md) philosophy.
### Key Characteristics
1. **Local JSON Storage**
RxDB stores each record as a JSON document, closely matching how front-end frameworks handle state. This eliminates complex transformations or manual JSON parsing before writing to a table.
2. **Reactive Queries**
Instead of complex SQL, RxDB uses JSON-based [query](../rx-query.md) definitions. You can subscribe to query results, letting your UI automatically refresh when data changes locally or from remote sync updates:
3. **Offline-First Sync**
Built-in replication plugins push/pull changes to or from a remote server. If your app is offline, updates get stored locally, then sync up seamlessly once a connection is available.
4. **Optional JSON-Schema**
Though itβs a document database, RxDB encourages you to define a JSON-based schema for clarity, indexing, and type validation. This helps maintain data consistency while still allowing a measure of flexibility for new fields.
### Advanced JSON Features in RxDB
- **JSON-Schema**: By specifying a JSON-Schema, you can define which fields exist, whether they are required, and their data types. This is invaluable for catching malformed documents early and imposing mild structure in a NoSQL setting.
- **JSON Key-Compression**: Large, verbose field names can bloat storage usage. RxDBβs optional [key-compression plugin](../key-compression.md) automatically shortens field names in your JSON documents internally, reducing disk space and bandwidth:
```ts
// Example: how key-compression can transform your documents
const uncompressed = {
"firstName": "Corrine",
"lastName": "Ziemann",
"shoppingCartItems": [
{ "productNumber": 29857, "amount": 1 },
{ "productNumber": 53409, "amount": 6 }
]
};
const compressed = {
"|e": "Corrine",
"|g": "Ziemann",
"|i": [
{ "|h": 29857, "|b": 1 },
{ "|h": 53409, "|b": 6 }
]
};
```
The user sees no difference in their codeβRxDB automatically decompresses data on readβbut the overhead is drastically reduced behind the scenes.
## Follow Up
JSON-based databases naturally align with NoSQL because they accommodate evolving, nested data without rigid schemas. This makes them appealing for many UI-centric or offline-first applications where flexible documents and agile development cycles matter more than heavy relational queries or constraints.
SQL can still store JSONβwhether in PostgreSQLβs JSONB columns, MySQLβs JSON fields, or SQLiteβs JSON1 extension. For some teams, a hybrid approach pairing SQL for relational data with JSON columns for more flexible fields works well. However, storing everything in a single monolithic JSON text file is rarely advisable for anything beyond trivial tasksβdatabases excel at concurrency, indexing, and partial writes.
Tools like RxDB provide an even simpler, [local-first](./local-first-future.md) take on JSON documentsβparticularly for JavaScript projects. With offline [replication](../replication.md), reactive queries, optional JSON-Schema, and advanced optimizations such as key-compression, RxDB streamlines building dynamic, user-facing features while preserving the core benefits of a robust document database.
To explore more about RxDB and its capabilities for browser database development, check out the following resources:
- [RxDB GitHub Repository](/code/): Visit the official GitHub repository of RxDB to access the source code, documentation, and community support.
- [RxDB Quickstart](../quickstart.md): Get started quickly with RxDB by following the provided quickstart guide, which offers step-by-step instructions for setting up and using RxDB in your projects.
- [RxDB Examples](https://github.com/pubkey/rxdb/tree/master/examples): Browse official examples to see RxDB in action and learn best practices you can apply to your own project - even if jQuery isn't explicitly featured, the patterns are similar.
---
## RxDB - The JSON Database Built for JavaScript
# RxDB - JSON Database for JavaScript
Storing data as **JSON documents** in a **[NoSQL](./in-memory-nosql-database.md)** database is not just a trend; it's a practical choice. JSON data is highly compatible with various tools and is human-readable, making it an excellent fit for modern applications. JSON documents offer more flexibility compared to traditional SQL table rows, as they can contain nested data structures. This article introduces [RxDB](https://rxdb.info/), an open-source, flexible, performant, and battle-tested NoSQL JSON database specifically designed for **JavaScript** applications.
## Why Choose a JSON Database?
- **JavaScript Friendliness**: JavaScript, a prevalent language for web development, naturally uses JSON for data representation. Using a JSON database aligns seamlessly with JavaScript's native data format.
- **Compatibility**: JSON is widely supported across different programming languages and platforms. Storing data in JSON format ensures compatibility with a broad range of tools and systems. All modern programming ecosystems have packages to parse, validate and process JSON data.
- **Flexibility**: JSON documents can accommodate complex and nested data structures, allowing developers to store data in a more intuitive and hierarchical manner compared to SQL table rows. Nested data can be just stored in-document instead of having related tables.
- **Human-Readable**: JSON is easy to read and understand, simplifying debugging and data inspection tasks.
## Storage and Access Options for JSON Documents
When incorporating JSON documents into your application, you have several storage and access options to consider:
- **Local In-App Database with In-Memory Storage**: Ideal for lightweight applications or temporary data storage, this option keeps data in memory, ensuring fast read and write operations. However, data is not persistet beyond the current application session, making it suitable for temporary data storage. With RxDB, the [memory RxStorage](../rx-storage-memory.md) can be utilized to create an in-memory database.
- **Local In-App Database with Persistent Storage**: Suitable for applications requiring data retention across sessions. Data is stored on the user's device or inside of the Node.js application, offering persistence between application sessions. It balances speed and data retention, making it versatile for various applications. With RxDB, a whole range of persistent storages is available. As example, for browser there is the [IndexedDB storage](../rx-storage-indexeddb.md). For server side applications, the [Node.js Filesystem storage](../rx-storage-filesystem-node.md) can be used. There are [many more storages](../rx-storage.md) for React-Native, Flutter, Capacitors.js and others.
- **Server Database Connected to the Application**: For applications requiring data synchronization and accessibility from multiple processes, a server-based database is the preferred choice. Data is stored on a **remote server**, facilitating data sharing, synchronization, and accessibility across multiple processes. It's suitable for scenarios requiring centralized data management and enhanced security and backup capabilities on the server. RxDB supports the [FoundationDB](../rx-storage-foundationdb.md) and [MongoDB](../rx-storage-mongodb.md) as a remote database server.
## Compression Storage for JSON Documents
Compression storage for JSON documents is made effortless with RxDB's [key-compression plugin](../key-compression.md). This feature enables the efficient storage of compressed document data, reducing storage requirements while maintaining data integrity. Queries on compressed documents remain seamless, ensuring that your application benefits from both space-saving advantages and optimal query performance, making RxDB a compelling choice for managing JSON data efficiently. The compression happens inside of the [RxDatabase](../rx-database.md) and does not affect the API usage. The only limitation is that encrypted fields themself cannot be used inside a query.
## Schema Validation and Data Migration on Schema Changes
Storing JSON documents inside of a database in an application, can cause a problem when the format of the data changes. Instead of having a single server where the data must be migrated, many client devices are out there that have to run a migration.
When your application's schema evolves, RxDB provides [migration strategies](../migration-schema.md) to facilitate the transition, ensuring data consistency throughout schema updates.
**JSONSchema Validation Plugins**: RxDB supports multiple [JSONSchema validation plugins](../schema-validation.md), guaranteeing that only valid data is stored in the database. RxDB uses the JsonSchema standardization that you might know from other technologies like OpenAPI (aka Swagger).
```javascript
// RxDB Schema example
const mySchema = {
version: 0,
primaryKey: 'id', // <- define the primary key for your documents
type: 'object',
properties: {
id: {
type: 'string',
maxLength: 100 // <- the primary key must have set maxLength
},
name: {
type: 'string',
maxLength: 100
},
done: {
type: 'boolean'
},
timestamp: {
type: 'string',
format: 'date-time'
}
},
required: ['id', 'name', 'done', 'timestamp']
}
```
## Store JSON with RxDB in Browser Applications
RxDB offers versatile storage solutions for browser-based applications:
- **Multiple Storage Plugins**: RxDB supports various storage backends, including [IndexedDB](../rx-storage-indexeddb.md), [localstorage](../rx-storage-localstorage.md) and [In-Memory](../rx-storage-memory.md), catering to a range of browser environments.
- **Observable Queries**: With RxDB, you can create observable [queries](../rx-query.md) that work seamlessly across multiple browser tabs, providing real-time updates and synchronization.
## RxDB JSON Database Performance
Certainly! Let's delve deeper into the performance aspects of RxDB when it comes to working with JSON data.
1. **Efficient Querying:** RxDB is engineered for rapid and efficient querying of JSON data. It employs a well-optimized indexing system that allows for lightning-fast retrieval of specific data points within your JSON documents. Whether you're fetching individual values or complex nested structures, RxDB's query performance is designed to keep your application responsive, even when dealing with large datasets.
2. **Scalability:** As your application grows and your [JSON dataset](./json-based-database.md) expands, RxDB scales gracefully. Its performance remains consistent, enabling you to handle increasingly larger volumes of data without compromising on speed or responsiveness. This scalability is essential for applications that need to accommodate growing user bases and evolving data needs.
3. **Reduced Latency:** RxDB's streamlined data access mechanisms significantly reduce latency when working with JSON data. Whether you're reading from the database, making updates, or synchronizing data between clients and servers, RxDB's optimized operations help minimize the delays often associated with data access. Observed queries are optimized with the [EventReduce algorithm](https://github.com/pubkey/event-reduce) to provide nearly-instand UI updates on data changes.
4. **RxStorage Layer**: Because RxDB allows you to swap out the storage layer. A storage with the most optimal performance can be chosen for each runtime while not touching other database code. Depending on the access patterns, you can pick exactly the storage that is best:
## RxDB in Node.js
Node.js developers can also benefit from RxDB's capabilities. By integrating RxDB into your Node.js applications, you can harness the power of a NoSQL JSON db to efficiently manage your data on the server-side. RxDB's flexibility, performance, and essential features are equally valuable in server-side development. [Read more about RxDB+Node.js](../nodejs-database.md).
## RxDB to store JSON documents in React Native
For mobile app developers working with React Native, RxDB offers a convenient solution for handling JSON data. Whether you're building Android or iOS applications, RxDB's compatibility with JavaScript and its ability to work with JSON documents make it a natural choice for data management within your React Native apps. [Read more about RxDB+React-Native](../react-native-database.md).
## Using SQLite as a JSON Database
In some cases, you might want to use SQLite as a backend storage solution for your JSON data. RxDB can be configured [to work with SQLite](../rx-storage-sqlite.md), providing the benefits of both a relational database system and JSON document storage. This hybrid approach can be advantageous when dealing with complex data relationships while retaining the flexibility of JSON data representation.
## Follow Up
To further explore RxDB and get started with using it in your frontend applications, consider the following resources:
- [RxDB Quickstart](../quickstart.md): A step-by-step guide to quickly set up RxDB in your project and start leveraging its features.
- [RxDB GitHub Repository](https://github.com/pubkey/rxdb): The official repository for RxDB, where you can find the code, examples, and community support.
By embracing [RxDB](https://rxdb.info/) as your **JSON database** solution, you can tap into the extensive capabilities of JSON data storage. This empowers your applications with offline accessibility, caching, enhanced performance, and effortless data synchronization. RxDB's focus on JavaScript and its robust feature set render it the perfect selection for frontend developers in pursuit of efficient and scalable data storage solutions.
---
## What is a Local Database and Why RxDB is the Best Local Database for JavaScript Applications
A **local database** is a data storage system residing on a user's device, allowing applications to store, query, and manipulate information without needing continuous network access. This approach prioritizes quick data retrieval, efficient updates, and the ability to function in [offline-first](../offline-first.md) scenarios. In contrast, server-based databases require an active internet connection for each request and response cycle, making them more vulnerable to latency, network disruptions, and downtime.
Local databases often leverage technologies such as **IndexedDB**, **SQLite**, or **WebSQL** (though WebSQL has been deprecated). These technologies manage both structured data (like relational tables) and unstructured data (such as [JSON documents](./json-database.md)). When connectivity is restored, local databases typically sync their changes back to a central server-side database, maintaining consistent and up-to-date records across multiple devices.
### Use Cases of Local Databases
Local databases are particularly beneficial for:
- **Offline Functionality**: Essential for apps that must remain usable without a consistent internet connection, such as note-taking apps or offline-first CRMs. Users can continue adding and editing data, then sync changes once they reconnect.
- **Low Latency**: By reducing round-trips to remote servers, local databases enable real-time responsiveness. This feature is critical for interactive applications such as gaming platforms, data dashboards, or analytics tools that need near-instant feedback.
- **Data Synchronization**: Many modern applications - like chat systems or collaborative editing tools - require continuous data exchange between multiple users or devices. Local databases can handle intermittent connectivity gracefully, queuing updates locally and syncing them when possible.
In addition, local databases are increasingly integral to **Progressive Web Apps (PWAs)**, offering a native app-like user experience that is fast and available, even when offline.
### Performance Optimization
The primary performance benefit of a local database is its proximity to the application: queries and updates happen directly on the user's device, eliminating the overhead of multiple network hops. Common optimizations include:
- **Caching**: Storing frequently accessed data in memory or on disk to minimize expensive operations.
- **Batching Writes**: Grouping database operations into a single write transaction to reduce overhead and lock contention.
- **Efficient Indexing**: Using appropriate indexes to speed up queries, especially important for applications that handle large data sets or frequent lookups.
These techniques ensure that local databases run smoothly, even on lower-powered or [mobile devices](./mobile-database.md).
### Security and Encryption
Storing data on user devices introduces unique security considerations, such as the risk of physical theft or unauthorized access. Consequently, many local databases support **encryption** options to safeguard sensitive information. Developers can implement additional security measures like **device-level encryption**, **secure storage plugins**, and user authentication to further protect data from prying eyes.
---
## Why RxDB is Optimized for JavaScript Applications
RxDB (Reactive Database) is an offline-first, NoSQL database designed to meet the needs of modern JavaScript applications. Built with a focus on reactivity and real-time data handling, RxDB excels in scenarios where low-latency, offline availability, and scalability are essential.
### Real-Time Reactivity
At the core of RxDB is **reactive programming**, allowing you to subscribe to changes in your data collections and receive immediate UI updates when records change - no manual polling or refetching required. For instance, a chat application can display incoming messages as soon as they arrive, maintaining a smooth and responsive experience.
### Offline-First Support
RxDB's primary design goal is to work seamlessly in offline environments. Even if your device loses internet connectivity, RxDB enables you to continue reading and writing data. Once the connection is restored, all pending changes are automatically synchronized with your backend. This [offline-first approach](../offline-first.md) is ideal for productivity apps, field service tools, and other scenarios where reliability and user autonomy are paramount.
### Flexible Data Replication
A standout feature of RxDB is its [bi-directional replication](../replication.md). It supports synchronization with a variety of backends, such as:
- [CouchDB](../replication-couchdb.md): Via the CouchDB replication, facilitating easy integration with any Couch-compatible server.
- [GraphQL Endpoints](../replication-graphql.md): Through community plugins, developers can replicate JSON documents to and from GraphQL servers.
- [Custom Backends](../replication-http.md): RxDB provides hooks to build custom replication strategies for proprietary or specialized server APIs.
This flexibility ensures that RxDB fits into diverse architectures without locking you into a single vendor or technology stack.
### Schema Validation and Versioning
Rather than relying on implicit data models, RxDB leverages **JSON schema** to define document structures. This approach promotes data consistency by enforcing constraints such as required fields and acceptable data formats. As your application grows and changes, RxDB's built-in **schema versioning** and migration tools help you evolve your database schema safely, minimizing risks of data corruption or loss.
### Rich Plugin Ecosystem
One of RxDB's greatest strengths is its **pluggable architecture**, allowing you to add functionality as needed:
- [Encryption](../encryption.md): Secure your data at rest using advanced encryption plugins.
- [Full-Text Search](../fulltext-search.md): Integrate powerful text search capabilities for applications that require quick and flexible query options.
- [Storage Adapters](../rx-storage.md): Swap out the underlying storage layer (e.g., [IndexedDB in the browser](../rx-storage-indexeddb.md), [SQLite](../rx-storage-sqlite.md) in [React Native](../react-native-database.md), or a custom engine) without rewriting your application logic.
You can fine-tune RxDB to your exact needs, avoiding the performance overhead of unnecessary features.
### Multi-Platform Compatibility
RxDB is a perfect fit for cross-platform development, as it supports numerous environments:
- **Browsers (IndexedDB)**: For web and PWA projects.
- **[Node.js](../nodejs-database.md)**: Ideal for server-side rendering or background services.
- **React Native**: Leverage SQLite or other adapters for mobile app development.
- [Electron](../electron-database.md): Create offline-capable desktop apps with a unified codebase.
This versatility empowers teams to reuse application logic across multiple platforms while maintaining a consistent data model.
### Performance Optimization
With **lazy loading** of data and the ability to utilize efficient storage engines, RxDB delivers high-speed operations and quick response times. By minimizing disk I/O and leveraging indexes effectively, RxDB ensures that even large-scale applications remain performant. Its reactive nature also helps avoid unnecessary re-renders, improving the end-user experience.
### Proven Reliability
RxDB is battle-tested in production environments, handling use cases from small single-user applications to large-scale enterprise solutions. Its robust replication mechanism resolves conflicts, manages concurrent writes, and ensures data integrity. The active open-source community provides ongoing support, documentation updates, and feature improvements.
### Developer-Friendly Features
For developers, RxDB offers:
- **Straightforward APIs**: Built on top of familiar JavaScript paradigms like promises and observables.
- **Excellent Documentation**: Detailed guides, tutorials, and references for every major feature.
- **Rich Community Support**: Benefit from an active ecosystem of contributors creating plugins, answering questions, and maintaining core libraries.
These qualities streamline development, making RxDB an appealing choice for teams of all sizes.
## FAQ
What is the main advantage of a local database?
A local database provides data storage on the user device. This eliminates the need for continuous internet access. You achieve near-instant data retrieval and fast updates. An offline-first application requires a local database to function without a network connection. You reduce server load and bandwidth costs. You provide a smooth user experience regardless of network conditions.
## Follow Up
Ready to get started? Here are some next steps:
- Try the [Quickstart Tutorial](../quickstart.md) and build a basic project to see RxDB in action.
- Compare RxDB with [other local database solutions](../alternatives.md) to determine the best fit for your unique requirements.
Ultimately, **RxDB** is more than just a database - it's a robust, reactive toolkit that empowers you to build fast, resilient, and user-centric applications. Whether you're creating an offline-first note-taking app or a real-time collaborative platform, RxDB can handle your local storage needs with ease and flexibility.
---
## Why Local-First Software Is the Future and its Limitations
import {Tabs} from '@site/src/components/tabs';
import {Steps} from '@site/src/components/steps';
import {QuoteBlock} from '@site/src/components/quoteblock';
import {VideoBox} from '@site/src/components/video-box';
# Why Local-First Software Is the Future and what are its Limitations
Imagine a web app that behaves seamlessly even with zero internet access, provides sub-millisecond response times, and keeps most of the user's data on their device. This is the **local-first** or [offline-first](../offline-first.md) approach. Although it has been around for a while, local-first has recently become more practical because of **maturing browser storage APIs** and new frameworks that simplify **data synchronization**. By allowing data to live on the client and only syncing with a server or other peers when needed, local-first apps can deliver a user experience that is **fast, resilient**, and **privacy-friendly**.
However, local-first is no silver bullet. It introduces tricky distributed-data challenges like conflict resolution and schema migrations on client devices. In this article, we'll dive deep into what local-first means, why it's trending, its pros and cons, and how to implement it in real applications. We'll also discuss other tools, criticisms, backend considerations, and how local-first compares to traditional cloud-centric approaches.
## What is the Local-First Paradigm
In **local-first** software, the primary copy of your data lives on the **client** rather than a remote server. Rather than sending each read or write over the network, you store and manipulate data in a [local database](./local-database.md) on the userβs device. Sync then happens in the background, ensuring all devices eventually converge to a consistent state.
This approach is increasingly popular because it leads to **instant** app responses (no network delay for most operations), genuine **offline capability**, and more direct **data ownership** for users. Local-first apps also sidestep outages and if the server or internet goes down, users can keep working with their local data. When connectivity returns, everything syncs. This makes the user experience **more resilient** and gives them control of their data, which is especially appealing when privacy concerns or limited connectivity are key factors.
Local-First software: A set of principles for software that enables both collaboration and ownership for users. Local-first ideals include the ability to work offline and collaborate across multiple devices, while also improving the security, privacy, long-term preservation, and user control of data.
## Why Local-First is Gaining Traction
The push for local-first is driven by a few key new technological capabilities that previously restricted client devices from running heavy local-first computing:
- **Relaxed Browser Storage Limits**: In the past, true local-first web apps were not very feasible due to **storage limitations** in browsers. Early web storage options like cookies or [localStorage](./localstorage.md#understanding-the-limitations-of-local-storage) had tiny limits (~5-10MB) and were unsuitable for complex data. Even **IndexedDB**, the structured client storage introduced over a decade ago, had restrictive quotas on many browsers: For example, older Firefox versions would **prompt the user if more than 50MB** was being stored. Mobile browsers often capped IndexedDB to 5MB without user permission. Such limits made it impractical to cache large application datasets on the client. However, modern browsers have dramatically [increased these limits](./indexeddb-max-storage-limit.md). Today, IndexedDB can typically store **hundreds of megabytes to multiple gigabytes** of data, depending on device capacity. Chrome allows up to ~80% of free disk space per origin (tens of GB on a desktop), Firefox now supports on the order of gigabytes per site (10% of disk size), and even Safari (historically strict) permits around 1GB per origin on iOS. In short, the storage quotas of 5-50MB are a thing of the past and modern web apps can cache very large datasets locally without hitting a ceiling. This shift in storage capabilities has unlocked new possibilities for **local-first web apps** that simply weren't viable a few years ago.
- **New Storage APIs (OPFS)**: The new Browser API [Origin Private File System](../rx-storage-opfs.md) (OPFS), part of the File System Access API, enables near-native file I/O from within a browser. It allows web apps to manage file handles securely and perform fast, synchronous reads/writes in Web Workers. This is a huge deal for local-first computing because it makes it feasible to embed robust database engines directly in the browser, persisting data to real files on a virtual filesystem. With OPFS, you can avoid some of the performance overhead that comes with [IndexedDB-based workarounds](../slow-indexeddb.md), providing a near-native [speed experience](./localstorage-indexeddb-cookies-opfs-sqlite-wasm.md#big-bulk-writes) for file-structured data.
- **Bandwidth Has Grown, But Latency Is Capped**: Internet infrastructure has rapidly expanded to provide higher throughput making it possible to transfer large amounts of data more quickly. However, latency (i.e., round-trip delay) is constrained by the **speed of light** and other physical limitations in fiber, satellite links, and routing. We can always build out bigger "pipes" to stream or send bulk data, but we can't significantly reduce the base round-trip time for each request. This is a physical limit, not a technological one. Local-first strategies mitigate this fundamental latency limit by avoiding excessive client-server calls in interactive workflows, once data is on the client, it's instantly available for reads and writes without waiting on a network round-trip. Imagine, transferring **around 100,000** "average" JSON documents might only consume **about the same bandwidth as two frames of a 4K YouTube video** which can be transferred in milliseconds. This shows just how far raw data throughput has come. Yet each request still has a 100-200ms latency or more, which becomes noticeable in user interactions. Local-first mitigates this by minimizing round-trip calls during active use and using the available bandwidth to directly transfer most of the data on the first app start.
- **WebAssembly**: Another advancement is **WebAssembly (WASM)**, which allows developers to compile low-level languages (C, C++, Rust) for execution in the browser at near-native speed. This means database engines, search algorithms, [vector databases](./javascript-vector-database.md), and other performance-heavy tasks can run right on the client. However, a key limitation is that **WASM cannot directly access persistent storage APIs** in the browser. Instead, all data must be sent from WASM to JavaScript (or the main thread) and then go through something like IndexedDB or OPFS. This extra indirection [is slower](./localstorage-indexeddb-cookies-opfs-sqlite-wasm.md) compared to plain JavaScript->storage calls. Looking ahead, there might come up future APIs that allow WASM to interface with persistent storage directly, and if those land, local-first systems could see another major boost in [performance](../rx-storage-performance.md).
- **Improvements in Local-First Tooling**: A major factor fueling the rise of local-first architectures is the **dramatic leap in client-side tooling and performance**. For instance, consider a local-first **email client** that stores **one million messages**. In 2014, searching through that many documents, especially with something like early PouchDB, could take **minutes** in a browser. Today, with advanced offline databases like **RxDB**, you can use the [OPFS storage](../rx-storage-opfs.md) with [sharding](../rx-storage-sharding.md) across multiple [web workers](../rx-storage-worker.md) (one per CPU) and use [memory-mapped](../rx-storage-memory-mapped.md) techniques. The result is a **regex search** of one million of these email documents in around **120 milliseconds** - all in JavaScript, running inside a standard web browser, on a mobile phone.
Better yet, this performance ceiling is likely to keep rising. Newer browser features and **WebAssembly** optimizations could enable even faster indexing and query operations, closing the gap with native desktop clients. I even experimented with GPU-accelarated queries (using **WebGPU**) which, while still in experimental stage, might deliver client-side performance that outperforms servers which do not have a graphics card.
These transformations highlight why local-first has become truly practical: not only can you sync and work offline, but you can handle **serious data loads** with performance that would have been unthinkable just a few years ago.
## What you can expect from a Local First App
[Jevons' Paradox](https://en.wikipedia.org/wiki/Jevons_paradox) says that making a _resource cheaper or more efficient to use often leads to greater overall consumption_. Originally about coal, it applies to the local-first paradigm in a way where we require apps to have more features, simply because it is technically possible, the app users and developers start to expect them:
### User Experience Benefits
- **Performance & UX:** Running from local storage means **low latency** and instantaneous interactions. There's no round-trip delay for most operations. Local-first apps aim to provide [near-zero latency](./zero-latency-local-first.md) responses by querying a local database instead of waiting for a server responseβ. This results in a snappy UX (often no need for loading spinners) because data reads/writes happen immediately on-device. Modern users expect real-time feedback, and local-first delivers that by default.
- **User Control & Privacy:** Storing data locally can limit how much sensitive information is sent off to remote servers. End users have greater control over their data, and the app can implement [client-side encryption](../encryption.md), thereby reducing the risk of mass data breaches. Its even possible to only replicated encrypted data with a server so that the backend does not know about the data at all and just acts as a backup/replication endpoint.
- **Offline Resilience:** Obviously, being able to work offline is a major benefit. Users can continue using the app with no internet (or flaky connectivity), and their changes sync up once online. This is increasingly important not just for remote areas, but for any app that needs to be available 24/7. Even though mobile networks have improved, connectivity can still drop; local-first ensures the app doesn't grind to a halt. The app _"stores data locally at the client so that it can still access it when the internet goes away."_
- **Realtime Apps**: Today's users expect data to stay in sync across browser tabs and devices without constant page reloads. In a typical cloud app, if you want real-time updates (say to show that a friend edited a document), you'd need to implement a [websocket or polling](./websockets-sse-polling-webrtc-webtransport.md) system for the server to push changes to clients, which is complex. Local-first architectures naturally lend themselves to realtime-by-default updates because the application state lives in a local database that can be observed for changes. Any edits (local or incoming from the server) immediately trigger [UI updates](./optimistic-ui.md). Similarly, background sync mechanisms ensure that new server-side data flows into the local store and into the user interface right away, no need to hit F5 to fetch the latest changes like on a traditional webpage.
### Developer Experience Benefits
- **Reduced Server Load**: Because local-first architectures typically **transfer large chunks of data once** (e.g., during an initial sync) and then sync only small diffs (delta changes) afterward, the server does not have to handle repeated requests for the same dataset. This bulk-first, diff-later approach drastically decreases the total number of round-trip requests to the backend. In scenarios where hundreds of simultaneous users each require continuous data access, an offline-ready client that only periodically sends or receives changes can scale more efficiently, freeing your servers to handle more users or other tasks. Instead of being bombarded with frequent small queries and updates, the server focuses on periodic sync operations, which can be more easily optimized or batched. It **Scales with Data, Not Load**. In fact for most type of apps, most of the data itself rarely changes. Imagine a CRM system. How often does the data of a customer really change compared to how often a user opens the customer-overview page which would load data from a server in traditional systems?
- **Less Need for Custom API Endpoints**: A local-first architecture often simplifies backend design. Instead of writing extensive REST routes for each client operation (create, read, update, delete, etc.), you can build a **single replication endpoint** or a small set of endpoints to handle data synchronization for each entity. The client manages local data, merges edits, and pushes/pulls changes with the server automatically. This not only **reduces boilerplate code** on the backend but also **frees developers** to focus on business logic and domain-specific concerns rather than spending time creating and maintaining dozens of narrowly scoped endpoints. As a result, the overall system can be easier to scale and maintain, delivering a **smoother developer experience**.
- **Simplified State Management in Frontend**: Because the local database holds the authoritative state, you might rely less on complex state management libraries (Redux, MobX, etc.). The DB becomes a single source of truth for your UI. In an offline-first app, your global state is already there in a single place stored inside of the local database, so you don't need as many in-memory state layers to synchronizeβ. The UI can directly bind to the database (using queries or reactive subscriptions). All sources of changes (user input, remote updates, other tabs) funnel through the database. This can significantly reduce the "glue code" to keep UI state in sync with server state because the local DB does that for you. With Local-First tools, "you might not need Redux" because the reactive DB fulfills that roleβ of state management already.
- **Observable Queries**: One of the big advantages of storing data locally is the ability to **subscribe** to data changes in real time, often called **observable queries**. When the data changes - either from the user's actions or from a remote sync - the local database automatically updates the subscribed query results, and the UI can redraw without a manual refresh. This reactive pattern can make apps feel much more live and responsive.
In the beginnings this was mostly done by watching a changes feed (like in PouchDB) and **re-running queries** whenever data changed. However, this early approach was **slow and didn't scale well**, because the entire query had to be recalculated each time. Later, RxDB introduced the [EventReduce Algorithm](https://github.com/pubkey/event-reduce), which merges incoming document changes into an existing query result by using a big [binary decision tree](https://github.com/pubkey/binary-decision-diagram). With this, updated query results can be "calculated" on the CPU instead of re-running them over the database. This makes query updates almost instantaneous and scales better as your data grows. Nowadays, many local databases have added similar features: for example, **dexie.js** introduced `liveQuery`, letting developers build real-time UIs without repeatedly scanning the entire dataset.
- **Better Multi-Tab and Multi-Device Consistency**: Because the source of truth is on the client, if the user has the app open in multiple tabs or even multiple devices, each has a full copy of the data. In browsers, many offline databases use a storage like IndexedDB that is shared across tabs of the same origin. This means all tabs see the same up-to-date local state. For example, if a user logs in or adds data in one tab, the other tab can automatically reflect that change via the shared local DB and eventsβ. This solves a common issue in web apps where one tab doesn't know that something changed in another tab. With local-first, **multi-tab just works** by default because there's "exactly one state of the data across all tabs". Similarly, on multiple devices, once sync runs, each device eventually converges to the same state.
> If your users have to press F5 all the time, your app is broken!
- **Potential for P2P and Decentralization**: While most current local-first apps still use a central backend for syncing, the paradigm opens the door to [peer-to-peer data syncing](../replication-webrtc.md). Because each device has the full data, devices could sync directly via LAN or other P2P channels for collaboration, reducing reliance on central servers. There are experimental frameworks that allow truly decentralized sync. This is a more advanced benefit, but it aligns with the ethos of giving users more control and reducing dependence on any one cloud provider.
These advantages show why developers are excited about local-first. You get happier users thanks to a fast, offline-capable app, and you can differentiate your product by working in scenarios where others fail (e.g. poor connectivity). Companies like Google, Amazon, etc., invest heavily in reducing latency and adding offline modes for a reason: it improves retention and usability. Local-first design takes that to the extreme by default.
## Challenges and Limitations of Local-First
However, this approach is not without significant challenges. It's important to understand the drawbacks and trade-offs before deciding to go all-in on local-first. Let's examine the flip side.
You fully understood a technology when you know when not to use it
Critics of local-first approaches often point out these challenges. Here's a comprehensive list of cons, criticisms, and obstacles associated with local-first development and proposed solutions on how to solve these obstacles:
- **Data Synchronization**: Data synchronization is arguably the hardest part of local-first development, because when every user's device can be offline for extended periods, data inevitably diverges. Ensuring those changes propagate and reconcile with minimal user headaches is a major challenge in distributed systems. Two main approaches have emerged:
- **Use a bundled frontend+backend solution** where the backend is tightly coupled to the client SDK and knows exactly how to handle sync. A common example is Firestore (part of the Firebase ecosystem) where Google's servers and client libraries collectively manage storage, change detection, conflict resolution, and syncing. The upside is you have a turnkey solution, developers can focus on features rather than writing sync logic. The downside is lock-in, because the sync protocol is proprietary and tailored to that vendor. In many organizations, this is a non-starter: existing company infrastructure can't be uprooted or replaced just for a single offline-capable app. This lock-in issue can arise even if the backend is not strictly a third-party vendor but simply another technology like PostgreSQL, because it still forces you to consolidate all your data into a single system that you might not use otherwise.
- **Custom Replication with Your Own Endpoints**: Alternatively, tools like RxDB allow you to implement your own replication endpoints on top of your existing infrastructure. This is the approach relies on a lightweight, git-like [Sync Engine](../replication.md). The server remains relatively "dumb," focusing on storing revisions, tracking changes, and marking conflicts, while the client library does the actual [conflict resolution](../transactions-conflicts-revisions.md). During sync, if the server detects a conflict (e.g., two offline edits to the same document), it notifies the client, which then decides how to merge them, whether via last-write-wins, a custom merge function, or a [CRDT](../crdt.md). Setting up custom endpoints does require more development effort, but you avoid vendor lock-in and can integrate seamlessly with your existing database(s). Your system simply needs to support incrementally fetching changes (pull) and accepting local modifications (push), which can be layered on top of nearly any data store or architecture.
Tools which support "any backend" are of course harder to monetize because they cannot sell SaaS services or a Cloud Subscription which is why most tools use a fixed backend instead of an open Sync Engine.
- **Conflict Resolution**: When multiple offline edits happen on the same data, you inevitably get **merge conflicts**. For example, if two users (or the same user on two devices) both edit the same document offline, when both sync, whose changes win? Local-first systems need a conflict resolution strategy. Some systems use **last-write-wins** (firestore) or deterministic revision hashing to pick a "winner" (as in CouchDB/PouchDB)β. This is simple but may drop one user's changes. Other approaches keep both versions and merge them either via an implement ["merge-function"](../transactions-conflicts-revisions.md#custom-conflict-handler) or require a **manual merge** step (e.g., like git conflicts or showing the user a diff UI). More advanced solutions involve **CRDTs (Conflict-free Replicated Data Types)** which mathematically merge changes (used for rich text collaboration, for instance). Libraries like Automerge or Yjs implement CRDTs to "magically solve conflicts". But in practice, using CRDTs is also complex and has its own trade-offs and sometimes not even possible like when you need additional data from another instance for a "correct" merge. No matter which route, handling conflicts adds complexity to your app logic or infrastructure. In cloud-based (online-first) apps, you avoid this because everyone is always editing the single up-to-date copy on the server. Local-first shifts that burden to the client side.
Here is an example on how a client-side merge functions works in RxDB:
```ts
import { deepEqual } from 'rxdb/plugins/utils';
export const myConflictHandler = {
/**
* isEqual() is used to detect if two documents are
* equal. This is used internally to detect conflicts.
*/
isEqual(a, b) {
/**
* isEqual() is used to detect conflicts or to detect if a
* document has to be pushed to the remote.
* If the documents are deep equal,
* we have no conflict.
* Because deepEqual is CPU expensive,
* on your custom conflict handler you might only
* check some properties, like the updatedAt time or revision-strings
* for better performance.
*/
return deepEqual(a, b);
},
/**
* resolve() a conflict. This can be async so
* you could even show an UI element to let your user
* resolve the conflict manually.
*/
async resolve(i) {
/**
* In this example we drop the local state and use the server-state.
* This basically implements a "first-on-server-wins" strategy.
*
* In your custom conflict handler you could want to merge properties
* of the i.realMasterState, i.assumedMasterState and i.newDocumentState
* or return i.newDocumentState to have a "last-write-wins" strategy.
*/
return i.realMasterState;
}
};
```
- **Eventual Consistency (No Single Source of Truth):** A local-first system is **eventually consistent** by nature. There is no single authoritative copy of the data at all times. Instead you have one per device (and maybe one on the server), and they sync to become consistent eventually. This means at any given moment, two users might not see the same data if one hasn't synced recently. Users could even make decisions based on stale data. In many applications this is acceptable (the data will catch up), but for some scenarios it's problematic. For instance, an offline-first banking app that lets you initiate a money transfer offline could be dangerous if the account balance was out-of-date or if the transfer needs immediate consistency. Essentially, **not all apps can tolerate eventual consistency**. If your use case demands strong consistency (e.g., inventory systems where overselling is a big issue, or real-time collaborative editing where every keystroke must be seen by others instantly), a purely local-first approach might need augmentation or may not fit.
- **Initial Data Load and Data Size Limits:** Local-first requires pulling data **down to the client**. If your dataset is huge (gigabytes), it's simply not feasible to download everything to every client. For example, syncing every tweet on Twitter to every user's phone is impossible. Local-first works best when the data set per user is reasonably sized (up to 2 Gigabytes). In practice, you often **limit the data** to just that user's own data or a subset relevant to them. Even then, on first use the app might need to download a significant chunk of data to initialize the local database. There is a **upper bound on dataset size** beyond which the initial sync or storage needs become impractical. You cannot assume unlimited local storage. If your data is too large, local-first will either fail or you'll need to only sync partial data (and then handle what happens if the needed data isn't present locally). In short, **local-first is unsuitable for massive datasets** or data that cannot be partitioned per user.
- **Storage Persistence (Browser Limitations):** Storing data in the browser (via IndexedDB or similar) is not as durable as on a server disk. Browsers may **evict data** to save space (especially on mobile devices). For instance, Safari notoriously wipes out IndexedDB data if the user hasn't used the site in ~7 days. Other browsers have their own eviction policies for offline data. Also, users can at any time clear their browser storage (intentionally or via something like "Clear site data"). This means the local data **cannot be 100% trusted to stay forever**. A well-behaved local-first app needs to be able to recover if local data is lost, usually by pulling from the server againβ. Essentially, the server still often serves as a backup. But if your app had any purely local data (not intended to sync), that's at risk. **Mobile apps** (with SQLite or filesystem storage) are a bit more stable than web browsers, but even there, uninstalls or certain OS actions can remove local data. This is a challenge: How to cache data offline for speed while ensuring if it's wiped, the user doesn't lose everything important. Cloud-only apps by contrast keep data in the cloud so it's typically safe unless the server fails (and servers are easier to backup reliably).
- **Complex Client-Side Logic & Increased App Size**: A local-first app tends to be more complex on the client side. You're essentially putting what used to be server responsibilities (storage, query engine, sync logic) into the frontend. This can increase the size of your frontend bundle (including a database library, possibly CRDT or sync code, etc.). It also increases memory and CPU usage on the client, as the browser/phone is doing more work. Low-end devices or older phones might struggle if the app is not optimized. Developers need to consider performance tuning for the local database (indexing, query efficiency) just like they would on a server. So while the user gains benefits, the app developer has to manage this complexity.
- **Performance Constraints in JavaScript:** Even though devices are fast, a local JS database is generally **slower than a server DB** on robust hardware. There are many layers (JS -> IndexedDB -> possibly SQLite under the hood) that add overheadβ. For example, inserting a record might go through the DB library, the storage engine, the browser's implementation, down to disk. For most UI uses this is fine (you don't need 10k writes/sec in a to-do app, you need maybe a few writes per second at most). But if your app does heavy data processing, the browser might become a bottleneck.
The key question is _"Is it fast enough?"_. Often the answer is yes for typical usage, but developers must be mindful of not doing something on the client that truly requires big iron servers. For instance, full-text indexing of a million documents might be too slow in a client-side DB. **Unpredictable performance** is also a factor: Different users have different devices. A query that takes 50ms on a high-end desktop might take 500ms on a low-end phone in battery saving mode. So performance tuning and testing across devices is needed, and some heavy tasks might still belong on the server sideβ. For example if you build a [local vector database](./javascript-vector-database.md) you might want to create the embeddings on the server and sync them instead of creating them on the client.
- **Client Database Migrations:** As your app evolves, you'll change data models or add new fields. In a cloud-first app, you'd typically run a migration on the server database. In a local-first app, you have not only the server DB (if any) but also every client's local database to consider. Upgrading the schema means you need to write migration logic that runs on each client, perhaps the next time they launch the app after an update. Clients may be offline or not upgrade the app immediately, so you could have different versions of the schema in the wild. This complicates data handling (the sync protocol might need to handle multiple schema versions until everyone is updated). Providing a smooth migration path for local data is doable (many libraries provide [migration facilities](../migration-schema.md)), but it requires careful testing. In a worst case, a failed migration on a client could brick the app for that user or force a full resync. This is a **much bigger headache** than just migrating a centralized DB at midnight while your service is in maintenance mode. π
- **Security and Access Control:** In cloud-based apps, enforcing data security (who can see what) is done on the server. The client only gets the data it's authorized to get. In a local-first scenario, you often need to **partition data per user** on the backend as well, to ensure users only sync down their own data (or data they have permission for). One simple strategy is to give each user their own database or dataset on the server and only replicate that. For example, CouchDB allows creating one database per user and replication can be scoped to that DB which makes permission handling easy. But if you ever need to query across users (say an admin view or aggregate analytics), having data split into many small DBs becomes a pain. The alternative is a single backend database with a **fine-grained access control**, and the client asks to sync only certain documents/fields. That usually means writing a custom sync server or using something like GraphQL with resolvers that respect permissions. In short, **implementing auth and permissions in sync** adds complexity. Also, any data stored on the client is theoretically vulnerable to extraction (if someone compromises the device or uses dev tools). You can [encrypt local databases](../encryption.md) to prevent extraction after the server "revokes" the decryption password to mitigate the data extraction risk.
- **Relational Data and Complex Queries:** Most client-side/offline databases are [NoSQL/document oriented](../why-nosql.md) for flexibility in syncing and easy conflict handling. They may not support complex join queries or ACID transactions across multiple tables like a full SQL database would. This is partly because replicating a full relational model is much harder (maintaining referential integrity, etc., when data is partial on a client) or not even logically possible. For example if you have two offline clients running a complex `UPDATE X WHERE Y FROM Z INNER JOIN Alice INNER JOIN Bob` query and then they go online, you have no easy way of handling these conflicts.
If your app has heavy relational data requirements or relies on complex server-side queries (aggregations, multi-join reports), you might find the local database either cannot do it or is too slow to do it client-side. The lack of robust relational querying is something to plan for and you might need to adjust your data model to be more document-oriented or use client-side libraries to run [joins in memory](../why-nosql.md#relational-queries-in-nosql). Most tools use NoSQL because it makes replication easy and implementing true relational sync would require extremely sophisticated solutions and even needing an atomic clock for full consistency across nodes (like google spanner). So, **if your app truly needs SQL power on the client**, local-first might complicate things.
> In Local-First, most tools use NoSQL because it makes replication and conflict handling easy.
That's a long list of challenges! In summary, local-first approaches introduce distributed data issues on the client side that web developers usually didn't have to deal with. Despite these challenges, the local-first movement is steadily growing because the **benefits to user experience and data control are very compelling** and modern tools are emerging to mitigate a lot of these difficulties. All of these are solved or solvable for your specific use-case, just keep them in mind before you start architecting your local-first app.
## Local-First vs. Traditional Online-First Approaches
So now that you know the pros and cons about Local-First. Lets directly compare it to your previous "online-first" stack:
### Connectivity and Offline Usage
- **Local-first**: Apps like WhatsApp store messages locally on your device, letting you read past messages and even compose new ones without a stable network connection. Once online, the messages sync with the server and other participants.
- **Online-first**: Purely cloud-based apps typically stop functioning (beyond simple caching) when the network or server goes down. They rely on a constant connection to fetch or store user data.
### Latency and Performance
- **Local-first**: Because data operations happen on the device, you see near-instant interactions (e.g., typing and reading chats feels very responsive, as the messages are on your phone). Syncing occurs in the background without delaying the user.
- **Online-first**: Most interactions involve a round-trip to the server, adding network latency and potentially requiring loading states or fallback UIs. If the network is slow or unreliable, users can experience delays when sending or receiving updates.
### Complexity and Conflict Resolution
- **Local-first**: Much of the complexity like storing data or handling conflicts, shifts to the client. WhatsApp, for instance, caches all messages locally and queues unsent messages offline. Once reconnected, it must reconcile with the server and other devices, especially if the same account is used on multiple platforms.
- **Online-first**: A single central server manages data and concurrency, so clients remain thinner. However, the app becomes unusable if connectivity is lost, and any server downtime directly impacts all users.
### Data Ownership and Storage Limits
- **Local-first**: Users hold a complete copy of data on their own device, retaining control and quick access. This can raise challenges when data sets are very large; phone storage might be insufficient, or backups may be harder to guarantee.
- **Online-first**: Storing all data in the cloud scales more easily, and providers often manage backups automatically. On the downside, users depend on the service and must trust it to protect their data.
### When to Choose Which
- **Local-first**: Particularly helpful for scenarios where offline operation and immediate responsiveness matter, such as chat apps (like WhatsApp), field tools in low-connectivity environments, or any use case where users can't rely on being online 24/7.
- **Online-first**: Well-suited for systems that demand real-time central control or massive data aggregation with minimal offline needs, such as large-scale analytics platforms or services where guaranteed immediate global consistency is essential.
- **Hybrid**: In reality, most modern apps often blend these approaches, giving users partial offline capabilities (caching, queued updates) alongside a robust central service. This hybrid method offers the benefits of local speed and resilience, while still leveraging cloud infrastructure for collaboration and global reach. But a truth local-first app is way more than just a cache.
---
## Offline-First vs. Local-First
In the early days of offline-capable web apps (around 2014), the common phrase was **"Offline-First"**. Tools like **PouchDB** popularized the notion that developers should assume devices are often offline or have flaky connections, so apps must continue to work seamlessly without a network. The guiding principle was *"apps should treat being online as optional."* If a user has no internet access, the application's core features still function, saving or queuing data locally, and automatically synchronizing once connectivity is restored.
Over time, this focus on offline support evolved into the broader concept of **"Local-First Software,"** (see [Ink & Switch](https://martin.kleppmann.com/papers/local-first.pdf)) emphasizing not just offline operation but also the technical underpinnings of **storing data locally** in the client application. While offline-first is primarily about resilience to network loss, local-first highlights ownership, privacy, and performance benefits of keeping the primary data on the user's device. Most tools these days extended the original offline-first concepts, adding real-time reactivity, custom sync, and more nuances like conflict resolution or encryption.
However, the term **"local-first"** can be **confusing** to non-technical audiences because many people (especially in the US) associate "local first" with *community-oriented movements* that encourage buying from nearby businesses or supporting local initiatives. To reduce ambiguity, it may be clearer to use **"local first software"** or **"local first development"** in your documentation and marketing materials. When creating branding or logos around local-first software, **avoid using the "Google Maps Pin"** as a symbol. This icon typically implies geolocation or physical locality further mixing up the notion of "location-based services" with "on-device data storage."
## Do People Actually Use Local-First or Is It Just a Trend?
If we look at **npm download statistics**, we see that **PouchDB** - one of the oldest libraries for local-first apps - has about **53k** downloads each week, and **RxDB** - a newer library - has about **22k** weekly downloads. Other local-first tools often have even fewer downloads. In comparison, a popular library like **react-query**, which does not focus on local storage, is downloaded about **1.6 million** times a week. These numbers show that local-first libraries, while used, are not as common as some of the more traditional tools.
One reason is that **local-first** is still a new idea. Many developers are used to traditional "online-first" approaches, so switching to a local database and then syncing changes later can feel unfamiliar. Developers must learn different patterns, deal with offline synchronization, and handle possible conflicts. That extra work can be a barrier to adoption which might change in the future as tooling improves.
While most of RxDB is open source, there are also **premium plugins** that help sustain RxDB as a long-term project. Because people purchase these plugins, we gains insights into how developers are using local-first features:
- About **half** of these users mainly want **offline functionality** for cases such as farming equipment, mining, construction, or even a shrimp farm app.
- The **other half** focus on **faster, real-time UIs** for to-do or reading apps, a space launch planning tool, and various dashboard apps. This range of use cases highlights both the resilience offline mode can offer and the performance boost that local databases can provide when synced in the background.
## Why Local-First Is the Future
Early in the history of the web, users **expected** static pages. If you wanted to see new content, you **reloaded** the page. That was normal at the time, and nobody found it strange because everything worked that way.
Then, as more sites added **real-time** features - auto-updating feeds, live notifications, single-page apps - suddenly those older "reload-only" sites began to feel **slow** or **outdated**. Why wait for a manual refresh when real-time data was possible and readily available?
The same pattern is happening with **local-first** apps. Right now, most sites are still built around network availability. We see loading spinners whenever data is fetched, and we simply wait for the server response. As local-first experiences become **commonplace** - removing spinners, letting users keep working when offline, and syncing in the background - everything else will start to feel **frustratingly behind**. Users won't tolerate slow or blocked interactions if they've seen apps that respond instantly and remain usable offline. They'll expect that as the default and we'll likely see growing pressure on developers to eliminate those extra loading steps. For many users, the experience of **immediate local writes will become not just a perk, but an expectation!**
## FAQ
What are the differences between an offline-first database and a cloud-first database for remote areas?
An offline-first database stores data locally on the user device. A cloud-first database stores data on a remote server. Field workers in remote areas often face poor network connectivity. An offline-first database allows users to read and write data without an internet connection. A cloud-first database requires continuous internet access to function. An offline-first approach synchronizes data automatically when a connection becomes available. A cloud-first approach blocks users from working during offline periods. You choose an offline-first database to ensure continuous productivity in remote locations.
## See also
- Discuss [this topic on HackerNews](https://news.ycombinator.com/item?id=43289885)
- [Local-First Technologies](../alternatives.md): A list of databases and technologies (besides [RxDB](/)) that support offline-first or local-first use cases.
- [Discord](/chat/): Join our Discord server to talk with people and share ideas about this topic.
- [Ink & Switch](https://martin.kleppmann.com/papers/local-first.pdf): The "original" paper about Local-First from 2019 where the naming of local-first Software was first used and described.
- [Learn how to build a local-first Application with RxDB](../quickstart.md).
---
## LocalStorage vs. IndexedDB vs. Cookies vs. OPFS vs. WASM-SQLite
# LocalStorage vs. IndexedDB vs. Cookies vs. OPFS vs. WASM-SQLite
So you are building that web application and you want to **store data inside of your users browser**. Maybe you just need to store some small flags or you even need a fully fledged database.
The types of web applications we build have changed significantly. In the early years of the web we served static html files. Then we served dynamically rendered html and later we build **single page applications** that run most logic on the client. And for the coming years you might want to build so called [local first apps](../offline-first.md) that handle big and complex data operations solely on the client and even work when offline, which gives you the opportunity to build **zero-latency** user interactions.
In the early days of the web, **cookies** were the only option for storing small key-value assignments.. But JavaScript and browsers have evolved significantly and better storage APIs have been added which pave the way for bigger and more complex data operations.
In this article, we will dive into the various technologies available for storing and querying data in a browser. We'll explore traditional methods like **Cookies**, **localStorage**, **WebSQL**, **IndexedDB** and newer solutions such as **OPFS** and **SQLite via WebAssembly**. We compare the features and limitations and through performance tests we aim to uncover how fast we can write and read data in a web application with the various methods.
:::note
You are reading this in the [RxDB](/) docs. RxDB is a JavaScript database that has different storage adapters which can utilize the different storage APIs.
**Since 2017** I spend most of my time working with these APIs, doing performance tests and building [hacks](../slow-indexeddb.md) and plugins to reach the limits of browser database operation speed.
:::
## The available Storage APIs in a modern Browser
First lets have a brief overview of the different APIs, their intentional use case and history:
### What are Cookies
Cookies were first introduced by [netscape in 1994](https://www.baekdal.com/thoughts/the-original-cookie-specification-from-1997-was-gdpr-compliant/).
Cookies store small pieces of key-value data that are mainly used for session management, personalization, and tracking. Cookies can have several security settings like a time-to-live or the `domain` attribute to share the cookies between several subdomains.
Cookies values are not only stored at the client but also sent with **every http request** to the server. This means we cannot store much data in a cookie but it is still interesting how good cookie access performance compared to the other methods. Especially because cookies are such an important base feature of the web, many performance optimizations have been done and even these days there is still progress being made like the [Shared Memory Versioning](https://blog.chromium.org/2024/06/introducing-shared-memory-versioning-to.html) by chromium or the asynchronous [CookieStore API](https://developer.mozilla.org/en-US/docs/Web/API/Cookie_Store_API).
### What is LocalStorage
The [localStorage API](./localstorage.md) was first proposed as part of the [WebStorage specification in 2009](https://www.w3.org/TR/2009/WD-webstorage-20090423/#the-localstorage-attribute).
LocalStorage provides a simple API to store key-value pairs inside of a web browser. It has the methods `setItem`, `getItem`, `removeItem` and `clear` which is all you need from a key-value store. LocalStorage is only suitable for storing small amounts of data that need to persist across sessions and it is [limited by a 5MB storage cap](./localstorage.md#understanding-the-limitations-of-local-storage). Storing complex data is only possible by transforming it into a string for example with `JSON.stringify()`.
The API is not asynchronous which means if fully blocks your JavaScript process while doing stuff. Therefore running heavy operations on it might block your UI from rendering.
> There is also the **SessionStorage** API. The key difference is that localStorage data persists indefinitely until explicitly cleared, while sessionStorage data is cleared when the browser tab or window is closed.
### What is IndexedDB
IndexedDB was first introduced as "Indexed Database API" [in 2015](https://www.w3.org/TR/IndexedDB/#sotd).
[IndexedDB](../rx-storage-indexeddb.md) is a low-level API for storing large amounts of structured JSON data. While the API is a bit hard to use, IndexedDB can utilize indexes and asynchronous operations. It lacks support for complex queries and only allows to iterate over the indexes which makes it more like a base layer for other libraries then a fully fledged database.
In 2018, IndexedDB version 2.0 [was introduced](https://hacks.mozilla.org/2016/10/whats-new-in-indexeddb-2-0/). This added some major improvements. Most noticeable the `getAll()` method which improves performance dramatically when fetching bulks of JSON documents.
IndexedDB [version 3.0](https://w3c.github.io/IndexedDB/) is in the workings which contains many improvements. Most important the addition of `Promise` based calls that makes modern JS features like `async/await` more useful.
### What is OPFS
The [Origin Private File System](../rx-storage-opfs.md) (OPFS) is a [relatively new](https://caniuse.com/mdn-api_filesystemfilehandle_createsyncaccesshandle) API that allows web applications to store large files directly in the browser. It is designed for data-intensive applications that want to write and read **binary data** in a simulated file system.
OPFS can be used in two modes:
- Either asynchronous on the [main thread](../rx-storage-opfs.md#using-opfs-in-the-main-thread-instead-of-a-worker)
- Or in a WebWorker with the faster, asynchronous access with the `createSyncAccessHandle()` method.
Because only binary data can be processed, OPFS is made to be a base filesystem for library developers. You will unlikely directly want to use the OPFS in your code when you build a "normal" application because it is too complex. That would only make sense for storing plain files like images, not to store and query [JSON data](./json-based-database.md) efficiently. I have build a [OPFS based storage](../rx-storage-opfs.md) for RxDB with proper indexing and querying and it took me several months.
### What is WASM SQLite
[WebAssembly](https://webassembly.org/) (Wasm) is a binary format that allows high-performance code execution on the web.
Wasm was added to major browsers over the course of 2017 which opened a wide range of opportunities on what to run inside of a browser. You can compile native libraries to WebAssembly and just run them on the client with just a few adjustments. WASM code can be shipped to browser apps and generally runs much faster compared to JavaScript, but still about [10% slower then native](https://www.usenix.org/conference/atc19/presentation/jangda).
Many people started to use compiled SQLite as a database inside of the browser which is why it makes sense to also compare this setup to the native APIs.
The compiled byte code of SQLite has a size of [about 938.9Β kB](https://sqlite.org/download.html) which must be downloaded and parsed by the users on the first page load. WASM cannot directly access any persistent storage API in the browser. Instead it requires data to flow from WASM to the main-thread and then can be put into one of the browser APIs. This is done with so called [VFS (virtual file system) adapters](https://www.sqlite.org/vfs.html) that handle data access from SQLite to anything else.
### What was WebSQL
WebSQL **was** a web API [introduced in 2009](https://www.w3.org/TR/webdatabase/) that allowed browsers to use SQL databases for client-side storage, based on SQLite. The idea was to give developers a way to store and query data using SQL on the client side, similar to server-side databases.
WebSQL has been **removed from browsers** in the current years for multiple good reasons:
- WebSQL was not standardized and having an API based on a single specific implementation in form of the SQLite source code is hard to ever make it to a standard.
- WebSQL required browsers to use a [specific version](https://developer.chrome.com/blog/deprecating-web-sql#reasons_for_deprecating_web_sql) of SQLite (version 3.6.19) which means whenever there would be any update or bugfix to SQLite, it would not be possible to add that to WebSQL without possible breaking the web.
- Major browsers like firefox never supported WebSQL.
Therefore in the following we will **just ignore WebSQL** even if it would be possible to run tests on in by setting specific browser flags or using old versions of chromium.
-------------
## Feature Comparison
Now that you know the basic concepts of the APIs, lets compare some specific features that have shown to be important for people using RxDB and browser based storages in general.
### Storing complex JSON Documents
When you store data in a web application, most often you want to store complex JSON documents and not only "normal" values like the `integers` and `strings` you store in a server side database.
- Only IndexedDB works with JSON objects natively.
- With SQLite WASM you can [store JSON](https://www.sqlite.org/json1.html) in a `text` column since version 3.38.0 (2022-02-22) and even run deep queries on it and use single attributes as indexes.
Every of the other APIs can only store strings or binary data. Of course you can transform any JSON object to a string with `JSON.stringify()` but not having the JSON support in the API can make things complex when running queries and running `JSON.stringify()` many times can cause performance problems.
### Multi-Tab Support
A big difference when building a Web App compared to [Electron](../electron-database.md) or [React-Native](../react-native-database.md), is that the user will open and close the app in **multiple browser tabs at the same time**. Therefore you have not only one JavaScript process running, but many of them can exist and might have to share state changes between each other to not show **outdated data** to the user.
> If your users' muscle memory puts the left hand on the **F5** key while using your website, you did something wrong!
Not all storage APIs support a way to automatically share write events between tabs.
Only localstorage has a way to automatically share write events between tabs by the API itself with the [storage-event](./localstorage.md#localstorage-vs-indexeddb) which can be used to observe changes.
```js
// localStorage can observe changes with the storage event.
// This feature is missing in IndexedDB and others
addEventListener("storage", (event) => {});
```
There was the [experimental IndexedDB observers API](https://stackoverflow.com/a/33270440) for chrome, but the proposal repository has been archived.
To workaround this problem, there are two solutions:
- The first option is to use the [BroadcastChannel API](https://github.com/pubkey/broadcast-channel) which can send messages across browser tabs. So whenever you do a write to the storage, you also send a notification to other tabs to inform them about these changes. This is the most common workaround which is also used by RxDB. Notice that there is also the [WebLocks API](https://developer.mozilla.org/en-US/docs/Web/API/Web_Locks_API) which can be used to have mutexes across browser tabs.
- The other solution is to use the [SharedWorker](https://developer.mozilla.org/en-US/docs/Web/API/SharedWorker) and do all writes inside of the worker. All browser tabs can then subscribe to messages from that **single** SharedWorker and know about changes.
### Indexing Support
The big difference between a database and storing data in a plain file, is that a database is writing data in a format that allows running operations over indexes to facilitate fast performant queries. From our list of technologies only **IndexedDB** and **WASM SQLite** support for indexing out of the box. In theory you can build indexes on top of any storage like localstorage or OPFS but you likely should not want to do that by yourself.
In IndexedDB for example, we can fetch a bulk of documents by a given index range:
```ts
// find all products with a price between 10 and 50
const keyRange = IDBKeyRange.bound(10, 50);
const transaction = db.transaction('products', 'readonly');
const objectStore = transaction.objectStore('products');
const index = objectStore.index('priceIndex');
const request = index.getAll(keyRange);
const result = await new Promise((res, rej) => {
request.onsuccess = (event) => res(event.target.result);
request.onerror = (event) => rej(event);
});
```
Notice that IndexedDB has the limitation of [not having indexes on boolean values](https://github.com/w3c/IndexedDB/issues/76). You can only index strings and numbers. To workaround that you have to transform boolean to numbers and backwards when storing the data.
### WebWorker Support
When running heavy data operations, you might want to move the processing away from the JavaScript main thread. This ensures that our app keeps being responsive and fast while the processing can run in parallel in the background. In a browser you can either use the [WebWorker](https://developer.mozilla.org/en-US/docs/Web/API/Web_Workers_API), [SharedWorker](https://developer.mozilla.org/en-US/docs/Web/API/SharedWorker) or the [ServiceWorker](https://developer.mozilla.org/en-US/docs/Web/API/Service_Worker_API) API to do that. In RxDB you can use the [WebWorker](../rx-storage-worker.md) or [SharedWorker](../rx-storage-shared-worker.md) plugins to move your storage inside of a worker.
The most common API for that use case is spawning a **WebWorker** and doing most work on that second JavaScript process. The worker is spawned from a separate JavaScript file (or base64 string) and communicates with the main thread by sending data with `postMessage()`.
Unfortunately **LocalStorage** and **Cookies** [cannot be used in WebWorker or SharedWorker](https://stackoverflow.com/questions/6179159/accessing-localstorage-from-a-webworker) because of the design and security constraints. WebWorkers run in a separate global context from the main browser thread and therefore cannot do stuff that might impact the main thread. They have no direct access to certain web APIs, like the DOM, localStorage, or cookies.
Everything else can be used from inside a WebWorker.
The fast version of OPFS with the `createSyncAccessHandle` method can **only** [be used in a WebWorker](../rx-storage-opfs.md#opfs-limitations), and **not on the main thread**. This is because all the operations of the returned `AccessHandle` are **not async** and therefore block the JavaScript process, so you do want to do that on the main thread and block everything.
-------------
## Storage Size Limits
- **Cookies** are limited to about `4 KB` of data in [RFC-6265](https://datatracker.ietf.org/doc/html/rfc6265#section-6.1). Because the stored cookies are send to the server with every HTTP request, this limitation is reasonable. You can test your browsers cookie limits [here](http://www.ruslog.com/tools/cookies.html). Notice that you should never fill up the full `4 KB` of your cookies because your web server will not accept too long headers and reject the requests with `HTTP ERROR 431 - Request header fields too large`. Once you have reached that point you can not even serve updated JavaScript to your user to clean up the cookies and you will have locked out that user until the cookies get cleaned up manually.
- **LocalStorage** has a storage size limitation that varies depending on the browser, but generally ranges from 4 MB to 10 MB per origin. You can test your localStorage size limit [here](https://arty.name/localstorage.html).
- Chrome/Chromium/Edge: 5 MB per domain
- Firefox: 10 MB per domain
- Safari: 4-5 MB per domain (varies slightly between versions)
- **IndexedDB** does not have a specific fixed size limitation like localStorage. The maximum storage size for IndexedDB depends on the browser implementation. The upper limit is typically based on the available disc space on the user's device. In chromium browsers it can use up to 80% of total disk space. You can get an estimation about the storage size limit by calling `await navigator.storage.estimate()`. Typically you can store gigabytes of data which can be tried out [here](https://demo.agektmr.com/storage/). Notice that we have a full article about [storage max size limits of IndexedDB](./indexeddb-max-storage-limit.md) that covers this topic.
- **OPFS** has the same storage size limitation as IndexedDB. Its limit depends on the available disc space. This can also be tested [here](https://demo.agektmr.com/storage/).
-------------
## Performance Comparison
Now that we've reviewed the features of each storage method, let's dive into performance comparisons, focusing on initialization times, read/write latencies, and bulk operations.
Notice that we only run simple tests and for your specific use case in your application the results might differ. Also we only compare performance in google chrome (version 128.0.6613.137). Firefox and Safari have similar **but not equal** performance patterns. You can run the test by yourself on your own machine from this [github repository](https://github.com/pubkey/localstorage-indexeddb-cookies-opfs-sqlite-wasm). For all tests we throttle the network to behave like the average german internet speed. (download: 135,900 kbit/s, upload: 28,400 kbit/s, latency: 125ms). Also all tests store an "average" JSON object that might be required to be stringified depending on the storage. We also only test the performance of storing documents by id because some of the technologies (cookies, OPFS and localstorage) do not support indexed range operations so it makes no sense to compare the performance of these.
### Initialization Time
Before you can store any data, many APIs require a setup process like creating databases, spawning WebAssembly processes or downloading additional stuff. To ensure your app starts fast, the initialization time is important.
The APIs of localStorage and Cookies do not have any setup process and can be directly used. IndexedDB requires to open a database and a store inside of it. WASM SQLite needs to download a WASM file and process it. OPFS needs to download and start a worker file and initialize the virtual file system directory.
Here are the time measurements from how long it takes until the first bit of data can be stored:
| Technology | Time in Milliseconds |
| ----------------------- | -------------------- |
| IndexedDB | 46 |
| OPFS Main Thread | 23 |
| OPFS WebWorker | 26.8 |
| WASM SQLite (memory) | 504 |
| WASM SQLite (IndexedDB) | 535 |
Here we can notice a few things:
- Opening a new IndexedDB database with a single store takes surprisingly long
- The latency overhead of sending data from the main thread to a WebWorker OPFS is about 4 milliseconds. Here we only send minimal data to init the OPFS file handler. It will be interesting if that latency increases when more data is processed.
- Downloading and parsing WASM SQLite and creating a single table takes about half a second. Using also the IndexedDB VFS to store data persistently adds additional 31 milliseconds. Reloading the page with enabled caching and already prepared tables is a bit faster with 420 milliseconds (memory).
### Latency of small Writes
Next lets test the latency of small writes. This is important when you do many small data changes that happen independent from each other. Like when you stream data from a websocket or persist pseudo randomly happening events like mouse movements.
| Technology | Time in Milliseconds |
| ----------------------- | -------------------- |
| Cookies | 0.058 |
| LocalStorage | 0.017 |
| IndexedDB | 0.17 |
| OPFS Main Thread | 1.46 |
| OPFS WebWorker | 1.54 |
| WASM SQLite (memory) | 0.17 |
| WASM SQLite (IndexedDB) | 3.17 |
Here we can notice a few things:
- LocalStorage has the lowest write latency with only 0.017 milliseconds per write.
- IndexedDB writes are about 10 times slower compared to localStorage.
- Sending the data to the WASM SQLite process and letting it persist via IndexedDB is slow with over 3 milliseconds per write.
The OPFS operations take about 1.5 milliseconds to write the JSON data into one document per file. We can see the sending the data to a webworker first is a bit slower which comes from the overhead of serializing and deserializing the data on both sides.
If we would not create on OPFS file per document but instead append everything to a single file, the performance pattern changes significantly. Then the faster file handle from the `createSyncAccessHandle()` only takes about 1 millisecond per write. But this would require to somehow remember at which position the each document is stored. Therefore in our tests we will continue using one file per document.
### Latency of small Reads
Now that we have stored some documents, lets measure how long it takes to read single documents by their `id`.
| Technology | Time in Milliseconds |
| ----------------------- | -------------------- |
| Cookies | 0.132 |
| LocalStorage | 0.0052 |
| IndexedDB | 0.1 |
| OPFS Main Thread | 1.28 |
| OPFS WebWorker | 1.41 |
| WASM SQLite (memory) | 0.45 |
| WASM SQLite (IndexedDB) | 2.93 |
Here we can notice a few things:
- LocalStorage reads are **really really fast** with only 0.0052 milliseconds per read.
- The other technologies perform reads in a similar speed to their write latency.
### Big Bulk Writes
As next step, lets do some big bulk operations with 200 documents at once.
| Technology | Time in Milliseconds |
| ----------------------- | -------------------- |
| Cookies | 20.6 |
| LocalStorage | 5.79 |
| IndexedDB | 13.41 |
| OPFS Main Thread | 280 |
| OPFS WebWorker | 104 |
| WASM SQLite (memory) | 19.1 |
| WASM SQLite (IndexedDB) | 37.12 |
Here we can notice a few things:
- Sending the data to a WebWorker and running it via the faster OPFS API is about twice as fast.
- WASM SQLite performs better on bulk operations compared to its single write latency. This is because sending the data to WASM and backwards is faster if it is done all at once instead of once per document.
### Big Bulk Reads
Now lets read 100 documents in a bulk request.
| Technology | Time in Milliseconds |
| ----------------------- | ------------------------------- |
| Cookies | 6.34 |
| LocalStorage | 0.39 |
| IndexedDB | 4.99 |
| OPFS Main Thread | 54.79 |
| OPFS WebWorker | 25.61 |
| WASM SQLite (memory) | 3.59 |
| WASM SQLite (IndexedDB) | 5.84 (35ms without cache) |
Here we can notice a few things:
- Reading many files in the OPFS webworker is about **twice as fast** compared to the slower main thread mode.
- WASM SQLite is surprisingly fast. Further inspection has shown that the WASM SQLite process keeps the documents in memory cached which improves the latency when we do reads directly after writes on the same data. When the browser tab is reloaded between the writes and the reads, finding the 100 documents takes about **35 milliseconds** instead.
## Performance Conclusions
- LocalStorage is really fast but remember that is has some downsides:
- It blocks the main JavaScript process and therefore should not be used for big bulk operations.
- Only Key-Value assignments are possible, you cannot use it efficiently when you need to do index based range queries on your data.
- OPFS is way faster when used in the WebWorker with the `createSyncAccessHandle()` method compare to using it directly in the main thread.
- SQLite WASM can be fast but you have to initially download the full binary and start it up which takes about half a second. This might not be relevant at all if your app is started up once and the used for a very long time. But for web-apps that are opened and closed in many browser tabs many times, this might be a problem.
-------------
## Possible Improvements
There is a wide range of possible improvements and performance hacks to speed up the operations.
- For IndexedDB I have made a list of [performance hacks here](../slow-indexeddb.md). For example you can do sharding between multiple database and webworkers or use a custom index strategy.
- OPFS is slow in writing one file per document. But you do not have to do that and instead you can store everything at a single file like a normal database would do. This improves performance dramatically like it was done with the RxDB [OPFS RxStorage](../rx-storage-opfs.md).
- You can mix up the technologies to optimize for multiple scenarios at once. For example in RxDB there is the [localstorage meta optimizer](../rx-storage-localstorage-meta-optimizer.md) which stores initial metadata in localstorage and "normal" documents inside of IndexedDB. This improves the initial startup time while still having the documents stored in a way to query them efficiently.
- There is the [memory-mapped](../rx-storage-memory-mapped.md) storage plugin in RxDB which maps data directly to memory. Using this in combination with a shared worker can improve pageloads and query time significantly.
- [Compressing](../key-compression.md) data before storing it might improve the performance for some of the storages.
- Splitting work up between [multiple WebWorkers](../rx-storage-worker.md) via [sharding](../rx-storage-sharding.md) can improve performance by utilizing the whole capacity of your users device.
Here you can see the [performance comparison](../rx-storage-performance.md) of various RxDB storage implementations which gives a better view of real world performance:
## Future Improvements
You are reading this in 2024, but the web does not stand still. There is a good chance that browser get enhanced to allow faster and better data operations.
- Currently there is no way to directly access a persistent storage from inside a WebAssembly process. If this changes in the future, running SQLite (or a similar database) in a browser might be the best option.
- Sending data between the main thread and a WebWorker is slow but might be improved in the future. There is a [good article](https://surma.dev/things/is-postmessage-slow/) about why `postMessage()` is slow.
- IndexedDB lately [got support](https://developer.chrome.com/blog/maximum-idb-performance-with-storage-buckets) for storage buckets (chrome only) which might improve performance.
## Follow Up
- Share my [announcement tweet](https://x.com/rxdbjs/status/1846145062847062391) -->
- Reproduce the benchmarks at the [github repo](https://github.com/pubkey/localstorage-indexeddb-cookies-opfs-sqlite-wasm)
- Learn how to use RxDB with the [RxDB Quickstart](../quickstart.md)
- Check out the [RxDB github repo](https://github.com/pubkey/rxdb) and leave a star β
---
## Using localStorage in Modern Applications - A Comprehensive Guide
# Using localStorage in Modern Applications: A Comprehensive Guide
When it comes to client-side storage in web applications, the localStorage API stands out as a simple and widely supported solution. It allows developers to store key-value pairs directly in a user's browser. In this article, we will explore the various aspects of the localStorage API, its advantages, limitations, and alternative storage options available for modern applications.
## What is the localStorage API?
The localStorage API is a built-in feature of web browsers that enables web developers to store small amounts of data persistently on a user's device. It operates on a simple key-value basis, allowing developers to save strings, numbers, and other simple data types. This data remains available even after the user closes the browser or navigates away from the page. The API provides a convenient way to maintain state and store user preferences without relying on server-side storage.
## Exploring local storage Methods: A Practical Example
Let's dive into some hands-on code examples to better understand how to leverage the power of localStorage. The API offers several methods for interaction, including setItem, getItem, removeItem, and clear. Consider the following code snippet:
```js
// Storing data using setItem
localStorage.setItem('username', 'john_doe');
// Retrieving data using getItem
const storedUsername = localStorage.getItem('username');
// Removing data using removeItem
localStorage.removeItem('username');
// Clearing all data
localStorage.clear();
```
## Storing Complex Data in JavaScript with JSON Serialization
While js localStorage excels at handling simple key-value pairs, it also supports more intricate data storage through JSON serialization. By utilizing JSON.stringify and JSON.parse, you can store and retrieve structured data like objects and arrays. Here's an example of storing a document:
```js
const user = {
name: 'Alice',
age: 30,
email: 'alice@example.com'
};
// Storing a user object
localStorage.setItem('user', JSON.stringify(user));
// Retrieving and parsing the user object
const storedUser = JSON.parse(localStorage.getItem('user'));
```
## Understanding the Limitations of local storage
Despite its convenience, localStorage does come with a set of limitations that developers should be aware of:
- **Non-Async Blocking API**: One significant drawback is that js localStorage operates as a non-async blocking API. This means that any operations performed on localStorage can potentially block the main thread, leading to slower application performance and a less responsive user experience.
- **Limited Data Structure**: Unlike more advanced databases, localStorage is limited to a simple key-value store. This restriction makes it unsuitable for storing complex data structures or managing relationships between data elements.
- **Stringification Overhead**: Storing [JSON data](./json-based-database.md) in localStorage requires stringifying the data before storage and parsing it when retrieved. This process introduces performance overhead, potentially slowing down operations by up to 10 times.
- **Lack of Indexing**: localStorage lacks indexing capabilities, making it challenging to perform efficient searches or iterate over data based on specific criteria. This limitation can hinder applications that rely on complex data retrieval.
- **Tab Blocking**: In a multi-tab environment, one tab's localStorage operations can impact the performance of other tabs by monopolizing CPU resources. You can reproduce this behavior by opening [this test file](https://pubkey.github.io/client-side-databases/database-comparison/index.html) in two browser windows and trigger localstorage inserts in one of them. You will observe that the indication spinner will stuck in both windows.
- **Storage Limit**: Browsers typically impose a storage limit of [around 5 MiB](https://developer.mozilla.org/en-US/docs/Web/API/Storage_API/Storage_quotas_and_eviction_criteria#web_storage) for each origin's localStorage.
## Reasons to Still Use localStorage
### Is localStorage Slow?
Contrary to concerns about performance, the localStorage API in JavaScript is surprisingly fast when compared to alternative storage solutions like [IndexedDB or OPFS](./localstorage-indexeddb-cookies-opfs-sqlite-wasm.md). It excels in handling small key-value assignments efficiently. Due to its simplicity and direct integration with browsers, accessing and modifying localStorage data incur minimal overhead. For scenarios where quick and straightforward data storage is required, localStorage remains a viable option. For example RxDB uses localStorage in the [localStorage meta optimizer](../rx-storage-localstorage-meta-optimizer.md) to manage simple key values pairs while storing the "normal" documents inside of another storage like IndexedDB.
## When Not to Use localStorage
While localStorage offers convenience, it may not be suitable for every use case. Consider the following situations where alternatives might be more appropriate:
- **Data Must Be Queryable**: If your application relies heavily on querying data based on specific criteria, localStorage might not provide the necessary querying capabilities. Complex data retrieval might lead to inefficient code and slow performance.
- **Big JSON Documents**: Storing large JSON documents in localStorage can consume a significant amount of memory and degrade performance. It's essential to assess the size of the data you intend to store and consider more robust solutions for handling substantial datasets.
- **Many Read/Write Operations**: Excessive read and write operations on localStorage can lead to performance bottlenecks. Other storage solutions might offer better performance and scalability for applications that require frequent data manipulation.
- **Lack of Persistence**: If your application can function without persistent data across sessions, consider using in-memory data structures like `new Map()` or `new Set()`. These options offer speed and efficiency for transient data.
## What to use instead of the localStorage API in JavaScript
### localStorage vs IndexedDB
While **localStorage** serves as a reliable storage solution for simpler data needs, it's essential to explore alternatives like **[IndexedDB](../rx-storage-indexeddb.md)** when dealing with more complex requirements. **IndexedDB** is designed to store not only key-value pairs but also JSON documents. Unlike localStorage, which usually has a storage limit of around 5-10MB per domain, IndexedDB can handle significantly larger datasets. IndexDB with its support for indexing facilitates efficient querying, making range queries possible. However, it's worth noting that IndexedDB lacks observability, which is a feature unique to localStorage through the `storage` event. Also,
complex queries can pose a challenge with IndexedDB, and while its performance is acceptable, IndexedDB can be [too slow](../slow-indexeddb.md) for some use cases.
```js
// localStorage can observe changes with the storage event.
// This feature is missing in IndexedDB
addEventListener("storage", (event) => {});
```
For those looking to harness the full power of IndexedDB with added capabilities, using wrapper libraries like [RxDB](https://rxdb.info/) is recommended. These libraries augment IndexedDB with features such as complex queries and observability, enhancing its usability for modern applications by providing a real database instead of only a key-value store.
In summary when you compare IndexedDB vs localStorage, IndexedDB will win at any case where much data is handled while localStorage has better performance on small key-value datasets.
### File System API (OPFS)
Another intriguing option is the OPFS (File System API). This API provides direct access to an origin-based, sandboxed filesystem which is highly optimized for performance and offers in-place write access to its content.
OPFS offers impressive performance benefits. However, working with the OPFS API can be complex, and it's only accessible within a **WebWorker**. To simplify its usage and extend its capabilities, consider using a wrapper library like [RxDB's OPFS RxStorage](../rx-storage-opfs.md), which builds a comprehensive database on top of the OPFS API. This abstraction allows you to harness the power of the OPFS API without the intricacies of direct usage.
### localStorage vs Cookies
Cookies, once a primary method of client-side data storage, have fallen out of favor in modern web development due to their limitations. While they can store data, they are about **100 times slower** when compared to the localStorage API. Additionally, cookies are included in the HTTP header, which can impact network performance. As a result, cookies are not recommended for data storage purposes in contemporary web applications.
### localStorage vs WebSQL
WebSQL, despite offering a SQL-based interface for client-side data storage, is a **deprecated technology** and should be avoided. Its API has been phased out of modern browsers, and it lacks the robustness of alternatives like IndexedDB. Moreover, WebSQL tends to be around 10 times slower than IndexedDB, making it a suboptimal choice for applications that demand efficient data manipulation and retrieval.
### localStorage vs sessionStorage
In scenarios where data persistence beyond a session is unnecessary, developers often turn to sessionStorage. This storage mechanism retains data only for the duration of a tab or browser session. It survives page reloads and restores, providing a handy solution for temporary data needs. However, it's important to note that sessionStorage is limited in scope and may not suit all use cases.
### AsyncStorage for React Native
For [React Native](../react-native-database.md) developers, the [AsyncStorage API](https://reactnative.dev/docs/asyncstorage) is the go-to solution, mirroring the behavior of localStorage but with asynchronous support. Since not all JavaScript runtimes support localStorage, AsyncStorage offers a seamless alternative for data persistence in React Native applications.
### `node-localstorage` for Node.js
Because native localStorage is absent in the **[Node.js](../nodejs-database.md)** JavaScript runtime, you will get the error `ReferenceError: localStorage is not defined` in Node.js or node based runtimes like Next.js. The [node-localstorage npm package](https://github.com/lmaccherone/node-localstorage) bridges the gap. This package replicates the browser's localStorage API within the Node.js environment, ensuring consistent and compatible data storage capabilities.
## localStorage in browser extensions
While browser extensions for chrome and firefox support the localStorage API, it is not recommended to use it in that context to store extension-related data. The browser will clear the data in many scenarios like when the users clear their browsing history.
Instead the [Extension Storage API](https://developer.mozilla.org/en-US/docs/Mozilla/Add-ons/WebExtensions/API/storage#properties) should be used for browser extensions.
In contrast to localStorage, the storage API works `async` and all operations return a Promise. Also it provides automatic sync to replicate data between all instances of that browser that the user is logged into. The storage API is even able to storage JSON-ifiable objects instead of plain strings.
```ts
// Using the storage API in chrome
await chrome.storage.local.set({ foobar: {nr: 1} });
const result = await chrome.storage.local.get('foobar');
console.log(result.foobar); // {nr: 1}
```
## localStorage in Deno and Bun
The **Deno** JavaScript runtime has a working localStorage API so running `localStorage.setItem()` and the other methods, will just work and the locally stored data is persisted across multiple runs.
**Bun** does not support the localStorage JavaScript API. Trying to use `localStorage` will error with `ReferenceError: Can't find variable: localStorage`. To store data locally in Bun, you could use the `bun:sqlite` module instead or directly use a in-JavaScript database with Bun support like [RxDB](https://rxdb.info/).
## Conclusion: Choosing the Right Storage Solution
In the world of modern web development, **localStorage** serves as a valuable tool for lightweight data storage. Its simplicity and speed make it an excellent choice for small key-value assignments. However, as application complexity grows, developers must assess their storage needs carefully. For scenarios that demand advanced querying, complex data structures, or high-volume operations, alternatives like IndexedDB, wrapper libraries with additional features like [RxDB](../), or platform-specific APIs offer more robust solutions. By understanding the strengths and limitations of various storage options, developers can make informed decisions that pave the way for efficient and scalable applications.
## Follow up
- Learn how to store and query data with RxDB in the [RxDB Quickstart](../quickstart.md)
- [Why IndexedDB is slow and how to fix it](../slow-indexeddb.md)
- [RxStorage performance comparison](../rx-storage-performance.md)
---
## Real-Time & Offline - RxDB for Mobile Apps
# Mobile Database - RxDB as Database for Mobile Applications
In today's digital landscape, mobile applications have become an integral part of our lives. From social media platforms to e-commerce solutions, mobile apps have transformed the way we interact with digital services. At the heart of any mobile app lies the database, a critical component responsible for storing, retrieving, and managing data efficiently. In this article, we will delve into the world of mobile databases, exploring their significance, challenges, and the emergence of [RxDB](https://rxdb.info/) as a powerful database solution for hybrid app development in frameworks like React Native and Capacitor.
## Understanding Mobile Databases
Mobile databases are specialized software systems designed to handle data storage and management for mobile applications. These databases are optimized for the unique requirements of mobile environments, which often include limited device resources, fluctuations in network connectivity, and the need for offline functionality.
There are various types of mobile databases available, each with its own strengths and use cases. Local databases, such as [SQLite](../rx-storage-sqlite.md) and Realm, reside directly on the user's device, providing offline capabilities and faster data access. Cloud-based databases, like [Firebase Realtime Database](./realtime-database.md) and Amazon DynamoDB, rely on remote servers to store and retrieve data, enabling synchronization across multiple devices. Hybrid databases, as the name suggests, combine the benefits of both local and cloud-based approaches, offering a balance between offline functionality and data synchronization.
## Introducing RxDB: A Paradigm Shift in Mobile Database Solutions
[RxDB](https://rxdb.info/), also known as Reactive Database, has emerged as a game-changer in the realm of mobile databases. Built on top of popular web technologies like JavaScript, TypeScript, and RxJS (Reactive Extensions for JavaScript), RxDB provides an elegant solution for seamless offline-first capabilities and real-time data synchronization in mobile applications.
Benefits of RxDB for Hybrid App Development
1. Offline-First Approach: One of the major advantages of RxDB is its ability to work in an offline mode. It allows mobile applications to store and access data locally, ensuring uninterrupted functionality even when the network connection is weak or unavailable. The database automatically syncs the data with the server once the connection is reestablished, guaranteeing data consistency.
2. [Real-Time Data Synchronization](../replication.md): RxDB leverages the power of real-time data synchronization, making it an excellent choice for applications that require collaborative features or live updates. It uses the concept of change streams to detect modifications made to the database and instantly propagates those changes across connected devices. This real-time synchronization enables seamless collaboration and enhances user experience.
3. Reactive Programming Paradigm: RxDB embraces the principles of reactive programming, which simplifies the development process by handling asynchronous events and data streams. By leveraging RxJS observables, developers can write concise, declarative code that reacts to changes in data, ensuring a highly responsive user experience. The reactive programming paradigm enhances code maintainability, scalability, and testability.
4. Easy Integration with Hybrid App Frameworks: RxDB seamlessly integrates with popular hybrid app development frameworks like [React Native](../react-native-database.md) and [Capacitor](../capacitor-database.md). This compatibility allows developers to leverage the existing ecosystem and tools of these frameworks, making the transition to RxDB smoother and more efficient. By utilizing RxDB within these frameworks, developers can harness the power of a robust database solution without sacrificing the advantages of hybrid app development.
5. Cross-Platform Support: RxDB enables developers to build cross-platform mobile applications that run seamlessly on both iOS and Android devices. This versatility eliminates the need for separate database implementations for different platforms, saving development time and effort. With RxDB, developers can focus on building a unified codebase and delivering a consistent user experience across platforms.
## Use Cases for RxDB in Hybrid App Development
1. [Offline-First Applications](../offline-first.md): [RxDB](https://rxdb.info/) is an ideal choice for applications that heavily rely on offline functionality. Whether it's a note-taking app, a task manager, or a survey application, RxDB ensures that users can continue working even when connectivity is compromised. The seamless synchronization capabilities of RxDB ensure that changes made offline are automatically propagated once the device reconnects to the internet.
2. Real-Time Collaboration: Applications that require real-time collaboration, such as messaging platforms or collaborative editing tools, can greatly benefit from RxDB. The real-time synchronization capabilities enable multiple users to work on the same data simultaneously, ensuring that everyone sees the latest updates in real-time.
3. Data-Intensive Applications: RxDB's performance and scalability make it suitable for data-intensive applications that handle large datasets or complex data structures. Whether it's a media-rich app, a data visualization tool, or an analytics platform, RxDB can handle the heavy lifting and provide a smooth user experience.
4. Cross-Platform Applications: Hybrid app frameworks like React Native and Capacitor have gained popularity due to their ability to build cross-platform applications. By utilizing RxDB within these frameworks, developers can create a unified codebase that runs seamlessly on both iOS and Android, significantly reducing development time and effort.
## FAQ
What is the best mobile database with built-in peer-to-peer networking?
RxDB functions as the leading mobile database with built-in peer-to-peer networking support. You connect mobile applications directly to each other without requiring a central server. The WebRTC replication plugin facilitates real-time data synchronization between devices using local networks or the internet. This peer-to-peer architecture reduces server costs and guarantees data availability during backend outages.
What are the best real-time database platforms for mobile apps?
RxDB stands out as the premier real-time database platform for mobile applications. You build highly responsive mobile applications featuring observable query implementations. RxDB updates your application interface automatically whenever underlying local data changes. The replication protocol continuously synchronizes these changes between your mobile clients and backend servers. You create collaborative applications where multiple users interact with real-time data effortlessly.
## Conclusion
Mobile databases play a vital role in the performance and functionality of mobile applications. RxDB, with its offline-first approach, real-time data synchronization, and seamless integration with hybrid app development frameworks like React Native and Capacitor, offers a robust solution for managing data in mobile apps. By leveraging the power of reactive programming, RxDB empowers developers to build highly responsive, scalable, and cross-platform applications that deliver an exceptional user experience. With its versatility and ease of use, RxDB is undoubtedly a database solution worth considering for hybrid app development. Embrace the power of RxDB and unlock the full potential of your mobile applications.
---
## RxDB β The Ultimate Offline Database with Sync and Encryption
When building modern applications, a reliable **offline database** can make all the difference. Users need fast, uninterrupted access to data, even without an internet connection, and they need that data to stay secure. **RxDB** meets these requirements by providing a **local-first** architecture, **real-time sync** to any backend, and optional **encryption** for sensitive fields.
In this article, we'll cover:
- Why an **offline database** approach significantly improves user experience
- How RxDBβs **sync** and **encryption** features work
- Step-by-step guidance on getting started
---
## Why Choose an Offline Database?
[Offline-first](../offline-first.md) or **local-first** software stores data directly on the client device. This strategy isnβt just about surviving network outages; it also makes your application faster, more user-friendly, and better at handling multiple usage scenarios.
### 1. Zero Loading Spinners
Applications that call remote servers for every request inevitably show loading spinners. With an offline database, read and write operations happen locallyβproviding near-instant feedback. Users no longer stare at progress indicators or wait for server responses, resulting in a smoother and more fluid experience.
### 2. Multi-Tab Consistency
Many websites mishandle data across multiple browser tabs. In an offline database, all tabs share the same local datastore. If the user updates data in one tab (like completing a to-do item), changes instantly reflect in every other tab. This removes complex multi-window synchronization problems.
### 3. Real-Time Data Feeds
Apps that rely on a purely server-driven approach often show stale data unless they add a separate real-time push system (like websockets). Local-first solutions with built-in replication essentially get real-time updates βfor free.β Once the server sends any changes, your local offline database updatesβkeeping your UI live and accurate.
### 4. Reduced Server Load
In a traditional app, every interaction triggers a request to the server, scaling up resource usage quickly as traffic grows. Offline-first setups replicate data to the client once, and subsequent local reads or writes do not stress the backend. Your server usage grows with the amount of dataβrather than every user actionβleading to more efficient scaling.
### 5. Simpler Development: Fewer Endpoints, No Extra State Library
Typical apps require numerous REST endpoints and possibly a client-side state manager (like Redux) to handle data flow. If you adopt an offline database, you can replicate nearly everything to the client. The local DB becomes your single source of truth, and you may skip advanced state libraries altogether.
## Introducing RxDB β A Powerful Offline Database Solution
**RxDB (Reactive Database)** is a **[NoSQL](./in-memory-nosql-database.md)** JavaScript database that lives entirely in your client environment. Itβs optimized for:
- **Offline-first usage**
- **Reactive queries** (your UI updates in real time)
- **Flexible replication** with various backends
- **Field-level encryption** to protect sensitive data
You can run RxDB in:
- **Browsers** ([IndexedDB](../rx-storage-indexeddb.md), [OPFS](../rx-storage-opfs.md))
- **Mobile hybrid apps** ([Ionic](./ionic-database.md), [Capacitor](../capacitor-database.md))
- **Native modules** ([React Native](../react-native-database.md))
- **Desktop environments** ([Electron](../electron-database.md))
- **[Node.js](../nodejs-database.md)** [Servers](../rx-server.md) or Scripts
Wherever your JavaScript executes, RxDB can serve as a robust offline database.
---
## Quick Setup Example
Below is a short demo of how to create an RxDB [database](../rx-database.md), add a [collection](../rx-collection.md), and observe a [query](../rx-query.md). You can expand upon this to enable encryption or full sync.
```ts
import { createRxDatabase } from 'rxdb/plugins/core';
import { getRxStorageLocalstorage } from 'rxdb/plugins/storage-localstorage';
async function initDB() {
// Create a local offline database
const db = await createRxDatabase({
name: 'myOfflineDB',
storage: getRxStorageLocalstorage()
});
// Add collections
await db.addCollections({
tasks: {
schema: {
title: 'tasks schema',
version: 0,
type: 'object',
primaryKey: 'id',
properties: {
id: { type: 'string' },
title: { type: 'string' },
done: { type: 'boolean' }
}
}
}
});
// Observe changes in real time
db.tasks
.find({ selector: { done: false } })
.$ // returns an observable that emits whenever the result set changes
.subscribe(undoneTasks => {
console.log('Currently undone tasks:', undoneTasks);
});
return db;
}
```
Now the `tasks` collection is ready to store data offline. You could also [replicate](../replication.md) it to a backend, encrypt certain fields, or utilize more advanced features like conflict resolution.
## How Offline Sync Works in RxDB
RxDB uses a [Sync Engine](../replication.md) that pushes local changes to the server and pulls remote updates back down. This ensures local data is always fresh and that the server has the latest offline edits once the device reconnects.
**Multiple Plugins** exist to handle various backends or replication methods:
- [CouchDB](../replication-couchdb.md) or **PouchDB**
- [Google Firestore](./firestore-alternative.md)
- [GraphQL](../replication-graphql.md) endpoints
- REST / [HTTP](../replication-http.md)
- **WebSocket** or [WebRTC](../replication-webrtc.md) (for peer-to-peer sync)
You pick the plugin that fits your stack, and RxDB handles everything from conflict detection to event emission, allowing you to focus on building your user-facing features.
```ts
import { replicateRxCollection } from 'rxdb/plugins/replication';
replicateRxCollection({
collection: db.tasks,
replicationIdentifier: 'tasks-sync',
pull: { /* fetch updates from server */ },
push: { /* send local writes to server */ },
live: true // keep them in sync constantly
});
```
## Securing Your Offline Database with Encryption
Local data can be a risk if itβs sensitive or personal. RxDB offers [encryption plugins](../encryption.md) to keep specific document fields secure at rest.
#### Encryption Example
```ts
import { createRxDatabase } from 'rxdb/plugins/core';
import { getRxStorageLocalstorage } from 'rxdb/plugins/storage-localstorage';
import { wrappedKeyEncryptionCryptoJsStorage } from 'rxdb/plugins/encryption-crypto-js';
async function initSecureDB() {
// Wrap the storage with crypto-js encryption
const encryptedStorage = wrappedKeyEncryptionCryptoJsStorage({
storage: getRxStorageLocalstorage()
});
// Create database with a password
const db = await createRxDatabase({
name: 'secureOfflineDB',
storage: encryptedStorage,
password: 'myTopSecretPassword'
});
// Define an encrypted collection
await db.addCollections({
userSecrets: {
schema: {
title: 'encrypted user data',
version: 0,
type: 'object',
primaryKey: 'id',
properties: {
id: { type: 'string' },
secretData: { type: 'string' }
},
required: ['id'],
encrypted: ['secretData'] // field is encrypted at rest
}
}
});
return db;
}
```
When the device is off or the database file is extracted, `secretData` remains unreadable without the specified password. This ensures only authorized parties can access sensitive fields, even in offline scenarios.
## FAQ
Which offline-first database has the fastest sync speeds?
RxDB provides one of the fastest sync speeds among offline-first databases. RxDB synchronizes data in bulks. This bulk processing reduces overhead and minimizes network requests. The sync process operates as fast as your backend can provide or store the data. You avoid the bottlenecks common in row-by-row replication. You achieve near-instant data consistency across large datasets.
What are the best offline-first database solutions for field worker apps?
RxDB provides a reliable offline database for field worker applications. Field workers require data access in areas with poor network coverage. RxDB stores data locally on the device. Users read and write data without an internet connection. The sync engine pushes local changes to the server when the device reconnects. RxDB resolves conflicts automatically to prevent data loss. You ensure continuous operations for your field workforce regardless of network status.
What are the top rated mobile databases with built-in offline and peer syncing?
RxDB ranks highly among mobile databases that feature built-in offline capabilities and peer-to-peer synchronization. You construct responsive mobile applications using local data storage. RxDB operates directly on the mobile device to guarantee data availability without an internet connection. The WebRTC replication plugin facilitates direct peer-to-peer syncing between devices. You achieve real-time data sharing across mobile clients without relying on a centralized server.
What is the best offline-first database for real-time data syncing?
RxDB serves as the best offline-first database for real-time data syncing. RxDB uses observable queries to push updates to the user interface. You receive instant feedback when local data changes. The sync engine processes data replication with your server automatically. You eliminate manual data fetching and continuous polling. Real-time subscriptions guarantee your application state reflects the most recent data.
## Follow Up
Integrating an offline database approach into your app delivers near-instant interactions, true multi-tab data consistency, automatic real-time updates, and reduced server dependencies. By choosing RxDB, you gain:
- Offline-first local storage
- Flexible replication to various backends
- Encryption of sensitive fields
- Reactive queries for real-time UI updates
RxDB transforms how you build and scale appsβno more loading spinners, no more stale data, no more complicated offline handling. Everything is local, synced, and secured.
Continue your learning path:
- **Explore the RxDB Ecosystem**
Dive into additional features like [Compression](../key-compression.md) or advanced [Conflict Handling](../transactions-conflicts-revisions.md#custom-conflict-handler) to optimize your offline database.
- **Learn More About Offline-First**
Read our [Offline First documentation](../offline-first.md) for a deeper understanding of why local-first architectures improve user experience and reduce server load.
- **Join the Community**
Have questions or feedback? Connect with us on the [RxDB Chat](/chat/) or open an issue on [GitHub](/code/).
- **Upgrade to Premium**
If you need high-performance featuresβlike [SQLite storage](../rx-storage-sqlite.md) for mobile or the [Web Crypto-based encryption plugin](/premium/)βconsider our premium offerings.
By adopting an offline database approach with RxDB, you unlock speed, reliability, and security for your applicationsβleading to a truly seamless user experience.
---
## Building an Optimistic UI with RxDB
An **Optimistic User Interface (UI)** is a design pattern that provides instant feedback to the user by **assuming** that an operation or server call will succeed. Instead of showing loading spinners or waiting for server confirmations, the UI immediately reflects the user's intended action and later reconciles the displayed data with the actual server response. This approach drastically improves perceived performance and user satisfaction.
## Benefits of an Optimistic UI
Optimistic UIs offer a host of advantages, from improving the user experience to streamlining the underlying infrastructure. Below are some key reasons why an optimistic approach can revolutionize your application's performance and scalability.
### Better User Experience with Optimistic UI
- **No loading spinners, [near-zero latency](./zero-latency-local-first.md)**: Users perceive their actions as instant. Any actual network delays or slow server operations can be handled behind the scenes.
- **Offline capability**: Optimistic UI pairs perfectly with offline-first apps. Users can continue to interact with the application even when offline, and changes will be synced automatically once the network is available again.
### Better Scaling and Easier to Implement
- **Fewer server endpoints**: Instead of sending a separate HTTP request for every single user interaction, you can batch updates and sync them in bulk.
- **Less server load**: By handling changes locally and syncing in batches, you reduce the volume of server round-trips.
- **Automated error handling**: If a request fails or a document is in conflict, RxDB's [replication](../replication.md) mechanism can seamlessly retry and resolve conflicts in the background, without requiring a separate endpoint or manual user intervention.
## Building Optimistic UI Apps with RxDB
Now that we know what an optimistic UI is, lets build one with RxDB.
### Local Database: The Backbone of an Optimistic UI
A local database is the heart of an Optimistic UI. With RxDB, **all application state** is stored locally, ensuring seamless and instant updates. You can choose from multiple storage backends based on your runtime - check out [RxDB Storage Options](../rx-storage.md) to see which engines (IndexedDB, [SQLite](../rx-storage-sqlite.md), or custom) suit your environment best.
- **Instant Writes**: When users perform an action (like clicking a button or submitting a form), the changes are written directly to the local RxDB database. This immediate local write makes the UI feel snappy and removes the dependency on instantaneous server responses.
- [Offline-First](../offline-first.md): Because data is managed locally, your app continues to operate smoothly even without an internet connection. Users can view, create, and update data at any time, assured that changes will sync automatically once they're back online.
### Real-Time UI Changes on Updates
RxDB's core is built around observables that react to any state changes - whether from local writes or incoming replication from the server.
- **Automatic UI refresh**: Any query or document subscription in RxDB automatically notifies your UI layer when data changes. There's no need to manually poll or refetch.
- **Cross-tab updates**: If you have the same RxDB database open in multiple [browser](./browser-database.md) tabs, changes in one tab instantly propagate to the others.
- **Event-Reduce Algorithm**: Under the hood, RxDB uses the [event-reduce algorithm](https://github.com/pubkey/event-reduce) to minimize overhead. Instead of re-running expensive queries, RxDB calculates the smallest possible updates needed to keep query results accurate - further boosting real-time performance.
### Replication with a Server
While local storage is key to an Optimistic UI, most applications ultimately need to sync with a remote back end. RxDB offers a [powerful replication system](../replication.md) that can sync your local data with virtually any server/database in the background:
- **Incremental and real-time**: RxDB continuously pushes local changes to the server when a network is available and fetches server updates as they happen.
- **Conflict resolution**: If changes happen offline or multiple clients update the same data, RxDB detects conflicts and makes it straightforward to resolve them.
- **Flexible transport**: Beyond simple HTTP polling, you can incorporate WebSockets, Server-Sent Events (SSE), or other protocols for instant, server-confirmed changes broadcast to all connected clients. See [this guide](./websockets-sse-polling-webrtc-webtransport.md) to learn more.
By combining local-first data handling with real-time synchronization, RxDB delivers most of what an Optimistic UI needs - right out of the box. The result is a seamless user experience where interactions never feel blocked by slow networks, and any conflicts or final validations are quietly handled in the background.
#### Handling Offline Changes and Conflicts
- **Offline-first approach**: All writes are initially stored in the local database. When connectivity returns, RxDB's replication automatically pushes changes to the server.
- **Conflict resolution**: If multiple clients edit the same documents while offline, conflicts are automatically detected and can be resolved gracefully (more on conflicts below).
#### WebSockets, SSE, or Beyond
For truly real-time communication - where server-confirmed changes instantly reach all clients - you can go beyond simple HTTP polling. Use WebSockets, Server-Sent Events (SSE), or other streaming protocols to broadcast updates the moment they occur. This pattern excels in scenarios like chats, collaborative editors, or dynamic dashboards.
To learn more about these protocols and their integration with RxDB, check out [this guide](./websockets-sse-polling-webrtc-webtransport.md).
## Optimistic UI in Various Frameworks
### Angular Example
[Angular](./angular-database.md)'s `async` pipe works smoothly with RxDB's observables. Suppose you have a `myCollection` of documents, you can directly subscribe in the template:
```html
{{ doc.name }}
```
This snippet:
- Subscribes to `myCollection.find().$`, which emits live updates whenever [documents](../rx-document.md) in the [collection](../rx-collection.md) change.
- Passes the emitted array of documents into docs.
- Renders each document in a list item, instantly reflecting any changes.
### React Example
In [React](./react-database.md), you can utilize signals or other state management tools. For instance, if we have an [RxDB extension](../reactivity.md) that exposes a **signal**:
```tsx
import React from 'react';
function MyComponent({ myCollection }) {
// .find().$$ provides a signal that updates whenever data changes
const docsSignal = myCollection.find().$$;
return (
{docs.map((doc) => (
{doc.name}
))}
);
}
export default MyComponent;
```
When you call `docsSignal.value` or use a hook like useSignal, it pulls the latest value from the [RxDB query](../rx-query.md). Whenever the collection updates, the signal emits the new data, and React re-renders the component instantly.
## Downsides of Optimistic UI Apps
While Optimistic UIs feel snappy, there are some caveats:
- **Conflict Resolution**:
With an optimistic approach, multiple offline devices might edit the same data. When syncing back, conflicts occur that must be merged. RxDB uses [revisions](../transactions-conflicts-revisions.md) to detect and handle these conflicts.
- **User Confusion**:
Users may see changes that haven't yet been confirmed by the server. If a subsequent server validation fails, the UI must revert to a previous state. Clear visual feedback or user notifications help reduce confusion.
- **Server Compatibility**:
The server must be capable of storing and returning revision metadata (for instance, a timestamp or versioning system). Check out RxDB's [replication docs](../replication.md) for details on how to structure your back end.
- **Storage Limits**:
Storing data in the client has practical [size limits](./indexeddb-max-storage-limit.md). [IndexedDB](../rx-storage-indexeddb.md) or other client-side storages have constraints (though usually quite large). See [storage comparisons](./localstorage-indexeddb-cookies-opfs-sqlite-wasm.md).
## Conflict Resolution Strategies
- **Last Write to Server Wins**:
A simplest-possible method: whatever update reaches the server last overrides previous data. Good for non-critical data like βlike" counts or ephemeral states.
- **Revision-Based Merges**:
Use revision numbers or timestamps to track concurrent edits. Merge them intelligently by combining fields or choosing the latest sub-document changes. This is ideal for collaborative apps where you don't want to overwrite entire records.
- **User Prompts**:
In certain workflows (e.g., shipping forms, e-commerce checkout), you may need to notify the user about conflicts and let them choose which version to keep.
- **First Write to Server Wins (RxDB Default)**:
RxDB's default approach is to let the first successful push define the latest version. Any incoming push with an outdated revision triggers a conflict that must be resolved on the client side. Learn more at [here](../transactions-conflicts-revisions.md).
## When (and When Not) to Use Optimistic UI
- When to Use
- [Real-time interactions](./realtime-database.md) like chat apps, social feeds, or βLikes."
Situations where high success rates of operations are expected (most writes don't fail).
- Apps that need an [offline-first approach](../offline-first.md) or handle intermittent connectivity gracefully.
- When Not to Use
- Large, complex transactions with high failure rates.
- Scenarios requiring heavy server validations or approvals (for example, financial transactions with complex rules).
- Workflows where immediate feedback could mislead users about an operation's success probability.
- Assessing Risk
- Consider the likelihood that a user's action might fail. If it's very low, optimistic UI is often best.
- If frequent failures or complex validations occur, consider a hybrid approach: partial optimistic updates for some actions, while more critical operations rely on immediate server confirmation.
## Follow Up
Ready to start building your own Optimistic UI with RxDB? Here are some next steps:
1. **Do the [RxDB Quickstart](https://rxdb.info/quickstart.html)**
If you're brand new to RxDB, the quickstart guide will walk you through installation and setting up your first project.
2. **Check Out the Demo App**
A live [RxDB Quickstart Demo](https://pubkey.github.io/rxdb-quickstart/) showcases optimistic updates and real-time syncing. Explore the code to see how it works.
3. **Star the GitHub Repo**
Show your support for RxDB by starring the [RxDB GitHub Repository](https://github.com/pubkey/rxdb).
By combining RxDB's powerful offline-first capabilities with the principles of an Optimistic UI, you can deliver snappy, near-instant user interactions that keep your users engaged - no matter the network conditions. Get started today and give your users the experience they deserve!
---
## RxDB as a Database for Progressive Web Apps (PWA)
Progressive Web Apps (PWAs) have revolutionized the digital landscape, offering users an immersive blend of web and native app experiences. At the heart of every successful PWA lies effective data management, and this is where RxDB comes into play. In this article, we'll explore the dynamic synergy between RxDB, a robust client-side database, and Progressive Web Apps, uncovering how RxDB enhances data handling, synchronization, and overall performance, propelling PWAs into a new era of excellence.
## What is a Progressive Web App
Progressive Web Apps are the future of web development, seamlessly combining the best of both web and mobile app worlds. They can be easily installed on the user's home screen, function offline, and load at lightning speed. Unlike hybrid apps, PWAs offer a consistent user experience across platforms, making them a versatile choice for modern applications.
PWAs bring a plethora of advantages to the table. They eliminate the hassle of app store installations and updates, reduce dependency on network connectivity, and prioritize fast loading times. By harnessing the power of service workers and intelligent caching mechanisms, PWAs ensure users can access content even in offline mode. Furthermore, PWAs are device-agnostic, seamlessly adapting to various devices, from desktops to smartphones.
## Introducing RxDB as a Client-Side Database for PWAs
At the heart of PWAs lies efficient data management, and RxDB steps in as a reliable ally. As a client-side [NoSQL](./in-memory-nosql-database.md) database, RxDB seamlessly integrates into web applications, offering real-time data synchronization and manipulation capabilities. This article sheds light on the transformative potential of RxDB as it collaborates harmoniously with PWAs, enabling local-first strategies and elevating user interactions to a whole new level.
### Getting Started with RxDB
RxDB emerges as a reactive, schema-based NoSQL database crafted explicitly for client-side applications. Its real-time data synchronization and responsiveness align seamlessly with the dynamic demands of modern PWAs.
#### Local-First Approach
The cornerstone of RxDB's philosophy is the local-first approach, empowering PWAs to prioritize data storage and manipulation on the client side. This paradigm ensures that PWAs remain functional even when offline, allowing users to access and interact with data seamlessly. RxDB bridges any gaps in data synchronization once network connectivity is restored.
#### Observable Queries
Observable queries (aka **Live Queries**) serve as the engine of RxDB's dynamic capabilities. By leveraging these queries, PWAs can monitor and respond to data changes in real time. The result is an engaging user interface with instantaneous updates that captivate users and keep them engaged.
```ts
await db.heroes.find({
selector: {
healthpoints: {
$gt: 0
}
}
})
.$ // the $ returns an observable that emits each time the result set of the query changes
.subscribe(aliveHeroes => console.dir(aliveHeroes));
```
#### Multi-Tab Support
RxDB extends its prowess to multi-tab scenarios, guaranteeing data consistency across different tabs or windows of the same PWA. This feature promotes a seamless transition between various sections of the application, while minimizing data conflicts.
### Using RxDB in a Progressive Web App
Integrating RxDB into a Progressive Web App, driven by technologies like [React](./react-database.md), is a straightforward process. By configuring RxDB and installing the necessary packages, developers establish a solid foundation for robust data management within their PWA.
## Exploring Different RxStorage Layers
RxDB caters to diverse needs through its various RxStorage layers:
- [LocalStorage RxStorage](../rx-storage-localstorage.md): Leveraging the capabilities of the browser's LocalStorage API for storage.
- [IndexedDB RxStorage](../rx-storage-indexeddb.md): Tapping into the browser's IndexedDB for efficient data storage.
- [OPFS RxStorage](../rx-storage-opfs.md): Interfacing with the Offline-First Persistence System for seamless persistence.
- [Memory RxStorage](../rx-storage-memory.md): Storing data in memory, ideal for temporary data requirements.
This flexibility empowers developers to optimize data storage based on the unique needs of their PWA.
Synchronizing Data with RxDB between PWA Clients and Servers
To facilitate seamless data synchronization between PWA clients and servers, RxDB offers a range of replication options:
- [RxDB Replication Algorithm](../replication.md): RxDB introduces its own replication algorithm, enabling efficient and reliable data synchronization between clients and servers.
- [CouchDB Replication](../replication-couchdb.md): Leveraging its roots in CouchDB, RxDB facilitates smooth data replication between clients and CouchDB servers, ensuring data consistency and synchronization across devices.
- [Firestore Replication](../replication-firestore.md): RxDB synchronizes data with Google Firestore, a real-time cloud-hosted NoSQL database. This integration guarantees up-to-date data across different instances of the PWA.
- [Peer-to-Peer (P2P) via WebRTC](../replication-webrtc.md) Replication: RxDB supports P2P replication, facilitating direct data synchronization between clients without intermediaries. This decentralized approach is invaluable in scenarios where server infrastructure is limited.
## Advanced RxDB Features and Techniques
### Encryption of Local Data
RxDB empowers PWAs with the ability to encrypt local data, enhancing data security and safeguarding sensitive information. This feature is indispensable for applications handling user credentials, financial transactions, and other confidential data.
### Indexing and Performance Optimization
Performance optimization is a top priority for PWAs. RxDB addresses this concern by offering indexing options that expedite data retrieval, resulting in a snappier user interface and heightened responsiveness.
### JSON Key Compression
RxDB introduces JSON key compression, a feature that reduces storage requirements. This optimization is particularly beneficial for PWAs dealing with substantial data volumes, enhancing overall efficiency and resource utilization.
### Change Streams and Event Handling
RxDB introduces change streams, enabling PWAs to react to data changes in real time. This capability empowers dynamic updates to the user interface, promoting interactivity and engagement.
## Conclusion
In the ever-evolving landscape of web application development, Progressive Web Apps continue to redefine user experiences. RxDB emerges as a pivotal player, seamlessly integrating with PWAs and enhancing their capabilities. With features like the local-first approach, observable queries, replication mechanisms, and advanced encryption, RxDB empowers developers to create responsive, offline-capable, and data-driven PWAs. As the demand for sophisticated PWAs continues to surge, RxDB remains an indispensable tool for developers aiming to push the boundaries of innovation and redefine the standards of user engagement. By embracing RxDB, developers ensure their PWAs remain at the forefront of the digital revolution, offering seamless and immersive experiences to users around the world.
## Follow Up
To explore more about RxDB and leverage its capabilities for browser database development, check out the following resources:
- [RxDB GitHub Repository](https://github.com/pubkey/rxdb): Visit the official GitHub repository of RxDB to access the source code, documentation, and community support.
- [RxDB Quickstart](../quickstart.md): Get started quickly with RxDB by following the provided quickstart guide, which provides step-by-step instructions for setting up and using RxDB in your projects.
- [RxDB Progressive Web App in Angular Example](https://github.com/pubkey/rxdb/tree/master/examples/angular)
---
## RxDB as a Database for React Applications
In the rapidly evolving landscape of web development, React has emerged as a cornerstone technology for building dynamic and responsive user interfaces. With the increasing complexity of modern web applications, efficient data management becomes pivotal. This article delves into the integration of RxDB, a potent client-side database, with React applications to optimize data handling and elevate the overall user experience.
React has revolutionized the way web applications are built by introducing a component-based architecture. This approach enables developers to create reusable UI components that efficiently update in response to changes in data. The virtual DOM mechanism, a key feature of React, facilitates optimized rendering, enhancing performance and user interactivity.
While React excels at managing the user interface, the need for efficient data storage and retrieval mechanisms is equally significant. A client-side database brings several advantages to React applications:
- Improved Performance: Local data storage reduces the need for frequent server requests, resulting in faster data retrieval and enhanced application responsiveness.
- Offline Capabilities: A client-side database enables offline access to data, allowing users to interact with the application even when they are disconnected from the internet.
- Real-Time Updates: With the ability to observe changes in data, client-side databases facilitate real-time updates to the UI, ensuring users are always presented with the latest information.
- Reduced Server Load: By handling data operations locally, client-side databases alleviate the load on the server, contributing to a more scalable architecture.
## Introducing RxDB as a JavaScript Database
RxDB, a powerful JavaScript database, has garnered attention as an optimal solution for managing data in React applications. Built on top of the IndexedDB standard, RxDB combines the principles of reactive programming with database management. Its core features include reactive data handling, offline-first capabilities, and robust data replication.
## What is RxDB?
RxDB, short for Reactive Database, is an open-source JavaScript database that seamlessly integrates reactive programming with database operations. It offers a comprehensive API for performing database actions and synchronizing data across clients and servers. RxDB's underlying philosophy revolves around observables, allowing developers to reactively manage data changes and create dynamic user interfaces.
### Reactive Data Handling
One of RxDB's standout features is its support for reactive data handling. Traditional databases often require manual intervention for data fetching and updating, leading to complex and error-prone code. RxDB, however, automatically notifies subscribers whenever data changes occur, eliminating the need for explicit data manipulation. This reactive approach simplifies code and enhances the responsiveness of React components.
### Local-First Approach
RxDB embraces a [local-first](../offline-first.md) methodology, enabling applications to function seamlessly even in offline scenarios. By storing data locally, RxDB ensures that users can interact with the application and make updates regardless of internet connectivity. Once the connection is reestablished, RxDB synchronizes the local changes with the remote database, maintaining data consistency across devices.
### Data Replication
Data replication is a cornerstone of modern applications that require synchronization between multiple clients and servers. RxDB provides robust data replication mechanisms that facilitate real-time synchronization between different instances of the database. This ensures that changes made on one client are promptly propagated to others, contributing to a cohesive and unified user experience.
### Observable Queries
RxDB extends the concept of observables beyond data changes. It introduces observable queries, allowing developers to observe the results of database queries. This feature enables automatic updates to query results whenever relevant data changes occur. [Observable queries](../rx-query.md) simplify state management by eliminating the need to manually trigger updates in response to changing data.
```ts
await db.heroes.find({
selector: {
healthpoints: {
$gt: 0
}
}
})
.$ // the $ returns an observable that emits each time the result set of the query changes
.subscribe(aliveHeroes => console.dir(aliveHeroes));
```
### Multi-Tab Support
Web applications often operate in multiple browser tabs or windows. RxDB accommodates this scenario by offering built-in multi-tab support. It ensures that data changes made in one tab are efficiently propagated to other tabs, maintaining data consistency and providing a seamless experience for users interacting with the application across different tabs.
### RxDB vs. Other React Database Options
While considering database options for React applications, RxDB stands out due to its unique combination of reactive programming and database capabilities. Unlike traditional solutions such as IndexedDB or Web Storage, which provide basic data storage, RxDB offers a dedicated database solution with advanced features. Additionally, while state management libraries like Redux and MobX can be adapted for database use, RxDB provides an integrated solution specifically designed for handling data.
### IndexedDB in React and the Advantage of RxDB
Using IndexedDB directly in React can be challenging due to its low-level, callback-based API which doesn't align neatly with modern React's Promise and async/await patterns. This intricacy often leads to bulky and complex implementations for developers. Also, when used wrong, IndexedDB can have a worse [performance profile](../slow-indexeddb.md) than it could have. In contrast, RxDB, with the [IndexedDB RxStorage](../rx-storage-indexeddb.md) and the [LocalStorage RxStorage](../rx-storage-localstorage.md), abstracts these complexities, integrating reactive programming and providing a more streamlined experience for data management in React applications. Thus, RxDB offers a more intuitive approach, eliminating much of the manual overhead required with IndexedDB.
### Using RxDB in a React Application
The process of integrating RxDB into a React application is straightforward. Begin by installing RxDB as a dependency:
`npm install rxdb rxjs`
Once installed, RxDB can be imported and initialized within your React components. The following code snippet illustrates a basic setup:
```javascript
import { createRxDatabase } from 'rxdb';
import { getRxStorageLocalstorage } from 'rxdb/plugins/storage-localstorage';
const db = await createRxDatabase({
name: 'heroesdb', // <- name
storage: getRxStorageLocalstorage(), // <- RxStorage
password: 'myPassword', // <- password (optional)
multiInstance: true, // <- multiInstance (optional, default: true)
eventReduce: true, // <- eventReduce (optional, default: false)
cleanupPolicy: {} // <- custom cleanup policy (optional)
});
```
### Using RxDB React Hooks
The [rxdb-hooks](https://github.com/cvara/rxdb-hooks) package provides a set of React hooks that simplify data management within components. These hooks leverage RxDB's reactivity to automatically update components when data changes occur. The following example demonstrates the usage of the `useRxCollection` and `useRxQuery` hooks to query and observe a collection:
```ts
const collection = useRxCollection('characters');
const query = collection.find().where('affiliation').equals('Jedi');
const {
result: characters,
isFetching,
fetchMore,
isExhausted,
} = useRxQuery(query, {
pageSize: 5,
pagination: 'Infinite',
});
if (isFetching) {
return 'Loading...';
}
return (
{characters.map((character, index) => (
))}
{!isExhausted && }
);
```
### Different RxStorage Layers for RxDB
RxDB offers multiple storage layers, each backed by a different underlying technology. Developers can choose the storage layer that best suits their application's requirements. Some available options include:
- [LocalStorage RxStorage](../rx-storage-localstorage.md): Built on top of the browser's LocalStorage API.
- [IndexedDB RxStorage](../rx-storage-indexeddb.md): The default RxDB storage layer, providing efficient data storage in modern browsers.
- [OPFS RxStorage](../rx-storage-opfs.md): Uses the Origin Private File System (OPFS) for storage, suitable for [Electron applications](../electron-database.md).
- [Memory RxStorage](../rx-storage-memory.md): Stores data in memory, primarily intended for testing and development purposes.
- [SQLite RxStorage](../rx-storage-sqlite.md): Stores data in an SQLite database. Can be used in a browser with React by using a SQLite database that was [compiled to WebAssembly](https://sqlite.org/wasm/doc/trunk/index.md). Using SQLite in React might not be the best idea, because a compiled SQLite wasm file is about one megabyte of code that has to be loaded and rendered by your users. Using native browser APIs like IndexedDB and OPFS have shown to be a more optimal database solution for browser based React apps compared to SQLite.
### Synchronizing Data with RxDB between Clients and Servers
The offline-first approach is a fundamental principle of RxDB's design. When dealing with client-server synchronization, RxDB ensures that changes made offline are captured and propagated to the server once connectivity is reestablished. This mechanism guarantees that data remains consistent across different client instances, even when operating in an occasionally connected environment.
RxDB offers a range of [replication plugins](../replication.md) that facilitate data synchronization between clients and servers. These plugins support various synchronization strategies, such as one-way replication, two-way replication, and custom conflict resolution. Developers can select the appropriate plugin based on their application's synchronization requirements.
### Advanced RxDB Features and Techniques
Encryption of Local Data
Security is paramount when handling sensitive user data. RxDB supports [data encryption](./react-native-encryption.md), ensuring that locally stored information remains protected from unauthorized access. This feature is particularly valuable when dealing with sensitive data in offline scenarios.
### Indexing and Performance Optimization
Efficient indexing is critical for achieving optimal database performance. RxDB provides mechanisms to define indexes on specific fields, enhancing query speed and reducing the computational overhead of data retrieval.
### JSON Key Compression
RxDB employs JSON key compression to reduce storage space and improve performance. This technique minimizes the memory footprint of the database, making it suitable for applications with limited resources.
### Change Streams and Event Handling
RxDB enables developers to subscribe to change streams, which emit events whenever data changes occur. This functionality facilitates real-time event handling and provides opportunities for implementing features such as notifications and live updates.
## Conclusion
In the realm of React application development, efficient data management is pivotal to delivering a seamless and engaging user experience. RxDB emerges as a compelling solution, seamlessly integrating reactive programming principles with sophisticated database capabilities. By adopting RxDB, React developers can harness its powerful features, including reactive data handling, offline-first support, and real-time synchronization. With RxDB as a foundational pillar, React applications can excel in responsiveness, scalability, and data integrity. As the landscape of web development continues to evolve, RxDB remains a steadfast companion for creating robust and dynamic React applications.
## Follow Up
To explore more about RxDB and leverage its capabilities for browser database development, check out the following resources:
- [RxDB GitHub Repository](https://github.com/pubkey/rxdb): Visit the official GitHub repository of RxDB to access the source code, documentation, and community support.
- [RxDB Quickstart](../quickstart.md): Get started quickly with RxDB by following the provided quickstart guide, which provides step-by-step instructions for setting up and using RxDB in your projects.
- [RxDB React Example at GitHub](https://github.com/pubkey/rxdb/tree/master/examples/react)
---
## IndexedDB Database in React Apps - The Power of RxDB
Building robust, [offline-capable](../offline-first.md) React applications often involves leveraging browser storage solutions to manage data. IndexedDB is one such powerful tool, but its raw API can be challenging to work with directly. RxDB abstracts away much of IndexedDB's complexity, providing a more developer-friendly experience. In this article, we'll explore what IndexedDB is, why it's beneficial in React applications, the challenges of using plain IndexedDB, and how [RxDB](https://rxdb.info/) can simplify your development process while adding advanced features.
## What is IndexedDB?
[IndexedDB](https://developer.mozilla.org/en-US/docs/Web/API/IndexedDB_API) is a low-level API for storing significant amounts of structured data in the browser. It provides a transactional database system that can store key-value pairs, complex objects, and more. This storage engine is asynchronous and supports advanced data types, making it suitable for offline storage and complex web applications.
## Why Use IndexedDB in React
When building React applications, IndexedDB can play a crucial role in enhancing both performance and user experience. Here are some reasons to consider using IndexedDB:
- **Offline-First / Local-First**: By storing data locally, your application remains functional even without an internet connection.
- **Performance**: Using local data means [zero latency](./zero-latency-local-first.md) and no loading spinners, as data doesn't need to be fetched over a network.
- **Easier Implementation**: Replicating all data to the client once is often simpler than implementing multiple endpoints for each user interaction.
- **Scalability**: Local data reduces server load because queries run on the client side, decreasing server bandwidth and processing requirements.
## Why To Not Use Plain IndexedDB
While IndexedDB itself is powerful, its native API comes with several drawbacks for everyday application developers:
- **Callback-Based API**: IndexedDB was designed with callbacks rather than modern Promises, making asynchronous code more cumbersome.
- **Complexity**: IndexedDB is low-level, intended for library developers rather than for app developers who simply want to store data.
- **Basic Query API**: Its rudimentary query capabilities limit how you can efficiently perform complex queries, whereas libraries like RxDB offer more advanced query features.
- **TypeScript Support**: Ensuring good TypeScript support with IndexedDB is challenging, especially when trying to enforce document type consistency.
- **Lack of Observable API**: IndexedDB doesn't provide an observable API out of the box. RxDB solves this by enabling you to observe query results or specific document fields.
- **Cross-Tab Communication**: Managing cross-tab updates in plain IndexedDB is difficult. RxDB handles this seamlessly-changes in one tab automatically affect observed data in others.
- **Missing Advanced Features**: Features like encryption or compression aren't built into IndexedDB, but they are available via RxDB.
- **Limited Platform Support**: IndexedDB exists only in the browser. In contrast, RxDB offers swappable storages to use the same code in [React Native](../react-native-database.md), [Capacitor](../capacitor-database.md), or [Electron](../electron-database.md).
## Set up RxDB in React
Setting up RxDB with React is straightforward. It abstracts IndexedDB complexities and adds a layer of powerful features over it.
### Installing RxDB
First, install RxDB and RxJS from npm:
```bash
npm install rxdb rxjs --save```
```
### Create a Database and Collections
RxDB provides two main storage options:
- The free [localstorage-based storage](../rx-storage-localstorage.md)
- The premium plain [IndexedDB-based storage](../rx-storage-indexeddb.md), offering faster performance
Below is an example of setting up a simple RxDB [database](./react-database.md) using the localstorage-based storage in a React app:
```ts
import { createRxDatabase } from 'rxdb/plugins/core';
import { getRxStorageLocalstorage } from 'rxdb/plugins/storage-localstorage';
// create a database
const db = await createRxDatabase({
name: 'heroesdb', // the name of the database
storage: getRxStorageLocalstorage()
});
// Define your schema
const heroSchema = {
title: 'hero schema',
version: 0,
description: 'Describes a hero in your app',
primaryKey: 'id',
type: 'object',
properties: {
id: {
type: 'string',
maxLength: 100
},
name: {
type: 'string'
},
power: {
type: 'string'
}
},
required: ['id', 'name']
};
// add collections
await db.addCollections({
heroes: {
schema: heroSchema
}
});
```
### CRUD Operations
Once your database is initialized, you can perform all CRUD operations:
```ts
// insert
await db.heroes.insert({ name: 'Iron Man', power: 'Genius-level intellect' });
// bulk insert
await db.heroes.bulkInsert([
{ name: 'Thor', power: 'God of Thunder' },
{ name: 'Hulk', power: 'Superhuman Strength' }
]);
// find and findOne
const heroes = await db.heroes.find().exec();
const ironMan = await db.heroes.findOne({ selector: { name: 'Iron Man' } }).exec();
// update
const doc = await db.heroes.findOne({ selector: { name: 'Hulk' } }).exec();
await doc.update({ $set: { power: 'Unlimited Strength' } });
// delete
const doc = await db.heroes.findOne({ selector: { name: 'Thor' } }).exec();
await doc.remove();
```
## Reactive Queries and Live Updates
RxDB excels in providing reactive data capabilities, ideal for [real-time applications](./realtime-database.md). There are two main approaches to achieving live queries with RxDB: using RxJS Observables with React Hooks or utilizing Preact Signals.
### With RxJS Observables and React Hooks
RxDB integrates seamlessly with RxJS Observables, allowing you to build reactive components. Here's an example of a React component that subscribes to live data updates:
```ts
import { useState, useEffect } from 'react';
function HeroList({ collection }) {
const [heroes, setHeroes] = useState([]);
useEffect(() => {
// create an observable query
const query = collection.find();
const subscription = query.$.subscribe(newHeroes => {
setHeroes(newHeroes);
});
return () => subscription.unsubscribe();
}, [collection]);
return (
Hero List
{heroes.map(hero => (
{hero.name} - {hero.power}
))}
);
}
```
This component subscribes to the collection's changes, updating the UI automatically whenever the underlying data changes, even across browser tabs.
### With Preact Signals
RxDB also supports Preact Signals for reactivity, which can be integrated into React applications. Preact Signals offer a modern, fine-grained reactivity model.
First, install the necessary package:
```bash
npm install @preact/signals-core --save
```
Set up RxDB with Preact Signals reactivity:
```ts
import { PreactSignalsRxReactivityFactory } from 'rxdb/plugins/reactivity-preact-signals';
import { createRxDatabase } from 'rxdb/plugins/core';
import { getRxStorageLocalstorage } from 'rxdb/plugins/storage-localstorage';
const database = await createRxDatabase({
name: 'mydb',
storage: getRxStorageLocalstorage(),
reactivity: PreactSignalsRxReactivityFactory
});
```
Now, you can obtain signals directly from RxDB queries using the double-dollar sign (`$$`):
```ts
function HeroList({ collection }) {
const heroes = collection.find().$$;
return (
{heroes.map(hero => (
{hero.name}
))}
);
}
```
This approach provides automatic updates whenever the data changes, without needing to manage subscriptions manually.
## React IndexedDB Example with RxDB
A comprehensive example of using RxDB within a React application can be found in the [RxDB GitHub repository](https://github.com/pubkey/rxdb/tree/master/examples/react). This repository contains sample applications, showcasing best practices and demonstrating how to integrate RxDB for various use cases.
## Advanced RxDB Features
RxDB offers many advanced features that extend beyond basic data storage:
- **RxDB Replication**: Synchronize local data with remote databases seamlessly. Learn more: [RxDB Replication](https://rxdb.info/replication.html)
- **Data Migration**: Handle schema changes gracefully with automatic data migrations. See: [Data migration](https://rxdb.info/migration-schema.html)
- **Encryption**: Secure your data with built-in encryption capabilities. Explore: [Encryption](https://rxdb.info/encryption.html)
- **Compression**: Optimize storage using key compression. Details: [Compression](https://rxdb.info/key-compression.html)
## Limitations of IndexedDB
While IndexedDB is powerful, it has some inherent limitations:
- **Performance**: IndexedDB can be slow under certain conditions. Read more: [Slow IndexedDB](https://rxdb.info/slow-indexeddb.html)
- **Storage Limits**: Browsers [impose limits](./indexeddb-max-storage-limit.md) on how much data can be stored. See: [Browser storage limits](https://rxdb.info/articles/localstorage-indexeddb-cookies-opfs-sqlite-wasm.html)
## Alternatives to IndexedDB
Depending on your application's requirements, there are [alternative storage solutions](./localstorage-indexeddb-cookies-opfs-sqlite-wasm.md) to consider:
- **Origin Private File System (OPFS)**: A newer API that can offer better performance. RxDB supports OPFS as well. More info: [RxDB OPFS Storage](../rx-storage-opfs.md)
- **SQLite**: Ideal for React applications on Capacitor or [Ionic](./ionic-storage.md), offering native performance. Explore: [RxDB SQLite Storage](../rx-storage-sqlite.md)
## Performance comparison with other browser storages
Here is a [performance overview](../rx-storage-performance.md) of the various browser based storage implementation of RxDB:
## Follow Up
- Learn how to use RxDB with the [RxDB Quickstart](../quickstart.md) for a guided introduction.
- Check out the [RxDB GitHub repository](https://github.com/pubkey/rxdb) and leave a star β if you find it useful.
By leveraging RxDB on top of IndexedDB, you can create highly responsive, offline-capable React applications without dealing with the low-level complexities of IndexedDB directly. With reactive queries, seamless cross-tab communication, and powerful advanced features, RxDB becomes an invaluable tool in modern web development.
---
## React Native Encryption and Encrypted Database/Storage
Data security is a critical concern in modern mobile applications. As React Native continues to grow in popularity for building cross-platform apps, ensuring that your data is protected is paramount. RxDB, a real-time database for JavaScript applications, offers powerful encryption features that can help you secure your React Native app's data.
This article explains why encryption is important, how to set it up with RxDB in [React Native](../react-native-database.md), and best practices to keep your app secure.
## π Why Encryption Matters
Encryption ensures that, even if an unauthorized party obtains physical access to your device or intercepts data, they cannot read the information without the encryption key. Sensitive user data such as credentials, personal information, or financial details should always be encrypted. Proper encryption practices reduce the risk of data breaches and help your application remain compliant with regulations like [GDPR](https://gdpr.eu/) or [HIPAA](https://www.hhs.gov/hipaa/index.html).
## React Native Encryption Overview
React Native supports multiple ways to secure local data:
1. **Encrypted Databases**
Use databases with built-in encryption capabilities, such as SQLite with encryption layers or RxDB with its [encryption plugin](../encryption.md).
2. **Secure Storage Libraries**
For key-value data (like tokens or secrets), you can use libraries like [react-native-keychain](https://github.com/oblador/react-native-keychain) or [react-native-encrypted-storage](https://github.com/emeraldsanto/react-native-encrypted-storage).
3. **Custom Encryption**
If you need more fine-grained control, you can integrate libraries like [`crypto-js`](https://github.com/brix/crypto-js) or the [Web Crypto API](https://developer.mozilla.org/en-US/docs/Web/API/Web_Crypto_API) to encrypt data before storing it in a database or file.
## Setting Up Encryption in RxDB for React Native
### 1. Install RxDB and Required Plugins
Install RxDB and the encryption plugin(s) you need. For the CryptoJS plugin:
```bash
npm install rxdb
npm install crypto-js
```
### 2. Set Up Your RxDB Database with Encryption
RxDB offers two [encryption plugins](../encryption.md):
- **CryptoJS Plugin**: A free and straightforward solution for most basic use cases.
- **Web Crypto Plugin**: A [premium plugin](/premium) that utilizes the native Web Crypto API for better performance and security.
Below is an example showing how to set up RxDB using the CryptoJS plugin. This example uses the [in-memory storage](../rx-storage-memory.md) for testing purposes. In a real production scenario, you would use a persistent storage adapter, mostly the [SQLite-based storage](../rx-storage-sqlite.md).
```js
import { createRxDatabase } from 'rxdb';
import { wrappedKeyEncryptionCryptoJsStorage } from 'rxdb/plugins/encryption-crypto-js';
/*
* For testing, we use the in-memory storage of RxDB.
* In production you would use the persistent SQLite based storage instead.
*/
import { getRxStorageMemory } from 'rxdb/plugins/storage-memory';
async function initEncryptedDatabase() {
// Wrap the normal storage with the encryption plugin
const encryptedMemoryStorage = wrappedKeyEncryptionCryptoJsStorage({
storage: getRxStorageMemory()
});
// Create an encrypted database
const db = await createRxDatabase({
name: 'myEncryptedDatabase',
storage: encryptedMemoryStorage,
password: 'sudoLetMeIn' // Make sure not to hardcode in production
});
// Define a schema and create a collection
await db.addCollections({
secureData: {
schema: {
title: 'secure data schema',
version: 0,
type: 'object',
primaryKey: 'id',
properties: {
id: {
type: 'string',
maxLength: 100
},
normalField: {
type: 'string'
},
secretField: {
type: 'string'
}
},
required: ['id', 'normalField', 'secretField']
}
}
});
return db;
}
```
### 3. Inserting and Querying Encrypted Data
Once you've set up the database with encryption, data in fields specified by your schema will be encrypted automatically before it is stored, and decrypted when queried.
```js
(async () => {
const db = await initEncryptedDatabase();
// Insert encrypted data
const doc = await db.secureData.insert({
id: 'mySecretId',
normalField: 'foobar',
secretField: 'This is top secret data'
});
// Query encrypted data by its primary key or non-encrypted fields
const fetchedDoc = await db.secureData.findOne({
selector: {
normalField: 'foobar'
}
}).exec(true);
console.log(fetchedDoc.secretField); // 'This is top secret data'
// Update data
await fetchedDoc.patch({
secretField: 'Updated secret data'
});
})();
```
**Note**: You can only query directly by non-encrypted fields or primary keys. Encrypted fields cannot be used in queries because they are stored as ciphertext in the database. A common approach is to have a small subset of fields that need to be queried unencrypted while storing any sensitive data in encrypted fields.
## Best Practices for React Native Encryption
- **Secure Password Handling**
- Avoid hardcoding passwords or encryption keys.
- Use secure storage solutions like React Native Keychain or react-native-encrypted-storage to fetch the database password at runtime:
```js
// Example: using react-native-keychain to securely retrieve a stored password
import * as Keychain from 'react-native-keychain';
async function getDatabasePassword() {
const credentials = await Keychain.getGenericPassword();
if (credentials) {
return credentials.password;
}
throw new Error('No password stored in Keychain');
}
```
- **Encrypt Attachments**:
If you need to store files (images, text files, etc.), consider encrypting attachments. RxDB supports attachments that can be encrypted automatically, ensuring your files are protected:
```ts
import { createBlob } from 'rxdb/plugins/core';
const doc = await await db.secureData.findOne({
selector: {
normalField: 'foobar'
}
}).exec(true);
const attachment = await doc.putAttachment({
id: 'encryptedFile.txt',
data: createBlob('Sensitive content', 'text/plain'),
type: 'text/plain',
});
```
- **Optimize Performance**
- If performance is critical, consider using the premium Web Crypto plugin, which leverages native APIs for faster encryption and decryption.
- If big chunks of data are encrypted, store them in attachments instead of document fields. Attachments will only be decrypted on explicit fetches, not during queries.
- **Use DevMode in Development**: RxDB's [DevMode Plugin](../dev-mode.md) can help validate your schema and encryption setup during development. Disable it in production for performance reasons.
- **Secure Communication**:
- Use HTTPS to secure network communication between the app and any backend services.
- If you're synchronizing data to a server, ensure the data is also encrypted in transit. RxDB's [replication plugins](../replication.md) can work with secure endpoints to keep data consistent.
- **SSL Pinning**: Consider SSL Pinning if you want to prevent man-in-the-middle attacks. SSL Pinning ensures the device trusts only the pinned certificate, preventing attackers from swapping out valid certificates with their own.
## Follow Up
- Learn how to use RxDB with the [RxDB Quickstart](../quickstart.md) for a guided introduction.
- A good way to learn using RxDB database with React Native is to check out the [RxDB React Native example](https://github.com/pubkey/rxdb/tree/master/examples/react-native) and use that as a tutorial.
- Check out the [RxDB GitHub repository](https://github.com/pubkey/rxdb) and leave a star β if you find it useful.
- Learn more about the [RxDB encryption plugins](../encryption.md).
By following these best practices and leveraging RxDB's powerful encryption plugins, you can build secure, performant, and robust React Native applications that keep your users' data safe.
---
## ReactJS Storage - From Basic LocalStorage to Advanced Offline Apps with RxDB
# ReactJS Storage β From Basic LocalStorage to Advanced Offline Apps with RxDB
Modern **ReactJS** applications often need to store data on the client side. Whether youβre preserving simple user preferences or building offline-ready features, choosing the right **storage** mechanism can make or break your development experience. In this guide, weβll start with a basic **localStorage** approach for minimal data. Then, weβll explore more powerful, reactive solutions via [RxDB](/)βincluding offline functionality, indexing, `preact signals`, and even encryption.
---
## Part 1: Storing Data in ReactJS with LocalStorage
`localStorage` is a built-in browser API for storing key-value pairs in the userβs browser. Itβs straightforward to set and get itemsβideal for trivial preferences or small usage data.
```jsx
import React, { useState, useEffect } from 'react';
function LocalStorageExample() {
const [username, setUsername] = useState(() => {
const saved = localStorage.getItem('username');
return saved ? JSON.parse(saved) : '';
});
useEffect(() => {
localStorage.setItem('username', JSON.stringify(username));
}, [username]);
return (
ReactJS LocalStorage Demo
setUsername(e.target.value)}
placeholder="Enter your username"
/>
Stored: {username}
);
}
export default LocalStorageExample;
```
**Pros** of localStorage in ReactJS:
- Easy to implement quickly for minimal data
- Built-in to the browserβno extra libs
- Persistent across sessions
**Downsides of localStorage**
While localStorage is convenient for small amounts of data, it has certain limitations:
- Synchronous: Reading or writing localStorage can block the main thread if data is large.
- No advanced queries: You only store stringified objects by a single key. Searching or filtering requires manually scanning everything.
- No concurrency or offline logic: If multiple tabs or users need to manipulate the same data, localStorage doesnβt handle concurrency or sync with a server.
- No indexing: You canβt perform partial lookups or advanced matching.
For βremember user preferenceβ use cases, localStorage is excellent. But if your app grows complexβneeding structured data, large data sets, or offline-first featuresβyou might quickly surpass localStorageβs utility.
## Part 2: LocalStorage vs. IndexedDB
While localStorage is simple, itβs limited to string-based key-value lookups and can be synchronous for all reads/writes. For more robust ReactJS storage needs, browsers also provide IndexedDBβa low-level, asynchronous API that can store larger amounts of JSON data with indexing.
**LocalStorage:**
- Good for small amounts of data (like user settings or flags)
- String-only storage
- Single key-value access, no searching by subfields
**IndexedDB:**
- Stores [large](./indexeddb-max-storage-limit.md) [JSON](./json-database.md) objects, able to index by multiple fields
- Asynchronous and usually more scalable
- More complicated to use directly (i.e., not as simple as .getItem())
[RxDB](/), as youβll see, simplifies [IndexedDB](../rx-storage-indexeddb.md) usage in ReactJS by adding a more intuitive layer for queries, reactivity, and advanced capabilities like [encryption](../encryption.md).
## Part 3: Moving Beyond Basic Storage: RxDB for ReactJS
When data shapes get complexβlarge sets of nested documents, or you want offline sync to a serverβRxDB can transform your approach to ReactJS storage. It stores documents in (usually) IndexedDB or alternative backends but offers a reactive, NoSQL-based interface.
### RxDB Quick Example (Observables)
```ts
import { createRxDatabase } from 'rxdb';
import { getRxStorageLocalstorage } from 'rxdb/plugins/storage-localstorage';
(async function setUpRxDB() {
const db = await createRxDatabase({
name: 'heroDB',
storage: getRxStorageLocalstorage(),
multiInstance: false
});
const heroSchema = {
title: 'hero schema',
version: 0,
type: 'object',
primaryKey: 'id',
properties: {
id: { type: 'string' },
name: { type: 'string' },
power: { type: 'string' }
},
required: ['id', 'name']
};
await db.addCollections({ heroes: { schema: heroSchema } });
// Insert a doc
await db.heroes.insert({ id: '1', name: 'AlphaHero', power: 'Lightning' });
// Query docs once
const allHeroes = await db.heroes.find().exec();
console.log('Heroes: ', allHeroes);
})();
```
Reactive Queries: In a React component, you can subscribe to a query via RxDBβs $ property, letting your UI automatically update when data changes. React components can subscribe to updates from .find() queries, letting the UI automatically reflect changesβperfect for dynamic offline-first apps.
```tsx
import React, { useEffect, useState } from 'react';
function HeroList({ collection }) {
const [heroes, setHeroes] = useState([]);
useEffect(() => {
const query = collection.find();
// query.$ is an RxJS Observable that emits whenever data changes
const sub = query.$.subscribe(newHeroes => {
setHeroes(newHeroes);
});
return () => sub.unsubscribe(); // clean up subscription
}, [collection]);
return (
{heroes.map(hero => (
{hero.name} - Power: {hero.power}
))}
);
}
export default HeroList;
```
By using these reactive queries, your React app knows exactly when data changes locally (or from another browser tab) or from remote sync, keeping your UI in sync effortlessly.
## Part 4: Using Preact Signals Instead of Observables
RxDB typically exposes reactivity via RxJS observables. However, some developers prefer newer reactivity approaches like Preact Signals. RxDB supports them via a special plugin or advanced usage:
```ts
import { createRxDatabase } from 'rxdb/plugins/core';
import { getRxStorageLocalstorage } from 'rxdb/plugins/storage-localstorage';
import { PreactSignalsRxReactivityFactory } from 'rxdb/plugins/reactivity-preact-signals';
(async function setUpRxDBWithSignals() {
const db = await createRxDatabase({
name: 'heroDB_signals',
storage: getRxStorageLocalstorage(),
reactivity: PreactSignalsRxReactivityFactory
});
// Create a signal-based query instead of using Observables:
const collection = db.heroes;
const heroesSignal = collection.find().$$; // signals version
// Now you can reference heroesSignal() in Preact or React with adapter usage
})();
```
Preact Signals rely on `signals` instead of `Observables`. Some developers find them more straightforward to adopt, especially for fine-grained reactivity. In ReactJS, you might still prefer RxJS-based subscriptions unless you add bridging code for signals.
## Part 5: Encrypting the Storage with RxDB
For more advanced ReactJS storage needsβespecially when sensitive user data is involved - you might want to encrypt stored documents at rest. RxDB provides a robust [encryption plugin](../encryption.md):
```ts
import { createRxDatabase } from 'rxdb';
import { wrappedKeyEncryptionCryptoJsStorage } from 'rxdb/plugins/encryption-crypto-js';
import { getRxStorageLocalstorage } from 'rxdb/plugins/storage-localstorage';
(async function secureSetup() {
const encryptedStorage = wrappedKeyEncryptionCryptoJsStorage({
storage: getRxStorageLocalstorage()
});
// Provide a password for encryption
const db = await createRxDatabase({
name: 'secureReactStorage',
storage: encryptedStorage,
password: 'MyStrongPassword123'
});
await db.addCollections({
secrets: {
schema: {
title: 'secret schema',
version: 0,
type: 'object',
primaryKey: 'id',
properties: {
id: { type: 'string' },
secretInfo: { type: 'string' }
},
required: ['id'],
encrypted: ['secretInfo'] // field to encrypt
}
}
});
})();
```
All data in the marked `encrypted` fields is automatically encrypted at rest. This is crucial if you store user credentials, private messages, or other personal data in your ReactJS application storage.
## Offline Sync
If you need multi-device or multi-user data synchronization, RxDB provides [replication plugins](../replication.md) for various endpoints (HTTP, [GraphQL](../replication-graphql.md), [CouchDB](../replication-couchdb.md), [Firestore](../replication-firestore.md), etc.). Your [local offline](../offline-first.md) changes can then merge automatically with a remote database whenever internet connectivity is restored.
## Overview: [localStorage vs IndexedDB vs RxDB](./localstorage-indexeddb-cookies-opfs-sqlite-wasm.md)
| **Characteristic** | **localStorage** | **IndexedDB** | **RxDB** |
|--------------------------|---------------------------------------------------------------------|----------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------|
| **Data Model** | Key-value store (only strings) | Low-level, JSON-like storage engine with object stores and indexes | NoSQL JSON documents with optional JSON-Schema |
| **Query Capabilities** | Basic get/set by key; manual parse for more complex searches | Index-based queries, but API is fairly verbose; lacks a high-level query language | JSON-based queries, optional indexes, real-time reactivity |
| **Observability** | None. Must re-fetch data yourself. | None natively. Must implement eventing or manual re-check. | Built-in reactivity. UI auto-updates via Observables or Preact signals |
| **Large Data Usage** | Not recommended for large data (blocking, synchronous calls) | Better for large amounts of data, asynchronous reads/writes | Scales for medium to large data. Uses IndexedDB or other storages under the hood |
| **Concurrency** | Minimal. Overwrites if multiple tabs write simultaneously | Multiple tabs can open the same DB, but must handle concurrency logic carefully | Multi-instance concurrency with built-in conflict resolution plugins if needed |
| **Offline Sync** | None. Purely local. | None out of the box. Must be implemented manually | Built-in replication to remote endpoints (HTTP, GraphQL, CouchDB, etc.) for offline-first usage |
| **Encryption** | Not supported natively | Not supported natively; must encrypt data manually before storing | Encryption plugins available. Supports field-level encryption at rest |
| **Usage** | Great for small flags or settings
## Follow Up
If youβre looking to dive deeper into **ReactJS storage** topics and take full advantage of RxDBβs offline-first, real-time capabilities, here are some recommended resources:
- **[RxDB Official Documentation](../overview.md)**
Explore detailed guides on setting up storage adapters, defining [JSON schemas](../rx-schema.md), [handling conflicts](../transactions-conflicts-revisions.md), and enabling [offline synchronization](../replication.md).
- **[RxDB Quickstart](https://rxdb.info/quickstart.html)**
Get a step-by-step tutorial to create your first RxDB-based application in minutes.
- **[RxDB GitHub Repository](https://github.com/pubkey/rxdb)**
See the source code, open issues, and browse community-driven examples that integrate ReactJS, Preact Signals, and advanced features like encryption.
- **[RxDB Encryption Plugins](https://rxdb.info/encryption.html)**
Learn how to encrypt fields in your collections, protecting user data and meeting compliance requirements.
- **[Preact Signals React Integration (Example)](https://github.com/preactjs/signals#react)**
If you want to combine React with signals-based reactivity, check out example code and bridging approaches.
- **[MDN: Using the Web Storage API](https://developer.mozilla.org/en-US/docs/Web/API/Web_Storage_API)**
Refresh on localStorage basics, including best practices for small key-value data in traditional React apps.
With these follow-up steps, you can refine your **reactjs storage** strategy to meet your appβs unique needs, whether itβs simple user preferences or robust offline data syncing.
---
## What Really Is a Realtime Database?
# What is a realtime database?
I have been building [RxDB](https://rxdb.info/), a [NoSQL](./in-memory-nosql-database.md) **realtime** JavaScript database for many years.
Often people get confused by the word **realtime database**, because the word **realtime** is so vaguely defined that it can mean everything and nothing at the same time.
In this article we will explore what a realtime database is, and more important, what it is not.
## Realtime as in **realtime computing**
When "normal" developers hear the word "realtime", they think of **Real-time computing (RTC)**. Real-time computing is a type of computer processing that **guarantees specific response times** for tasks or events, crucial in applications like industrial control, automotive systems, and aerospace. It relies on specialized operating systems (RTOS) to ensure predictability and low latency. Hard real-time systems must never miss deadlines, while soft real-time systems can tolerate occasional delays. Real-time responses are often understood to be in the order of milliseconds, and sometimes microseconds.
Consider the role of real-time computing in car airbags: sensors detect collision force, swiftly process the data, and immediately decide to deploy the airbags within milliseconds. Such rapid action is imperative for safeguarding passengers. Hence, the controlling chip must **guarantee a certain response time** - it must operate in "realtime".
But when people talk about **realtime databases**, especially in the web-development world, they almost never mean realtime, as in **realtime computing**, they mean something else.
In fact, with any programming language that run on end users devices, it is not even possible to built a "real" realtime database. A program, like a JavaScript ([browser](./browser-database.md) or [Node.js](../nodejs-database.md)) process, can be halted by the operating systems task manager at any time and therefore it will never be able to guarantee specific response times. To build a realtime computing database, you would need a realtime capable operating system.
## Real time Database as in **realtime replication**
When talking about realtime databases, most people refer to realtime, as in realtime replication.
Often they mean a very specific product which is the **Firebase Realtime Database** (not the [Firestore](../replication-firestore.md)).
In the context of the Firebase Realtime Database, "realtime" means that data changes are synchronized and delivered to all connected clients or devices as soon as they occur, typically within milliseconds. This means that when any client updates, adds, or removes data in the database, all other clients that are connected to the same database instance receive those updates instantly, without the need for manual polling or frequent HTTP requests.
In short, when replicating data between databases, instead of polling, we use a [websocket connection](./websockets-sse-polling-webrtc-webtransport.md) to live-stream all changes between the server and the clients, this is labeled as "realtime database". A similar thing can be done with RxDB and the [RxDB Replication Plugins](../replication.md).
## Realtime as in **realtime applications**
In the context of realtime client-side applications, "realtime" refers to the immediate or near-instantaneous processing and response to events or data inputs. When data changes, the application must directly update to reflect the new data state, without any user interaction or delay. Notice that the change to the data could have come from any source, like a user action, an operation in another browser tab, or even an operation from another device that has been replicated to the client.
In contrast to push-pull based databases (e.g., MySQL or MongoDB servers), a realtime database contains **features which make it easy to build realtime applications**. For example with RxDB you can not only fetch query results once, but instead you can subscribe to a query and directly update the HTML dom tree whenever the query has a new result set:
```ts
await db.heroes.find({
selector: {
healthpoints: {
$gt: 0
}
}
})
.$ // The $ returns an observable that emits whenever the query's result set changes.
.subscribe(aliveHeroes => {
// Refresh the HTML list each time there are new query results.
const newContent = aliveHeroes.map(doc => '' + doc.name + '');
document.getElementById('#myList').innerHTML = newContent;
});
// You can even subscribe to any RxDB document's fields.
myDocument.firstName$.subscribe(newName => console.log('name is: ' + newName));
```
A competent realtime application is engineered to offer feedback or results swiftly, ideally within milliseconds to microseconds. Ideally, a data modification should be processed in under **16 milliseconds** (since 1 second divided by 60 frames equals 16.66ms) to ensure users don't perceive any lag from input to visualization. RxDB utilizes the [EventReduce algorithm](https://github.com/pubkey/event-reduce) to manage changes more swiftly than 16ms. However, it can never assure fixed response times as a "realtime computing database" would.
## Follow Up
- Dive into the [RxDB Quickstart](https://rxdb.info/quickstart.html)
- Discover more about the [RxDB realtime Sync Engine](../replication.md)
- Join the conversation at [RxDB Chat](https://rxdb.info/chat/)
---
## RxDB as a Database in a Vue.js Application
# RxDB as a Database in a Vue Application
In the modern web ecosystem, [Vue](https://vuejs.org/) has become a leading choice for building highly performant, reactive single-page applications (SPAs). However, while Vue excels at managing and updating the user interface, robust and efficient data handling also plays a pivotal role in delivering a great user experience. Enter [RxDB](https://rxdb.info/), a reactive JavaScript database that runs in the browser (and beyond), offering significant capabilities such as offline-first data handling, real-time synchronization, and straightforward integration with Vue's reactivity system.
This article explores how RxDB works, why it's a perfect match for Vue, and how you can leverage it to build more engaging, performant, and data-resilient Vue applications.
## Why Vue Applications Need a Database
Vue is renowned for its lightweight core and flexible architecture centered around reactive state management and reusable components. However, modern Vue applications often require:
- **Offline Capabilities:** Allowing users to continue working even without internet access.
- **Real-Time Updates:** Keeping UI data in sync with changes as they occur, whether locally or from other connected clients.
- **Improved Performance:** Reducing server round trips and leveraging local storage for faster data operations.
- **Scalable Data Handling:** Managing increasingly large datasets or complex queries right in the browser.
While you can store data in Vuex/Pinia stores or via direct AJAX calls, these solutions may not suffice when your application demands a full-featured offline-first database or complex synchronization with a server. RxDB addresses these needs with a dedicated, reactive, browser-based database that pairs seamlessly with Vue's reactivity system.
## Introducing RxDB as a Database Solution
RxDB - short for Reactive Database - is built on the principle of combining [NoSQL database](./in-memory-nosql-database.md) capabilities with reactive programming. It runs inside your client-side environment (browser, [Node.js](../nodejs-database.md), or [mobile devices](./mobile-database.md)) and provides:
1. **Real-Time Reactivity**: Automatically updates subscribed components whenever data changes.
2. **Offline-First Approach**: Stores data locally and syncs with the server when online connectivity is restored.
3. **Data Replication**: Effortlessly keeps data synchronized across multiple tabs, devices, or server instances.
4. **Multi-Tab Support**: Seamlessly propagates changes to all open tabs in the user's [browser](./browser-database.md).
5. **Observable Queries**: Automatically refresh the result set when documents in your queried collection change.
### RxDB vs. Other Vue Database Options
Compared to traditional approaches - like raw IndexedDB or local storage - RxDB adds a powerful, reactive layer that simplifies your data flow. While tools like Vuex or Pinia are great for state management, they are not fully fledged databases with features like replication, conflict resolution, and offline persistence. RxDB bridges the gap by providing an integrated data handling solution tailor-made for modern, data-intensive Vue applications.
## Getting Started with RxDB
Let's break down the essentials for using RxDB within a Vue application.
### Installation
You can install RxDB (and RxJS, which it depends on) via npm or yarn:
```bash
npm install rxdb rxjs
```
## Creating and Configuring Your Database
Within your Vue project, you can set up an RxDB instance in a dedicated file or a Vue plugin. Below is an example using [Localstorage](./localstorage.md) as the storage engine:
```ts
// db.js
import { createRxDatabase } from 'rxdb';
import { getRxStorageLocalstorage } from 'rxdb/plugins/storage-localstorage';
export async function initDatabase() {
const db = await createRxDatabase({
name: 'heroesdb',
storage: getRxStorageLocalstorage(),
password: 'myPassword', // optional encryption password
multiInstance: true, // multi-tab support
eventReduce: true // optimize event handling
});
await db.addCollections({
hero: {
schema: {
title: 'hero schema',
version: 0,
primaryKey: 'id',
type: 'object',
properties: {
id: { type: 'string' },
name: { type: 'string' },
healthpoints: { type: 'number' }
}
}
}
});
return db;
}
```
After creating the RxDB instance, you can share it across your application (for example, by providing it in a plugin or a global property in Vue).
## Vue Reactivity and RxDB Observables
RxDB queries return RxJS observables (`.$`). Vue can automatically update components when data changes if you manually subscribe and store results in Vue refs/reactive objects, or if you use RxDB's [custom reactivity for Vue](../reactivity.md).
**Example with Vue 3 Composition API:**
```js
// HeroList.vue
{{ hero.name }} - HP: {{ hero.healthpoints }}
```
## Different RxStorage Layers for RxDB
RxDB supports multiple storage backends - called "RxStorage layers" - giving you flexibility in how data is persisted:
- [LocalStorage RxStorage](../rx-storage-localstorage.md): Uses the browsers localstorage API.
- [IndexedDB RxStorage](../rx-storage-indexeddb.md): Direct usage of native IndexedDB.
- [OPFS RxStorage](../rx-storage-opfs.md): Uses the File System Access API for even faster storage in modern browsers.
- [Memory RxStorage](../rx-storage-memory.md): Stores data in memory, ideal for tests or ephemeral data.
- [SQLite RxStorage](../rx-storage-sqlite.md): Runs SQLite, which can be compiled to WebAssembly for the browser. While possible, it typically carries a larger bundle size compared to native browser APIs like [IndexedDB or OPFS](./localstorage-indexeddb-cookies-opfs-sqlite-wasm.md).
Choose the storage option that best aligns with your Vue application's requirements for performance, persistence, and platform compatibility.
## Synchronizing Data with RxDB between Clients and Servers
RxDB champions an offline-first approach: data is kept locally so that your Vue app remains usable, even without internet. When connectivity is restored, RxDB ensures your local changes synchronize to the server, resolving conflicts as necessary.
- [Real-Time Synchronization](./realtime-database.md): With RxDB's replication plugins, any local change can be instantly pushed to a remote endpoint while pulling down remote changes to ensure consistency.
- **Conflict Resolution**: In multi-user scenarios, conflicts may arise if two clients update the same document simultaneously. RxDB provides hooks to handle and resolve these gracefully.
- **Scalable Architecture**: By reducing reliance on continuous server requests, you can lighten server load and deliver a more responsive user experience.
## Advanced RxDB Features and Techniques
### Offline-First Approach
Vue applications can seamlessly function offline by leveraging RxDB's local database storage. The moment the network is restored, all unsynced data is pushed to the server. This capability is particularly beneficial for Progressive Web Apps (PWAs) and scenarios with spotty connectivity.
### Observable Queries and Change Streams
Beyond simply returning data, RxDB queries emit observables that respond to any change in the underlying documents. This real-time approach can drastically simplify state management, since updates flow directly into your Vue components without additional manual wiring.
### Encryption of Local Data
For applications handling sensitive information, RxDB supports [encryption](../encryption.md) of local data. Your data is stored securely in the browser, protecting it from unauthorized access.
### Indexing and Performance Optimization
By [defining indexes](../rx-schema.md) on frequently searched fields, you can speed up queries and reduce overall resource usage. This is crucial for larger datasets where performance might otherwise degrade.
### JSON Key Compression
This [optimization](../key-compression.md) shortens field names in stored JSON documents, thereby reducing storage space and potentially improving performance for read/write operations.
### Multi-Tab Support
If your users open multiple tabs of your Vue application, RxDB ensures data is synchronized across all instances in real time. Changes made in one tab are immediately reflected in others, creating a unified user experience.
## Best Practices for Using RxDB in Vue
Here are some recommendations to get the most out of RxDB in your Vue projects:
- Centralize Database Creation: Initialize and configure RxDB in a dedicated file or plugin, ensuring only one database instance is created.
- Leverage Vue's Composition API or a Global Store: Use watchers, refs, or a store like Pinia to neatly manage data subscriptions and updates, preventing scattered subscription logic.
- Async Subscriptions: Prefer using Vue's lifecycle hooks and the Composition API to manage subscriptions. Clean up subscriptions when components unmount or no longer need the data.
- Optimize Queries and Indexes: Only query the data you need, and define indexes to speed up lookups.
- Test [Offline Scenarios](../offline-first.md): Make sure your offline logic works as expected by simulating network disconnections and reconnections.
- [Plan Conflict Resolution](../transactions-conflicts-revisions.md): For multi-user apps, decide how to merge concurrent changes to prevent data inconsistencies.
## Follow Up
To explore more about RxDB and leverage its capabilities for browser database development, check out the following resources:
- [RxDB GitHub Repository](/code/): Visit the official GitHub repository of RxDB to access the source code, documentation, and community support.
- [RxDB Quickstart](../quickstart.md): Get started quickly with RxDB by following the provided quickstart guide, which offers step-by-step instructions for setting up and using RxDB in your projects.
- [RxDB Reactivity for Vue](../reactivity.md): Discover how RxDB observables can directly produce Vue refs, simplifying integration with your Vue components.
- [RxDB Vue Example at GitHub](https://github.com/pubkey/rxdb/tree/master/examples/vue): Explore an official Vue example to see RxDB in action within a Vue application.
- [RxDB Examples](https://github.com/pubkey/rxdb/tree/master/examples): Browse even more official examples to learn best practices you can apply to your own projects.
---
## IndexedDB Database in Vue Apps - The Power of RxDB
Building robust, [offline-capable](../offline-first.md) Vue applications often involves leveraging browser storage solutions to manage data. IndexedDB is one such powerful tool, but its raw API can be challenging to work with directly. RxDB abstracts away much of IndexedDB's complexity, providing a more developer-friendly experience. In this article, we'll explore what IndexedDB is, why it's beneficial in Vue applications, the challenges of using plain IndexedDB, and how [RxDB](https://rxdb.info/) can simplify your development process while adding advanced features.
## What is IndexedDB?
[IndexedDB](https://developer.mozilla.org/en-US/docs/Web/API/IndexedDB_API) is a low-level API for storing significant amounts of structured data in the browser. It provides a transactional database system that can store key-value pairs, complex objects, and more. This storage engine is asynchronous and supports advanced data types, making it suitable for offline storage and complex web applications.
## Why Use IndexedDB in Vue
When building Vue applications, IndexedDB can play a crucial role in enhancing both performance and user experience. Here are some reasons to consider using IndexedDB:
- **Offline-First / Local-First**: By storing data locally, your application remains functional even without an internet connection.
- **Performance**: Using local data means [zero latency](./zero-latency-local-first.md) and no loading spinners, as data doesn't need to be fetched over a network.
- **Easier Implementation**: Replicating all data to the client once is often simpler than implementing multiple endpoints for each user interaction.
- **Scalability**: Local data reduces server load because queries run on the client side, decreasing server bandwidth and processing requirements.
## Why To Not Use Plain IndexedDB
While IndexedDB itself is powerful, its native API comes with several drawbacks for everyday application developers:
- **Callback-Based API**: IndexedDB was originally designed around callbacks rather than modern Promises, making asynchronous code more cumbersome.
- **Complexity**: IndexedDB is low-level, intended for library developers rather than for app developers who just want to store and query data easily.
- **Basic Query API**: Its rudimentary query capabilities limit how you can efficiently perform complex queries. Libraries like RxDB offer more advanced querying and indexing.
- **TypeScript Support**: Ensuring good [TypeScript support](../tutorials/typescript.md) with IndexedDB is challenging, especially when trying to maintain schema consistency.
- **Lack of Observable API**: IndexedDB doesn't provide an observable API out of the box, making it hard to automatically update your Vue app in real time. RxDB solves this by enabling you to [observe queries](../rx-query.md#observe) or specific documents.
- **Cross-Tab Communication**: Managing cross-tab updates in plain IndexedDB is difficult. RxDB handles this seamlessly - changes in one tab automatically affect observed data in others.
- **Missing Advanced Features**: Features like [encryption](../encryption.md) or [compression](../key-compression.md) aren't built into IndexedDB, but they are available via RxDB.
- **Limited Platform Support**: IndexedDB is browser-only. RxDB offers [swappable storages](../rx-storage.md) so you can reuse the same data layer code in mobile or desktop environments.
## Set up RxDB in Vue
Setting up RxDB with Vue is straightforward. It abstracts IndexedDB complexities and adds a layer of powerful features over it.
### Installing RxDB
First, install RxDB (and RxJS) from npm:
```bash
npm install rxdb rxjs --save
```
### Create a Database and Collections
RxDB provides two main storage options:
- The free [LocalStorage-based storage](../rx-storage-localstorage.md)
- The premium plain [IndexedDB-based storage](../rx-storage-indexeddb.md), offering faster [performance](../rx-storage-performance.md)
Below is an example of setting up a simple RxDB database using the localstorage-based storage in a Vue app:
```ts
// db.ts
import { createRxDatabase } from 'rxdb';
import { getRxStorageLocalstorage } from 'rxdb/plugins/storage-localstorage';
export async function initDB() {
const db = await createRxDatabase({
name: 'heroesdb', // the name of the database
storage: getRxStorageLocalstorage()
});
// Define your schema
const heroSchema = {
title: 'hero schema',
version: 0,
description: 'Describes a hero in your app',
primaryKey: 'id',
type: 'object',
properties: {
id: {
type: 'string',
maxLength: 100
},
name: {
type: 'string'
},
power: {
type: 'string'
}
},
required: ['id', 'name']
};
// add collections
await db.addCollections({
heroes: {
schema: heroSchema
}
});
return db;
}
```
### CRUD Operations
Once your database is initialized, you can perform all CRUD operations:
```ts
// insert
await db.heroes.insert({ id: '1', name: 'Iron Man', power: 'Genius-level intellect' });
// bulk insert
await db.heroes.bulkInsert([
{ id: '2', name: 'Thor', power: 'God of Thunder' },
{ id: '3', name: 'Hulk', power: 'Superhuman Strength' }
]);
// find and findOne
const heroes = await db.heroes.find().exec();
const ironMan = await db.heroes.findOne({ selector: { name: 'Iron Man' } }).exec();
// update
const doc = await db.heroes.findOne({ selector: { name: 'Hulk' } }).exec();
await doc.update({ $set: { power: 'Unlimited Strength' } });
// delete
const thorDoc = await db.heroes.findOne({ selector: { name: 'Thor' } }).exec();
await thorDoc.remove();
```
## Reactive Queries and Live Updates
RxDB excels in providing reactive data capabilities, ideal for [real-time applications](./realtime-database.md). Subscribing to queries automatically updates your Vue components when underlying data changes - even across [browser](./browser-database.md) tabs.
### Using RxJS Observables with Vue 3 Composition API
Here's an example of a Vue component that subscribes to live data updates:
```html
Hero List
{{ hero.name }} - {{ hero.power }}
```
This component subscribes to the collection's changes, [updating the UI](./optimistic-ui.md) automatically whenever the underlying data changes in any browser tab.
### Using Vue Signals
If you're exploring Vue's reactivity transforms or signals, RxDB also offers [custom reactivity factories](../reactivity.md) ([premium plugins](/premium/) are required). This allows queries to emit data as signals instead of traditional Observables.
```ts
const heroesSignal = db.heroes.find().$$; // $$ indicates a reactive result
```
With this, in your Vue template or script, you can directly read from heroesSignal()
```html
Hero List
{{ hero.name }} - {{ hero.power }}
```
## Vue IndexedDB Example with RxDB
A comprehensive example of using RxDB within a Vue application can be found in the [RxDB GitHub repository](https://github.com/pubkey/rxdb/tree/master/examples/vue). This repository contains sample applications, showcasing best practices and demonstrating how to integrate RxDB for various use cases.
## Advanced RxDB Features
RxDB offers many advanced features that extend beyond basic data storage:
- [RxDB Replication](../replication.md): Synchronize local data with remote databases seamlessly.
- [Data Migration](../migration-schema.md): Handle schema changes gracefully with automatic data migrations.
- [Encryption](../encryption.md): Secure your data with built-in encryption capabilities.
- [Compression](../key-compression.md): Optimize storage using key compression.
## Limitations of IndexedDB
While IndexedDB is powerful, it has some inherent limitations:
- Performance: IndexedDB can be slow under certain conditions. Read more: [Slow IndexedDB](../slow-indexeddb.md)
- [Storage Limits](./indexeddb-max-storage-limit.md): Browsers impose limits on how much data can be stored. See: [Browser storage limits](./localstorage-indexeddb-cookies-opfs-sqlite-wasm.md#storage-size-limits).
## Alternatives to IndexedDB
Depending on your application's requirements, there are [alternative storage solutions](../rx-storage.md) to consider:
- **Origin Private File System (OPFS)**: A newer API that can offer better performance. RxDB supports OPFS as well. More info: [RxDB OPFS Storage](../rx-storage-opfs.md)
- **SQLite**: Ideal for hybrid frameworks or Capacitor, offering native performance. Explore: [RxDB SQLite Storage](../rx-storage-sqlite.md)
## Performance Comparison with Other Browser Storages
Here is a performance overview of the various browser-based storage implementations of RxDB:
## Follow Up
- Learn how to use RxDB with the [RxDB Quickstart](../quickstart.md) for a guided introduction.
- Check out the [RxDB GitHub repository](/code/) and leave a star β if you find it useful.
By leveraging RxDB on top of IndexedDB, you can create highly responsive, offline-capable Vue applications without dealing with the low-level complexities of IndexedDB directly. With [reactive queries](../rx-query.md), seamless cross-tab communication, and powerful advanced features, RxDB becomes an invaluable tool in modern web development.
---
## WebSockets vs Server-Sent-Events vs Long-Polling vs WebRTC vs WebTransport
For modern real-time web applications, the ability to send events from the server to the client is indispensable. This necessity has led to the development of several methods over the years, each with its own set of advantages and drawbacks. Initially, [long-polling](#what-is-long-polling) was the only option available. It was then succeeded by [WebSockets](#what-are-websockets), which offered a more robust solution for bidirectional communication. Following WebSockets, [Server-Sent Events (SSE)](#what-are-server-sent-events) provided a simpler method for one-way communication from server to client. Looking ahead, the [WebTransport](#what-is-the-webtransport-api) protocol promises to revolutionize this landscape even further by providing a more efficient, flexible, and scalable approach. For niche use cases, [WebRTC](#what-is-webrtc) might also be considered for server-client events.
This article aims to delve into these technologies, comparing their performance, highlighting their benefits and limitations, and offering recommendations for various use cases to help developers make informed decisions when building real-time web applications. It is a condensed summary of my gathered experience when I implemented the [RxDB Sync Engine](../replication.md) to be compatible with various backend technologies.
### What is Long Polling?
Long polling was the first "hack" to enable a server-client messaging method that can be used in browsers over HTTP. The technique emulates server push communications with normal XHR requests. Unlike traditional polling, where the client repeatedly requests data from the server at regular intervals, long polling establishes a connection to the server that remains open until new data is available. Once the server has new information, it sends the response to the client, and the connection is closed. Immediately after receiving the server's response, the client initiates a new request, and the process repeats. This method allows for more immediate data updates and reduces unnecessary network traffic and server load. However, it can still introduce delays in communication and is less efficient than other real-time technologies like WebSockets.
```js
// long-polling in a JavaScript client
function longPoll() {
fetch('http://example.com/poll')
.then(response => response.json())
.then(data => {
console.log("Received data:", data);
longPoll(); // Immediately establish a new long polling request
})
.catch(error => {
/**
* Errors can appear in normal conditions when a
* connection timeout is reached or when the client goes offline.
* On errors we just restart the polling after some delay.
*/
setTimeout(longPoll, 10000);
});
}
longPoll(); // Initiate the long polling
```
Implementing long-polling on the client side is pretty simple, as shown in the code above. However on the backend there can be multiple difficulties to ensure the client receives all events and does not miss out updates when the client is currently reconnecting.
### What are WebSockets?
[WebSockets](https://developer.mozilla.org/en-US/docs/Web/API/WebSocket?retiredLocale=de) provide a full-duplex communication channel over a single, long-lived connection between the client and server. This technology enables browsers and servers to exchange data without the overhead of HTTP request-response cycles, facilitating real-time data transfer for applications like live chat, gaming, or financial trading platforms. WebSockets represent a significant advancement over traditional HTTP by allowing both parties to send data independently once the connection is established, making it ideal for scenarios that require low latency and high-frequency updates.
```js
// WebSocket in a JavaScript client
const socket = new WebSocket('ws://example.com');
socket.onopen = function(event) {
console.log('Connection established');
// Sending a message to the server
socket.send('Hello Server!');
};
socket.onmessage = function(event) {
console.log('Message from server:', event.data);
};
```
While the basics of the WebSocket API are easy to use it has shown to be rather complex in production. A socket can loose connection and must be re-created accordingly. Especially detecting if a connection is still usable or not, can be very tricky. Mostly you would add a [ping-and-pong](https://developer.mozilla.org/en-US/docs/Web/API/WebSockets_API/Writing_WebSocket_servers#pings_and_pongs_the_heartbeat_of_websockets) heartbeat to ensure that the open connection is not closed.
This complexity is why most people use a library on top of WebSockets like [Socket.IO](https://socket.io/) which handles all these cases and even provides fallbacks to long-polling if required.
### What are Server-Sent-Events?
Server-Sent Events (SSE) provide a standard way to push server updates to the client over HTTP. Unlike WebSockets, SSEs are designed exclusively for one-way communication from server to client, making them ideal for scenarios like live news feeds, sports scores, or any situation where the client needs to be updated in real time without sending data to the server.
You can think of Server-Sent-Events as a single HTTP request where the backend does not send the whole body at once, but instead keeps the connection open and trickles the answer by sending a single line each time an event has to be send to the client.
Creating a connection for receiving events with SSE is straightforward. On the client side in a browser, you initialize an [EventSource](https://developer.mozilla.org/en-US/docs/Web/API/EventSource) instance with the URL of the server-side script that generates the events.
Listening for messages involves attaching event handlers directly to the EventSource instance. The API distinguishes between generic message events and named events, allowing for more structured communication. Here's how you can set it up in JavaScript:
```js
// Connecting to the server-side event stream
const evtSource = new EventSource("https://example.com/events");
// Handling generic message events
evtSource.onmessage = event => {
console.log('got message: ' + event.data);
};
```
In difference to WebSockets, an EventSource will automatically reconnect on connection loss.
On the server side, your script must set the `Content-Type` header to `text/event-stream` and format each message according to the [SSE specification](https://www.w3.org/TR/2012/WD-eventsource-20120426/). This includes specifying event types, data payloads, and optional fields like event ID and retry timing.
Here's how you can set up a simple SSE endpoint in a Node.js Express app:
```ts
import express from 'express';
const app = express();
const PORT = process.env.PORT || 3000;
app.get('/events', (req, res) => {
res.writeHead(200, {
'Content-Type': 'text/event-stream',
'Cache-Control': 'no-cache',
'Connection': 'keep-alive',
});
const sendEvent = (data) => {
// all message lines must be prefixed with 'data: '
const formattedData = `data: ${JSON.stringify(data)}\n\n`;
res.write(formattedData);
};
// Send an event every 2 seconds
const intervalId = setInterval(() => {
const message = {
time: new Date().toTimeString(),
message: 'Hello from the server!',
};
sendEvent(message);
}, 2000);
// Clean up when the connection is closed
req.on('close', () => {
clearInterval(intervalId);
res.end();
});
});
app.listen(PORT, () => console.log(`Server running on http://localhost:${PORT}`));
```
### What is the WebTransport API?
WebTransport is a cutting-edge API designed for efficient, low-latency communication between web clients and servers. It leverages the [HTTP/3 QUIC protocol](https://en.wikipedia.org/wiki/HTTP/3) to enable a variety of data transfer capabilities, such as sending data over multiple streams, in both reliable and unreliable manners, and even allowing data to be sent out of order. This makes WebTransport a powerful tool for applications requiring high-performance networking, such as real-time gaming, live streaming, and collaborative platforms. However, it's important to note that WebTransport is currently a working draft and has not yet achieved widespread adoption.
As of now (March 2024), WebTransport is in a [Working Draft](https://w3c.github.io/webtransport/) and not widely supported. You cannot yet use WebTransport in the [Safari browser](https://caniuse.com/webtransport) and there is also no native support [in Node.js](https://github.com/w3c/webtransport/issues/511). This limits its usability across different platforms and environments.
Even when WebTransport will become widely supported, its API is very complex to use and likely it would be something where people build libraries on top of WebTransport, not using it directly in an application's sourcecode.
### What is WebRTC?
[WebRTC](https://webrtc.org/) (Web Real-Time Communication) is an open-source project and API standard that enables real-time communication (RTC) capabilities directly within web browsers and mobile applications without the need for complex server infrastructure or the installation of additional plugins. It supports peer-to-peer connections for streaming audio, video, and data exchange between browsers. [WebRTC](../replication-webrtc.md) is designed to work through NATs and firewalls, utilizing protocols like ICE, STUN, and TURN to establish a connection between peers.
While WebRTC is made to be used for client-client interactions, it could also be leveraged for server-client communication where the server just simulated being also a client. This approach only makes sense for niche use cases which is why in the following WebRTC will be ignored as an option.
The problem is that for WebRTC to work, you need a signaling-server anyway which would then again run over websockets, SSE or WebTransport. This defeats the purpose of using WebRTC as a replacement for these technologies.
## Limitations of the technologies
### Sending Data in both directions
Only WebSockets and WebTransport allow to send data in both directions so that you can receive server-data and send client-data over the same connection.
While it would also be possible with **Long-Polling** in theory, it is not recommended because sending "new" data to an existing long-polling connection would require
to do an additional http-request anyway. So instead of doing that you can send data directly from the client to the server with an additional http-request without interrupting
the long-polling connection.
**Server-Sent-Events** do not support sending any additional data to the server. You can only do the initial request, and even there you cannot send POST-like data in the http-body by default with the native [EventSource API](https://developer.mozilla.org/en-US/docs/Web/API/EventSource). Instead you have to put all data inside of the url parameters which is considered a bad practice for security because credentials might leak into server logs, proxies and caches. To fix this problem, [RxDB](https://rxdb.info/) for example uses the [eventsource polyfill](https://github.com/EventSource/eventsource) instead of the native `EventSource API`. This library adds additional functionality like sending **custom http headers**. Also there is [this library](https://github.com/Azure/fetch-event-source) from microsoft which allows to send body data and use `POST` requests instead of `GET`.
### 6-Requests per Domain Limit
Most modern browsers allow six connections per domain () which limits the usability of all steady server-to-client messaging methods. The limitation of six connections is even shared across browser tabs so when you open the same page in multiple tabs, they would have to shared the six-connection-pool with each other. This limitation is part of the HTTP/1.1-RFC (which even defines a lower number of only two connections).
> Quote From [RFC 2616 - Section 8.1.4](https://www.w3.org/Protocols/rfc2616/rfc2616-sec8.html#sec8.1.4): "Clients that use persistent connections SHOULD limit the number of simultaneous connections that they maintain to a given server. A single-user client SHOULD NOT maintain more than **2 connections** with any server or proxy. A proxy SHOULD use up to 2*N connections to another server or proxy, where N is the number of simultaneously active users. These guidelines are intended to improve HTTP response times and avoid congestion."
While that policy makes sense to prevent website owners from using their visitors to D-DOS other websites, it can be a big problem when multiple connections are required to handle server-client communication for legitimate use cases. To workaround the limitation you have to use HTTP/2 or HTTP/3 with which the browser will only open a single connection per domain and then use multiplexing to run all data through a single connection. While this gives you a virtually infinity amount of parallel connections, there is a [SETTINGS_MAX_CONCURRENT_STREAMS](https://www.rfc-editor.org/rfc/rfc7540#section-6.5.2) setting which limits the actually connections amount. The default is 100 concurrent streams for most configurations.
In theory the connection limit could also be increased by the browser, at least for specific APIs like EventSource, but the issues have been marked as "won't fix" by [chromium](https://issues.chromium.org/issues/40329530) and [firefox](https://bugzilla.mozilla.org/show_bug.cgi?id=906896).
:::note Lower the amount of connections in Browser Apps
When you build a browser application, you have to assume that your users will use the app not only once, but in multiple browser tabs in parallel.
By default you likely will open one server-stream-connection per tab which is often not necessary at all. Instead you open only a single connection and shared it between tabs, no matter how many tabs are open. [RxDB](https://rxdb.info/) does that with the [LeaderElection](../leader-election.md) from the [broadcast-channel npm package](https://github.com/pubkey/broadcast-channel) to only have one stream of replication between server and clients. You can use that package standalone (without RxDB) for any type of application.
:::
### Connections are not kept open on mobile apps
In the context of mobile applications running on operating systems like Android and iOS, maintaining open connections, such as those used for WebSockets and the others, poses a significant challenge. Mobile operating systems are designed to automatically move applications into the background after a certain period of inactivity, effectively closing any open connections. This behavior is a part of the operating system's resource management strategy to conserve battery and optimize performance. As a result, developers often rely on **mobile push notifications** as an efficient and reliable method to send data from servers to clients. Push notifications allow servers to alert the application of new data, prompting an action or update, without the need for a persistent open connection.
### Proxies and Firewalls
From consulting many [RxDB](https://rxdb.info/) users, it was shown that in enterprise environments (aka "at work") it is often hard to implement a WebSocket server into the infrastructure because many proxies and firewalls block non-HTTP connections. Therefore using the Server-Sent-Events provides and easier way of enterprise integration. Also long-polling uses only plain HTTP-requests and might be an option.
## Performance Comparison
Comparing the performance of WebSockets, Server-Sent Events (SSE), Long-Polling and WebTransport directly involves evaluating key aspects such as latency, throughput, server load, and scalability under various conditions.
First lets look at the raw numbers. A good performance comparison can be found in [this repo](https://github.com/Sh3b0/realtime-web?tab=readme-ov-file#demos) which tests the messages times in a [Go Lang](https://go.dev/) server implementation. Here we can see that the performance of WebSockets, WebRTC and WebTransport are comparable:
:::note
Remember that WebTransport is a pretty new technology based on the also new HTTP/3 protocol. In the future (after March 2024) there might be more performance optimizations. Also WebTransport is optimized to use less power which metric is not tested.
:::
Lets also compare the Latency, the throughput and the scalability:
### Latency
- **WebSockets**: Offers the lowest latency due to its full-duplex communication over a single, persistent connection. Ideal for real-time applications where immediate data exchange is critical.
- **Server-Sent Events**: Also provides low latency for server-to-client communication but cannot natively send messages back to the server without additional HTTP requests.
- **Long-Polling**: Incurs higher latency as it relies on establishing new HTTP connections for each data transmission, making it less efficient for real-time updates. Also it can occur that the server wants to send an event when the client is still in the process of opening a new connection. In these cases the latency would be significantly larger.
- **WebTransport**: Promises to offer low latency similar to WebSockets, with the added benefits of leveraging the HTTP/3 protocol for more efficient multiplexing and congestion control.
### Throughput
- **WebSockets**: Capable of high throughput due to its persistent connection, but throughput can suffer from [backpressure](https://chromestatus.com/feature/5189728691290112) where the client cannot process data as fast as the server is capable of sending it.
- **Server-Sent Events**: Efficient for broadcasting messages to many clients with less overhead than WebSockets, leading to potentially higher throughput for unidirectional server-to-client communication.
- **Long-Polling**: Generally offers lower throughput due to the overhead of frequently opening and closing connections, which consumes more server resources.
- **WebTransport**: Expected to support high throughput for both unidirectional and bidirectional streams within a single connection, outperforming WebSockets in scenarios requiring multiple streams.
### Scalability and Server Load
- **WebSockets**: Maintaining a large number of WebSocket connections can significantly increase server load, potentially affecting scalability for applications with many users.
- **Server-Sent Events**: More scalable for scenarios that primarily require updates from server to client, as it uses less connection overhead than WebSockets because it uses "normal" HTTP request without things like [protocol updates](https://developer.mozilla.org/en-US/docs/Web/HTTP/Protocol_upgrade_mechanism) that have to be run with WebSockets.
- **Long-Polling**: The least scalable due to the high server load generated by frequent connection establishment, making it suitable only as a fallback mechanism.
- **WebTransport**: Designed to be highly scalable, benefiting from HTTP/3's efficiency in handling connections and streams, potentially reducing server load compared to WebSockets and SSE.
## Recommendations and Use-Case Suitability
In the landscape of server-client communication technologies, each has its distinct advantages and use case suitability. **Server-Sent Events** (SSE) emerge as the most straightforward option to implement, leveraging the same HTTP/S protocols as traditional web requests, thereby circumventing corporate firewall restrictions and other technical problems that can appear with other protocols. They are easily integrated into Node.js and other server frameworks, making them an ideal choice for applications requiring frequent server-to-client updates, such as news feeds, stock tickers, and live event streaming.
On the other hand, **WebSockets** excel in scenarios demanding ongoing, two-way communication. Their ability to support continuous interaction makes them the prime choice for browser games, chat applications, and live sports updates.
However, **WebTransport**, despite its potential, faces adoption challenges. It is not widely supported by server frameworks [including Node.js](https://github.com/w3c/webtransport/issues/511) and lacks compatibility with [safari](https://caniuse.com/webtransport). Moreover, its reliance on HTTP/3 further limits its immediate applicability because many WebServers like nginx only have [experimental](https://nginx.org/en/docs/quic.html) HTTP/3 support. While promising for future applications with its support for both reliable and unreliable data transmission, WebTransport is not yet a viable option for most use cases.
**Long-Polling**, once a common technique, is now largely outdated due to its inefficiency and the high overhead of repeatedly establishing new HTTP connections. Although it may serve as a fallback in environments lacking support for WebSockets or SSE, its use is generally discouraged due to significant performance limitations.
## Known Problems
For all of the realtime streaming technologies, there are known problems. When you build anything on top of them, keep these in mind.
### A client can miss out events when reconnecting
When a client is connecting, reconnecting or offline, it can miss out events that happened on the server but could not be streamed to the client.
This missed out events are not relevant when the server is streaming the full content each time anyway, like on a live updating stock ticker.
But when the backend is made to stream partial results, you have to account for missed out events. Fixing that on the backend scales pretty bad because the backend would have to remember for each client which events have been successfully send already. Instead this should be implemented with client side logic.
The [RxDB Sync Engine](../replication.md) for example uses two modes of operation for that. One is the [checkpoint iteration mode](../replication.md#checkpoint-iteration) where normal http requests are used to iterate over backend data, until the client is in sync again. Then it can switch to [event observation mode](../replication.md#event-observation) where updates from the realtime-stream are used to keep the client in sync. Whenever a client disconnects or has any error, the replication shortly switches to [checkpoint iteration mode](../replication.md#checkpoint-iteration) until the client is in sync again. This method accounts for missed out events and ensures that clients can always sync to the exact equal state of the server.
### Company firewalls can cause problems
There are many known problems with company infrastructure when using any of the streaming technologies. Proxies and firewall can block traffic or unintentionally break requests and responses. Whenever you implement a realtime app in such an infrastructure, make sure you first test out if the technology itself works for you.
## FAQ
What are the differences between long polling and traditional polling in web development?
Traditional polling forces the client to ask the server for updates at fixed intervals. The server responds immediately even when no new data exists. This approach wastes bandwidth and increases server load. Long polling improves this model. The client requests data and the server holds the connection open until it has new information to send. The server then responds and the client immediately opens a new connection. Long polling reduces unnecessary requests and provides updates faster than traditional polling.
## Follow Up
- Check out the [hackernews discussion of this article](https://news.ycombinator.com/item?id=39745993)
- Shared/Like my [announcement tweet](https://twitter.com/rxdbjs/status/1769507055298064818)
- Learn how to use Server-Sent-Events to [replicate a client side RxDB database with your backend](../replication-http.md#pullstream-for-ongoing-changes).
- Learn how to use RxDB with the [RxDB Quickstart](../quickstart.md)
- Check out the [RxDB github repo](https://github.com/pubkey/rxdb) and leave a star β
---
## Zero Latency Local First Apps with RxDB β Sync, Encryption and Compression
Creating a **zero-latency local first** application involves ensuring that most (if not all) user interactions occur instantaneously, without waiting on remote network responses. This design drastically enhances user experience, allowing apps to remain responsive and functional even when offline or experiencing poor connectivity. As developers, we can achieve this by storing data **locally on the client** and synchronizing it to the backend in the background. **RxDB** (Reactive Database) offers a comprehensive set of features - covering replication, offline support, encryption, compression, conflict handling, and more - that make it straightforward to build such high-performing apps.
## Why Zero Latency with a Local First Approach?
In a traditional architecture, each user action triggers requests to a server for reads or writes. Despite network optimizations, unavoidable latencies can delay responses and disrupt the user flow. By contrast, a local first model maintains data in the client's environment (browser, mobile, desktop), drastically reducing user-perceived delays. Once the user re-connects or resumes activity online, changes propagate automatically to the server, eliminating manual synchronization overhead.
1. **Instant Responsiveness**: Because user actions (queries, updates, etc.) happen against a local datastore, UI updates do not wait on round-trip times.
2. **Offline Operation**: Apps can continue to read and write data, even when there is zero connectivity.
3. **Reduced Backend Load**: Instead of flooding the server with small requests, replication can combine and push or pull changes in batches.
4. **Simplified Caching**: Instead of implementing multi-layer caching, local first transforms your data layer into a reliable, quickly accessible store for all user actions.
## RxDB: Your Key to Zero-Latency Local First Apps
**RxDB** is a JavaScript-based [NoSQL](./in-memory-nosql-database.md) database designed for offline-first and real-time replication scenarios. It supports a range of environments - browsers (IndexedDB or OPFS), mobile ([Ionic](./ionic-storage.md), [React Native](../react-native-database.md)), [Electron](../electron-database.md), Node.js - and is built around:
- **Reactive Queries** that trigger UI updates upon data changes
- **Schema-based NoSQL Documents** for flexible but robust data models
- [Advanced Sync Engine](../replication.md): to synchronize with diverse backends
- **Encryption** for secure data at rest
- **Compression** to reduce local and network overhead
### Real-Time Sync and Offline-First
RxDB's replication logic revolves around pulling down remote changes and pushing up local modifications. It maintains a checkpoint-based mechanism, so only new or updated documents flow between the client and server, reducing bandwidth usage and latency. This ensures:
- **Live Data**: Queries automatically reflect server-side changes once they arrive locally.
- **Background Updates**: No manual polling needed; replication streams or intervals handle synchronization.
- **Conflict Handling** (see below) ensures data merges gracefully when multiple clients edit the same document offline.
#### Multiple Replication Plugins and Approaches
RxDB's flexible replication system lets you connect to different backends or even peer-to-peer networks. There are official plugins for [CouchDB](../replication-couchdb.md), [Firestore](../replication-firestore.md), [GraphQL](../replication-graphql.md), [WebRTC](../replication-webrtc.md), and more. Many developers create a **custom HTTP replication** to work with their existing REST-based backend, ensuring a painless integration that doesn't require adopting an entirely new server infrastructure.
#### Example Setup of a local database
```ts
import { createRxDatabase } from 'rxdb/plugins/core';
import { getRxStorageLocalstorage } from 'rxdb/plugins/storage-localstorage';
async function initZeroLocalDB() {
// Create a local RxDB instance using localstorage-based storage
const db = await createRxDatabase({
name: 'myZeroLocalDB',
storage: getRxStorageLocalstorage(),
// optional: password for encryption if needed
});
// Define one or more collections
await db.addCollections({
tasks: {
schema: {
title: 'task schema',
version: 0,
type: 'object',
primaryKey: 'id',
properties: {
id: { type: 'string', maxLength: 100 },
title: { type: 'string' },
done: { type: 'boolean' }
}
}
}
});
// Reactive query - automatically updates on local or remote changes
db.tasks
.find()
.$ // returns an RxJS Observable
.subscribe(allTasks => {
console.log('All tasks updated:', allTasks);
});
return db;
}
```
When offline, reads and writes to `db.tasks` happen locally with near-zero delay. Once connectivity resumes, changes sync to the server automatically (if replication is configured).
#### Example Setup of the replication
```ts
import { replicateRxCollection } from 'rxdb/plugins/replication';
async function syncLocalTasks(db) {
replicateRxCollection({
collection: db.tasks,
replicationIdentifier: 'sync-tasks',
// Define how to pull server documents and push local documents
pull: {
handler: async (lastCheckpoint, batchSize) => {
// logic to retrieve updated tasks from the server since lastCheckpoint
},
},
push: {
handler: async (docs) => {
// logic to post local changes to the server
},
},
live: true, // continuously replicate
retryTime: 5000, // retry on errors or disconnections
});
}
```
This replication seamlessly merges server-side and client-side changes. Your app remains responsive throughout, regardless of the network status.
## Things you should also know about
### Optimistic UI on Local Data Changes
A local first approach, especially with RxDB, naturally supports an [optimistic UI](./optimistic-ui.md) pattern. Because writes occur on the client, you can instantly reflect changes in the interface as soon as the user performs an action - no need to wait for server confirmation. For example, when a user updates a task document to done: true, the UI can re-render immediately with that new state. This even works across multiple browser tabs.
If a server conflict arises later during replication, RxDB's [conflict handling](../transactions-conflicts-revisions.md) logic determines which changes to keep, and the UI can be updated accordingly. This is far more efficient than blocking the user or displaying a spinner while the backend processes the request.
### Conflict Handling
In local first models, conflicts emerge if multiple devices or clients edit the same document while offline. RxDB tracks document revisions so you can detect collisions and merge them effectively. By default, RxDB uses a last-write-wins approach, but developers can override it with a custom conflict handler. This provides fine-grained control - like merging partial fields, storing revision histories, or prompting users for resolution. Proper conflict handling keeps distributed data consistent across your entire system.
### Schema Migrations
Over time, apps evolve - new fields, changed field types, or altered indexes. RxDB allows incremental schema migrations so you can upgrade a user's local data from one schema version to another. You might, for instance, rename a property or transform data formats. Once you define your migration strategy, RxDB automatically applies it upon app initialization, ensuring the local database's structure aligns with your latest codebase.
## Advanced Features
### Setup Encryption
When storing data locally, you may handle user-sensitive information like PII (Personal Identifiable Information) or financial details. RxDB supports on-device [encryption](../encryption.md) to protect fields. For example, you can define:
```ts
import { wrappedKeyEncryptionCryptoJsStorage } from 'rxdb/plugins/encryption-crypto-js';
const encryptedStorage = wrappedKeyEncryptionCryptoJsStorage({
storage: getRxStorageLocalstorage()
});
const db = await createRxDatabase({
name: 'secureDB',
storage: encryptedStorage,
password: 'myEncryptionPassword'
});
await db.addCollections({
secrets: {
schema: {
title: 'secrets schema',
version: 0,
type: 'object',
primaryKey: 'id',
properties: {
id: { type: 'string', maxLength: 100 },
secretField: { type: 'string' }
},
required: ['id'],
encrypted: ['secretField'] // define which fields to encrypt
}
}
});
```
Then mark fields as `encrypted` in the schema. This ensures data is unreadable on disk without the correct password.
### Setup Compression
Local data can expand quickly, especially for large documents or repeated key names. RxDB's key compression feature replaces verbose field names with shorter tokens, decreasing storage usage and speeding up replication. You enable it by adding keyCompression: true to your collection schema:
```ts
await db.addCollections({
logs: {
schema: {
title: 'log schema',
version: 0,
keyCompression: true,
type: 'object',
primaryKey: 'id',
properties: {
id: { type: 'string'. maxLength: 100 },
message: { type: 'string' },
timestamp: { type: 'number' }
}
}
}
});
```
## Different RxDB Storages Depending on the Runtime
RxDB's storage layer is swappable, so you can pick the optimal adapter for each environment. Some common choices include:
- [IndexedDB](../rx-storage-indexeddb.md) in modern browsers (default).
- [OPFS](../rx-storage-opfs.md) (Origin Private File System) in browsers that support it for potentially better performance.
- [SQLite](../rx-storage-sqlite.md) for mobile or desktop environments via the premium plugin, offering native-like speed on Android, iOS, or Electron.
- [In-Memory](../rx-storage-memory.md) for tests or ephemeral data.
By choosing a suitable storage layer, you can adapt your zero-latency local first design to any runtime - web, [mobile](./mobile-database.md), or server-like contexts in [Node.js](../nodejs-database.md).
## Performance Considerations
Performant local data operations are crucial for a zero-latency experience. According to the RxDB [storage performance overview](../rx-storage-performance.md), differences in underlying storages can significantly impact throughput and latency. For instance, IndexedDB typically performs well across modern browsers, [OPFS](../rx-storage-opfs.md) offers improved throughput in supporting browsers, and [SQLite storage](../rx-storage-sqlite.md) (a premium plugin) often delivers near-native speed for mobile or desktop.
### Offloading Work from the Main Thread
In a browser environment, you can move database operations into a Web Worker using the [Worker RxStorage plugin](../rx-storage-worker.md). This approach lets you keep heavy data processing off the main thread, ensuring the UI remains smooth and responsive. Complex queries or large write operations no longer cause stuttering in the user interface.
### Sharding or Memory-Mapped Storages
For large datasets or high concurrency, advanced techniques like [sharding](../rx-storage-sharding.md) collections across multiple storages or leveraging a [memory-mapped](../rx-storage-memory-mapped.md) variant can further boost performance. By splitting data into smaller subsets or streaming it only as needed, you can scale to handle complex usage scenarios without compromising on the zero-latency user experience.
## Follow Up
- Dive into the [RxDB Quickstart](../quickstart.md) to set up your own local first database.
- Explore [Replication Plugins](../replication.md) for syncing with platforms like [CouchDB](../replication-couchdb.md), [Firestore](./firestore-alternative.md), or [GraphQL](../replication-graphql.md).
- Check out Advanced [Conflict Handling](../transactions-conflicts-revisions.md) and [Performance Tuning](../rx-storage-performance.md) for big data sets or complex multi-user interactions.
- Join the RxDB Community on [GitHub](/code/) and [Discord](/chat/) to share insights, file issues, and learn from other developers building zero-latency solutions.
-
By integrating RxDB into your stack, you achieve millisecond interactions, full [offline capabilities](../offline-first.md), secure data at rest, and minimal overhead for large or distributed teams. This zero-latency local first architecture is the future of modern software - delivering a fluid, always-available user experience without overcomplicating the developer workflow.
---
## Capacitor Database Guide - SQLite, RxDB & More
import {Tabs} from '@site/src/components/tabs';
import {Steps} from '@site/src/components/steps';
# Capacitor Database - SQLite, RxDB and others
[Capacitor](https://capacitorjs.com/) is an open source native JavaScript runtime to build Web based Native apps. You can use it to create cross-platform iOS, Android, and [Progressive Web Apps](./articles/progressive-web-app-database.md) with the web technologies JavaScript, HTML, and CSS.
It is developed by the Ionic Team and provides a great alternative to create hybrid apps. Compared to [React Native](./react-native-database.md), Capacitor is more Web-Like because the JavaScript runtime supports most Web APIs like [IndexedDB](./rx-storage-indexeddb.md), fetch, and so on.
To read and write persistent data in Capacitor, there are multiple solutions which are shown in the following.
## Database Solutions for Capacitor
### Preferences API
Capacitor comes with a native [Preferences API](https://capacitorjs.com/docs/apis/preferences) which is a simple, persistent key->value store for lightweight data, similar to the browsers localstorage or React Native [AsyncStorage](./react-native-database.md#asyncstorage).
To use it, you first have to install it from npm `npm install @capacitor/preferences` and then you can import it and write/read data.
Notice that all calls to the preferences API are asynchronous so they return a `Promise` that must be `await`-ed.
```ts
import { Preferences } from '@capacitor/preferences';
// write
await Preferences.set({
key: 'foo',
value: 'baar',
});
// read
const { value } = await Preferences.get({ key: 'foo' }); // > 'bar'
// delete
await Preferences.remove({ key: 'foo' });
```
The preferences API is good when only a small amount of data needs to be stored and when no query capabilities besides the key access are required. Complex queries or other features like indexes or replication are not supported which makes the preferences API not suitable for anything more than storing simple data like user settings.
### Localstorage/IndexedDB/WebSQL
Since Capacitor apps run in a web view, Web APIs like IndexedDB, [Localstorage](./articles/localstorage.md) and WebSQL are available. But the default browser behavior is to clean up these storages regularly when they are not in use for a long time or the device is low on space. Therefore you cannot 100% rely on the persistence of the stored data and your application needs to expect that the data will be lost eventually.
Storing data in these storages can be done in browsers, because there is no other option. But in Capacitor iOS and Android, you should not rely on these.
### SQLite
SQLite is a SQL based relational database written in C that was crafted to be embed inside of applications. Operations are written in the SQL query language and SQLite generally follows the PostgreSQL syntax.
To use SQLite in Capacitor, there are three options:
- The [@capacitor-community/sqlite](https://github.com/capacitor-community/sqlite) package
- The [cordova-sqlite-storage](https://github.com/storesafe/cordova-sqlite-storage) package
- The non-free [Ionic](./articles/ionic-database.md) [Secure Storage](https://ionic.io/products/secure-storage) which comes at **999$** per month.
It is recommended to use the `@capacitor-community/sqlite` because it has the best maintenance and is open source. Install it first `npm install --save @capacitor-community/sqlite` and then set the storage location for iOS apps:
```json
{
"plugins": {
"CapacitorSQLite": {
"iosDatabaseLocation": "Library/CapacitorDatabase"
}
}
}
```
Now you can create a database connection and use the SQLite database.
```ts
import { Capacitor } from '@capacitor/core';
import {
CapacitorSQLite, SQLiteDBConnection, SQLiteConnection, capSQLiteSet,
capSQLiteChanges, capSQLiteValues, capEchoResult, capSQLiteResult,
capNCDatabasePathResult
} from '@capacitor-community/sqlite';
const sqlite = new SQLiteConnection(CapacitorSQLite);
const database: SQLiteDBConnection = await this.sqlite.createConnection(
databaseName,
encrypted,
mode,
version,
readOnly
);
let { rows } = database.query('SELECT somevalue FROM sometable');
```
The downside of SQLite is that it is lacking many features that are handful when using a database together with an UI based application like your Capacitor app. For example it is not possible to observe queries or document fields. Also there is no realtime replication feature, you can only import json files. This makes SQLite a good solution when you just want to store data on the client, but when you want to sync data with a server or other clients or create big complex realtime applications, you have to use something else.
### RxDB
[RxDB](https://rxdb.info/) is an local first, NoSQL database for JavaScript Applications like hybrid apps. Because it is reactive, you can subscribe to all state changes like the result of a query or even a single field of a document. This is great for UI-based realtime applications in a way that makes it easy to develop realtime applications like what you need in Capacitor.
Because RxDB is made for Web applications, most of the [available RxStorage](./rx-storage.md) plugins can be used to store and query data in a Capacitor app. However it is recommended to use the [SQLite RxStorage](./rx-storage-sqlite.md) because it stores the data on the filesystem of the device, not in the JavaScript runtime (like IndexedDB). Storing data on the filesystem ensures it is persistent and will not be cleaned up by any process. Also the performance of SQLite is [much faster](./rx-storage.md#performance-comparison) compared to IndexedDB, because SQLite does not have to go through a browsers permission layers. For the SQLite binding you should use the [@capacitor-community/sqlite](https://github.com/capacitor-community/sqlite) package.
Because the SQLite RxStorage is part of the [π Premium Plugins](/premium/) which must be purchased, it is recommended to use the [LocalStorage RxStorage](./rx-storage-localstorage.md) while testing and prototyping your Capacitor app.
To use the SQLite RxStorage in Capacitor you have to install all dependencies via `npm install rxdb rxjs rxdb-premium @capacitor-community/sqlite`.
For iOS apps you should add a database location in your Capacitor settings:
```json
{
"plugins": {
"CapacitorSQLite": {
"iosDatabaseLocation": "Library/CapacitorDatabase"
}
}
}
```
Then you can assemble the RxStorage and create a database with it:
### Import RxDB and SQLite
```ts
import {
createRxDatabase
} from 'rxdb/plugins/core';
import {
CapacitorSQLite,
SQLiteConnection
} from '@capacitor-community/sqlite';
import { Capacitor } from '@capacitor/core';
const sqlite = new SQLiteConnection(CapacitorSQLite);
```
### Import the RxDB SQLite Storage
#### RxDB Core
```ts
import {
getRxStorageSQLiteTrial,
getSQLiteBasicsCapacitor
} from 'rxdb/plugins/storage-sqlite';
```
#### RxDB Premium π
```ts
import {
getRxStorageSQLite,
getSQLiteBasicsCapacitor
} from 'rxdb-premium/plugins/storage-sqlite';
```
### Create a Database with the Storage
#### RxDB Core
```ts
// create database
const myRxDatabase = await createRxDatabase({
name: 'exampledb',
storage: getRxStorageSQLiteTrial({
sqliteBasics: getSQLiteBasicsCapacitor(sqlite, Capacitor)
})
});
```
#### RxDB Premium π
```ts
// create database
const myRxDatabase = await createRxDatabase({
name: 'exampledb',
storage: getRxStorageSQLite({
sqliteBasics: getSQLiteBasicsCapacitor(sqlite, Capacitor)
})
});
```
### Add a Collection
```ts
// create collections
const collections = await myRxDatabase.addCollections({
humans: {
schema: {
version: 0,
type: 'object',
primaryKey: 'id',
properties: {
id: { type: 'string', maxLength: 100 },
name: { type: 'string' },
age: { type: 'number' }
},
required: ['id', 'name']
}
}
});
```
### Insert a Document
```ts
await collections.humans.insert({id: 'foo', name: 'bar'});
```
### Run a Query
```ts
const result = await collections.humans.find({
selector: {
name: 'bar'
}
}).exec();
```
### Observe a Query
```ts
await collections.humans.find({
selector: {
name: 'bar'
}
}).$.subscribe(result => {/* ... */});
```
## Follow up
- If you haven't done yet, you should start learning about RxDB with the [Quickstart Tutorial](./quickstart.md).
- There is a followup list of other [client side database alternatives](./alternatives.md).
---
## Contribute
# Contribution
We are open to, and grateful for, any contributions made by the community.
# Developing
## Requirements
Before you can start developing, do the following:
1. Make sure you have installed nodejs with the version stated in the [.nvmrc](https://github.com/pubkey/rxdb/blob/master/.nvmrc)
2. Clone the repository `git clone https://github.com/pubkey/rxdb.git`
3. Install the dependencies `cd rxdb && npm install`
4. Make sure that the tests work for you. At first, try it out with `npm run test:node:memory` which tests the [memory storage](./rx-storage-memory.md) in node. In the [package.json](https://github.com/pubkey/rxdb/blob/master/package.json) you can find more scripts to run the tests with different storages.
## Adding tests
Before you start creating a bugfix or a feature, you should create a test to reproduce it. Tests are in the `test/unit`-folder.
If you want to reproduce a bug, you can modify the test in [this file](https://github.com/pubkey/rxdb/blob/master/test/unit/bug-report.test.ts).
## Making a PR
If you make a pull-request, ensure the following:
1. Every feature or bugfix must be committed together with a unit-test which ensures everything works as expected.
2. Do not commit build-files (anything in the `dist`-folder)
3. Before you add non-trivial changes, create an issue to discuss if this will be merged and you don't waste your time.
4. To run the unit and integration-tests, do `npm run test` and ensure everything works as expected
## Getting help
If you need help with your contribution, ask at [discord](https://rxdb.info/chat/).
## No-Go
When reporting a bug, you need to make a PR with a test case that runs in the CI and reproduces your problem.
Sending a link with a repo does not help the maintainer because installing random peoples projects is time consuming and dangerous.
Also the maintainer will never go on a bug hunt based on your plain description. Either you report the bug with a test case, or the maintainer will likely not help you.
# Docs
The source of the documentation is at the `docs-src`-folder.
To read the docs locally, run `npm run docs:install && npm run docs:serve` and open `http://localhost:4000/`
# Thank you for contributing!
---
## Data Migration
This documentation page has been moved to [here](./migration-schema.md)
---
## Downsides of Local First / Offline First
import {QuoteBlock} from '@site/src/components/quoteblock';
# Downsides of Local First / Offline First
So you have read [all these things](./offline-first.md) about how the [local-first](./articles/local-first-future.md) (aka offline-first) paradigm makes it easy to create [realtime](./articles/realtime-database.md) web applications that even work when the user has no internet connection.
But there is no free lunch. The offline first paradigm is not the perfect approach for all kinds of apps.
You fully understood a technology when you know when not to use it
In the following I will point out the limitations you need to know before you decide to use [RxDB](https://github.com/pubkey/rxdb) or even before you decide to create an offline first application.
## It only works with small datasets
Making data available offline means it must be loaded from the server and then stored at the clients device.
You need to load the full dataset on the first pageload and on every ongoing load you need to download the new changes to that set.
While in theory you could download in infinite amount of data, in practice you have a limit how long the user can wait before having an up-to-date state.
You want to display chat messages like Whatsapp? No problem. Syncing all the messages a user could write, can be done with a few HTTP requests.
Want to make a tool that displays server logs? Good luck downloading terabytes of data to the client just to search for a single string. This will not work.
Besides the network usage, there is another limit for the size of your data.
In browsers you have some options for storage: Cookies, [Localstorage](./articles/localstorage.md), [WebSQL](./articles/localstorage-indexeddb-cookies-opfs-sqlite-wasm.md#what-was-websql) and [IndexedDB](./rx-storage-indexeddb.md).
Because Cookies and [Localstorage](./articles/localstorage.md) is slow and WebSQL is deprecated, you will use IndexedDB.
The [limit of how much data you can store in IndexedDB](./articles/indexeddb-max-storage-limit.md) depends on two factors: Which browser is used and how much disc space is left on the device. You can assume that at least a couple of [hundred megabytes](https://web.dev/storage-for-the-web/) are available at least. The maximum is potentially hundreds of gigabytes or more, but the browser implementations vary. Chrome allows the browser to use up to 60% of the total disc space per origin. Firefox allows up to 50%. But on safari you can only store up to 1GB and the browser will prompt the user on each additional 200MB increment.
The problem is, that you have no chance to really predict how much data can be stored. So you have to make assumptions that are hopefully true for all of your users. Also, you have no way to increase that space like you would add another hard drive to your backend server. Once your clients reach the limit, you likely have to rewrite big parts of your applications.
UPDATE (2023): Newer versions of browsers can store way more data, for example firefox stores up to 10% of the total disk size. For an overview about how much can be stored, read [this guide](https://developer.mozilla.org/en-US/docs/Web/API/Storage_API/Storage_quotas_and_eviction_criteria)
## Browser storage is not really persistent
When data is stored inside IndexedDB or one of the other storage APIs, it cannot be trusted to stay there forever.
Apple for example deletes the data when the website was not used in the [last 7 days](https://webkit.org/blog/10218/full-third-party-cookie-blocking-and-more/). The other browsers also have logic to clean up the stored data, and in the end the user itself could be the one that deletes the browsers local data.
The most common way to handle this, is to replicate everything from the backend to the client again.
Of course, this does not work for state that is not stored at the backend. So if you assume you can store the users private data inside the browser in a secure way, you are [wrong](https://medium.com/universal-ethereum/out-of-gas-were-shutting-down-unilogin-3b544838df1a#4f60).
## There can be conflicts
Imagine two of your users modify the same JSON document, while both are offline. After they go online again, their clients replicate the modified document to the server. Now you have two conflicting versions of the same document, and you need a way to determine how the correct new version of that document should look like. This process is called **conflict resolution**.
1. The default in [many](https://docs.couchdb.org/en/stable/replication/conflicts.html) offline first databases is a deterministic conflict resolution strategy. Both conflicting versions of the document are kept in the storage and when you query for the document, a winner is determined by comparing the hashes of the document and only the winning document is returned. Because the comparison is deterministic, all clients and servers will always pick the same winner. This kind of resolution only works when it is not that important that one of the document changes gets dropped. Because conflicts are rare, this might be a viable solution for some use cases.
2. A better resolution can be applied by listening to the changestream of the database. The changestream emits an event each time a write happens to the database. The event contains information about the written document and also a flag if there is a conflicting version. For each event with a conflict, you fetch all versions for that document and create a new document that contains the winning state. With that you can implement pretty complex conflict resolution strategies, but you have to manually code it for each collection of documents.
3. Instead of the solving conflict once at every client, it can be made a bit easier by solely relying on the backend. This can be done when all of your clients replicate with the same single backend server. With [RxDB's Graphql Replication](./replication-graphql.md) each client side change is sent to the server where conflicts can be resolved and the winning document can be sent back to the clients.
4. Sometimes there is no way to solve a conflict with code. If your users edit text based documents or images, often only the users themselves can decide how the winning revision has to look. For these cases, you have to implement complex UI parts where the users can inspect the conflict and manage its resolution.
5. You do not have to handle conflicts if they cannot happen in the first place. You can achieve that by designing a write only database where existing documents cannot be touched. Instead of storing the current state in a single document, you store all the events that lead to the current state. Sometimes called the ["everything is a delta"](https://pouchdb.com/guides/conflicts.html#accountants-dont-use-erasers) strategy, others would call it [Event Sourcing](https://martinfowler.com/eaaDev/EventSourcing.html). Like an accountant that does not need an eraser, you append all changes and afterwards aggregate the current state at the client.
```ts
// create one new document for each change to the users balance
{id: new Date().toJSON(), change: 100} // balance increased by $100
{id: new Date().toJSON(), change: -50} // balance decreased by $50
{id: new Date().toJSON(), change: 200} // balance increased by $200
```
6. There is this thing called **conflict-free replicated data type**, short **CRDT**. Using a CRDT library like [automerge](https://github.com/automerge/automerge) will magically solve all of your conflict problems. Until you use it in production where you observe that implementing CRDTs has basically the same complexity as implementing conflict resolution strategies.
## Realtime is a lie
So you replicate stuff between the clients and your backend. Each change on one side directly changes the state of the other sides in **realtime**. But this "realtime" is not the same as in [realtime computing](https://en.wikipedia.org/wiki/Real-time_computing). In the offline first world, the word realtime was introduced by firebase and is more meant as a marketing slogan than a technical description.
There is an internet between your backend and your clients and everything you do on one machine takes at least once the latency until it can affect anything on the other machines. You have to keep this in mind when you develop anything where the timing is important, like a multiplayer game or a stock trading app.
Even when you run a query against the local database, there is no "real" realtime.
Client side databases run on JavaScript and JavaScript runs on a single CPU that might be partially blocked because the user is running some background processes. So you can never guarantee a response deadline which violates the time constraint of realtime computing.
## Eventual consistency
An offline first app does not have a single source of truth. There is a source on the backend, one on the own client, and also each other client has its own definition of truth. At the moment your user starts the app, the local state is hopefully already replicated with the backend and all other clients. But this does not have to be true, the states can have converged and you have to plan for that.
The user could update a document based on wrong assumptions because it was not fully replicated at that point in time because the user is offline. A good way to handle this problem is to show the replication state in the UI and tell the user when the replication is running, stopped, paused or finished.
And some data is just too important to be "eventual consistent". Create a wire transfer in your online banking app while you are offline. You keep the smartphone laying at your night desk and when you use again in the next morning, it goes online and replicates the transaction. No thank you, do not use offline first for these kinds of things, or at least you have to display the replication state of each document in the UI.
## Permissions and authentication
Every offline first app that goes beyond a prototype, does likely not have the same global state for all of its users. Each user has a different set of documents that are allowed to be replicated or seen by the user. So you need some kind of authentication and permission handling to divide the documents.
The easy way is to just create one database for each user on the backend and only allow to replicate that one. Creating that many databases is not really a problem with for example CouchDB, and it makes permission handling easy.
But as soon as you want to query all of your data in the backend, it will bite back. Your data is not at a single place, it is distributed between all of the user specific databases. This becomes even more complex as soon as you store information together with the documents that is not allowed to be seen by outsiders. You not only have to decide which documents to replicate, but also which fields of them.
So what you really want is a single datastore in the backend and then replicate only the allowed document parts to each of the users.
This always requires you to implement your custom replication endpoint like what you do with RxDBs [GraphQL Replication](./replication-graphql.md).
## You have to migrate the client database
While developing your app, sooner or later you want to change the data layout. You want to add some new fields to documents or change the format of them. So you have to update the database schema and also migrate the stored documents.
With 'normal' applications, this is already hard enough and often dangerous. You wait until midnight, stop the webserver, make a database backup, deploy the new schema and then you hope that nothing goes wrong while it updates that many documents.
With offline first applications, it is even more fun. You do not only have to migrate your local backend database, you also have to provide a [migration strategy](./migration-schema.md) for all of these client databases out there. And you also cannot migrate everything at the same time. The clients can only migrate when the new code was updated from the appstore or the user visited your website again. This could be today or in a few weeks.
## Performance is not native
When you create a web based offline first app, you cannot store data directly on the users filesystem. In fact there are many layers between your JavaScript code and the filesystem of the operation system. Let's say you insert a document in [RxDB](https://github.com/pubkey/rxdb):
- You call the RxDB API to validate and store the data
- RxDB calls the underlying RxStorage, for example [PouchDB](./rx-storage-pouchdb.md).
- Pouchdb calls its underlying storage adapter
- The storage adapter calls IndexedDB
- The browser runs its internal handling of the IndexedDB API
- In most browsers IndexedDB is implemented on [top of SQLite](https://hackaday.com/2021/08/24/sqlite-on-the-web-absurd-sql/)
- [SQLite](./rx-storage-sqlite.md) calls the OS to store the data in the filesystem
All these layers are abstractions. They are not build for exactly that one use case, so you lose some performance to tunnel the data through the layer itself, and you also lose some performance because the abstraction does not exactly provide the functions that are needed by the layer above and it will overfetch data.
You will not find a benchmark comparison between how many transactions per second you can run on the browser compared to a server based database. Because it makes no sense to compare them. Browsers are slower, JavaScript is slower.
> **Is it fast enough?**
What you really care about is "Is it fast enough?". For most use cases, the answer is `yes`. Offline first apps are UI based and you do not need to process a million transactions per second, because your user will not click the save button that often. "Fast enough" means that the data is processed in under 16 milliseconds so that you can render the updated UI in the next frame. This is of course not true for all use cases, so you better think about the performance limit before starting with the implementation.
## Nothing is predictable
You have a PostgreSQL database and run a query over 1000ths of rows, which takes 200 milliseconds. Works great, so you now want to do something similar at the client device in your offline first app. How long does it take? You cannot know because people have different devices, and even equal devices have different things running in the background that slow the CPUs. So you cannot predict performance and as described above, you cannot even predict the storage limit. So if your app does heavy data analytics, you might better run everything on the backend and just send the results to the client.
## There is no relational data
I started creating [RxDB](https://github.com/pubkey/rxdb) many years ago and while still maintaining it, I often worked with all these other offline first databases out there. RxDB and all of these other ones, are based on some kind of document databases similar to NoSQL. Often people want to have a relational database like the SQL one they use at the backend.
So why are there no real relations in offline first databases? I could answer with these arguments like how JavaScript works better with document based data, how performance is better when having no joins or even how NoSQL queries are more composable. But the truth is, everything is NoSQL because it makes replication easy. An SQL query that mutates data in different tables based on some selects and joins, cannot be partially replicated without breaking the client. You have foreign keys that point to other rows and if these rows are not replicated yet, you have a problem. To implement a robust [Sync Engine](./replication.md) for relational data, you need some stuff like a [reliable atomic clock](https://www.theverge.com/2012/11/26/3692392/google-spanner-atomic-clocks-GPS) and you have to block queries over multiple tables while a transaction replicated. [Watch this guy](https://youtu.be/iEFcmfmdh2w?t=607) implementing offline first replication on top of SQLite or read this [discussion](https://github.com/supabase/supabase/discussions/357) about implementing [offline first in supabase](./replication-supabase.md).
So creating replication for an SQL offline first database is way more work than just adding some network protocols on top of PostgreSQL. It might not even be possible for clients that have no reliable clock.
---
## Electron Database - Storage adapters for SQLite, Filesystem and In-Memory
# Electron Database - RxDB with different storage for SQLite, Filesystem and In-Memory
[Electron](https://www.electronjs.org/) (aka Electron.js) is a framework developed by github that is designed to create desktop applications with the Web technology stack consisting of HTML, CSS and JavaScript.
Because the desktop application runs on the client's device, it is suitable to use a database that can store and query data locally. This allows you to create so-called [local first](./offline-first.md) apps that store data locally and even work when the user has no internet connection.
While there are many options to store data in Electron, for complex realtime apps using [RxDB](https://rxdb.info/) is recommended because it is a database made for UI-based client-side application, not a server-side database.
## Databases for Electron
An Electron runtime can be divided into two parts:
- The "main" process which is a Node.js JavaScript process that runs without a UI in the background.
- One or multiple "renderer" processes that consist of a Chrome browser engine and runs the user interface. Each renderer process represents one "browser tab".
This is important to understand because choosing the right database depends on your use case and on which of these JavaScript runtimes you want to keep the data.
### Server Side Databases in Electron.js
Because Electron runs on a desktop computer, you might think that it should be possible to use a common "server" database like MySQL, PostgreSQL or MongoDB. In theory, you could ship the correct database server binaries with your electron application and start a process on the client's device that exposes a port to the database that can be consumed by Electron. In practice, this is not a viable way to go because shipping the correct binaries and opening ports is way to complicated and troublesome. Instead you should use a database that can be bundled and run **inside** of Electron, either in the *main* or in the *renderer* process.
### Localstorage / IndexedDB / WebSQL as alternatives to SQLite in Electron
Because Electron uses a common Chrome web browser in the renderer process, you can access the common Web Storage APIs like [Localstorage](./articles/localstorage.md), [IndexedDB](./rx-storage-indexeddb.md) and WebSQL. This is easy to set up and storing small sets of data can be achieved in a short span of time.
But as soon as your application goes beyond a simple todo-app, there are multiple obstacles that come in your way. One thing is the bad multi-tab support. If you have more than one *renderer* process, it becomes hard to manage database writes between them. Each *browser tab* could modify the database state while the others do not know of the changes and keep an outdated UI.
Another thing is performance. [IndexedDB is slow](./slow-indexeddb.md), mostly because it has to go through layers of browser security and abstractions. Storing and querying a lot of data might become your performance bottleneck. Localstorage and WebSQL are even slower, by the way. Using these Web Storage APIs is generally only recommended when you know for sure that there will be always only **one rendering process** and performance is not that relevant. The main reason for that is the security- and abstraction layers that write- and read operations have to go through when using the browsers IndexedDB API. So instead of using IndexedDB in Electron in the renderer process, you should use something that runs in the "main" process in Node.js like the [Filesystem RxStorage](./rx-storage-filesystem-node.md) or the [In Memory RxStorage](./rx-storage-memory.md).
### RxDB
[RxDB](https://rxdb.info/) is a NoSQL database for JavaScript applications. It has many features that come in handy when RxDB is used with UI based applications like your Electron app. For example, it is able to subscribe to query results of single fields of documents. It has [encryption](./encryption.md) and [compression](./key-compression.md) features and most important it has a battle tested [Sync Engine](./replication.md) that can be used to do a realtime sync with your backend.
Because of the [flexible storage](https://rxdb.info/rx-storage.html) layer of RxDB, there are many options on how to use it with Electron:
- The [memory RxStorage](./rx-storage-memory.md) that stores the data inside of the JavaScript memory without persistence
- The [SQLite RxStorage](./rx-storage-sqlite.md)
- The [IndexedDB RxStorage](./rx-storage-indexeddb.md)
- The [LocalStorage RxStorage](./rx-storage-localstorage.md)
- The [Dexie.js RxStorage](./rx-storage-dexie.md)
- The [Node.js Filesystem](./rx-storage-filesystem-node.md)
It is recommended to use the [SQLite RxStorage](./rx-storage-sqlite.md) because it has the best performance and is the easiest to set up. However it is part of the [π Premium Plugins](/premium/) which must be purchased, so to try out RxDB with Electron, you might want to use one of the other options. To start with RxDB, I would recommend using the LocalStorage RxStorage in the renderer processes. Because RxDB is able to broadcast the database state between browser tabs, having multiple renderer processes is not a problem like it would be when you use plain IndexedDB without RxDB.
In production, you would always run the RxStorage in the main process with the [RxStorage Electron IpcRenderer & IpcMain](./electron.md#rxstorage-electron-ipcrenderer--ipcmain) plugins.
First, you have to install all dependencies via `npm install rxdb rxjs`.
Then you can assemble the RxStorage and create a database with it:
```ts
import { createRxDatabase } from 'rxdb';
import { getRxStorageLocalstorage } from 'rxdb/plugins/storage-localstorage';
// create database
const db = await createRxDatabase({
name: 'exampledb',
storage: getRxStorageLocalstorage()
});
// create collections
const collections = await myRxDatabase.addCollections({
humans: {
/* ... */
}
});
// insert document
await collections.humans.insert({id: 'foo', name: 'bar'});
// run a query
const result = await collections.humans.find({
selector: {
name: 'bar'
}
}).exec();
// observe a query
await collections.humans.find({
selector: {
name: 'bar'
}
}).$.subscribe(result => {/* ... */});
```
For better performance in the renderer tab, you can later switch to the [IndexedDB RxStorage](./rx-storage-indexeddb.md). But in production, it is recommended to use the [SQLite RxStorage](./rx-storage-sqlite.md) or the [Filesystem RxStorage](./rx-storage-filesystem-node.md) in the main process so that database operations do not block the rendering of the UI.
To learn more about using RxDB with Electron, you might want to check out [this example project](https://github.com/pubkey/rxdb/tree/master/examples/electron).
### SQLite in Electron.js without RxDB
SQLite is a SQL based relational database written in the C programming language that was crafted to be embedded inside of applications and stores data locally. Operations are written in the SQL query language similar to the PostgreSQL syntax.
Using SQLite in Electron is not possible in the *renderer process*, only in the *main process*. To communicate data operations between your main and your renderer processes, you have to use either [@electron/remote](https://github.com/electron/remote) (not recommended) or the [ipcRenderer](https://www.electronjs.org/de/docs/latest/api/ipc-renderer) (recommended). So you start up SQLite in your main process and whenever you want to read or write data, you send the SQL queries to the main process and retrieve the result back as JSON data.
To install SQLite, use the [SQLite3](https://github.com/TryGhost/node-sqlite3) package which is a native Node.js module. You also need the [@electron/rebuild](https://github.com/electron/rebuild) package to rebuild the SQLite module against the currently installed Electron version.
Install them with `npm install sqlite3 @electron/rebuild`.
Then you can rebuild SQLite with `./node_modules/.bin/electron-rebuild -f -w sqlite3`
In the JavaScript code of your main process you can now create a database:
```ts
const sqlite3 = require('sqlite3');
const db = new sqlite3.Database('/path/to/database/file.db');
// create a table and insert a row
db.serialize(() => {
db.run("CREATE TABLE Users (name, lastName)");
db.run("INSERT INTO Users VALUES (?, ?)", ['foo', 'bar']);
});
```
Also you have to set up the ipcRenderer so that message from the renderer process are handled:
```ts
ipcMain.handle('db-query', async (event, sqlQuery) => {
return new Promise(res => {
db.all(sqlQuery, (err, rows) => {
res(rows);
});
});
});
```
In your renderer process, you can now call the ipcHandler and fetch data from SQLite:
```ts
const rows = await ipcRenderer.invoke('db-query', "SELECT * FROM Users");
```
The downside of SQLite (or SQL in general) is that it is lacking many features that are handful when using a database together with **UI based** applications. It is not possible to observe queries or document fields and there is no replication method to sync data with a server. This makes SQLite a good solution when you just want to store data on the client or process expensive SQL queries on the server, but it is not suitable for more complex operations like two-way replication, encryption, compression and so on. Also developer helpers like TypeScript type safety are totally out of reach.
## Follow up
- Learn how to use RxDB as database in electron with the [Quickstart Tutorial](./quickstart.md).
- Check out the [RxDB Electron example](https://github.com/pubkey/rxdb/tree/master/examples/electron)
- There is a followup list of other [client side database alternatives](./alternatives.md) that you can try to use with Electron.
---
## RxDB - The Real-Time Database for Node.js
# Node.js Database
[RxDB](https://rxdb.info) is a fast, reactive realtime NoSQL **database** made for **JavaScript** applications like Websites, [hybrid Apps](./articles/mobile-database.md), [Electron-Apps](./electron-database.md), [Progressive Web Apps](./articles/progressive-web-app-database.md) and **Node.js**. While RxDB was initially created to be used with UI applications, it has been matured and optimized to make it useful for pure server-side use cases. It can be used as embedded, local database inside of the Node.js JavaScript process, or it can be used similar to a database server that Node.js can connect to. The [RxStorage](./rx-storage.md) layer makes it possible to switch out the underlying storage engine which makes RxDB a very flexible database that can be optimized for many scenarios.
## Persistent Database
To get a "normal" database connection where the data is persisted to a file system, the RxDB real time database provides multiple [storage implementations](./rx-storage.md) that work in Node.js.
The [FoundationDB](./rx-storage-foundationdb.md) storage connects to a [FoundationDB](https://github.com/apple/foundationdb) cluster which itself is just a distributed key-value engine. RxDB adds the NoSQL query-engine, indexes and other features on top of it.
It scales horizontally because you can always add more servers to the FoundationDB cluster to increase the capacity.
Setting up a RxDB database is pretty simple. You import the FoundationDB RxStorage and tell RxDB to use that when calling `createRxDatabase`:
```typescript
import { createRxDatabase } from 'rxdb';
import { getRxStorageFoundationDB } from 'rxdb/plugins/storage-foundationdb';
const db = await createRxDatabase({
name: 'exampledb',
storage: getRxStorageFoundationDB({
apiVersion: 620,
clusterFile: '/path/to/fdb.cluster'
})
});
// add a collection
await db.addCollections({
users: {
schema: mySchema
}
});
// run a query
const result = await db.users.find({
selector: {
name: 'foobar'
}
}).exec();
```
Another alternative storage is the [SQLite RxStorage](./rx-storage-sqlite.md) that stores the data inside of a SQLite filebased database. The SQLite storage is faster than FoundationDB and does not require to set up a cluster or anything because SQLite directly stores and reads the data inside of the filesystem. The downside of that is that it only scales vertically.
```ts
import { createRxDatabase } from 'rxdb';
import {
getRxStorageSQLite,
getSQLiteBasicsNode
} from 'rxdb-premium/plugins/storage-sqlite';
import sqlite3 from 'sqlite3';
const myRxDatabase = await createRxDatabase({
name: 'path/to/database/file/foobar.db',
storage: getRxStorageSQLite({
sqliteBasics: getSQLiteBasicsNode(sqlite3)
})
});
```
Because the SQLite RxStorage is not free and you might not want to set up a FoundationDB cluster, there is also the option to use the [LokiJS RxStorage](./rx-storage-lokijs.md) together with the filesystem adapter. This will store the data as plain json in a file and load everything into memory on startup. This works great for small prototypes but it is not recommended to be used in production.
```ts
import { createRxDatabase } from 'rxdb';
const LokiFsStructuredAdapter = require('lokijs/src/loki-fs-structured-adapter.js');
import { getRxStorageLoki } from 'rxdb/plugins/storage-lokijs';
import sqlite3 from 'sqlite3';
const myRxDatabase = await createRxDatabase({
name: 'path/to/database/file/foobar.db',
storage: getRxStorageLoki({
adapter: new LokiFsStructuredAdapter()
})
});
```
Here is a performance comparison chart of the different storages (lower is better):
## RxDB as Node.js In-Memory Database
One of the easiest way to use RxDB in Node.js is to use the [Memory RxStorage](./rx-storage-memory.md). As the name implies, it stores the data directly **in-memory** of the Node.js JavaScript process. This makes it really fast to read and write data but of course the data is not persisted and will be lost when the nodejs process exits. Often the in-memory option is used when RxDB is used in unit tests because it automatically cleans up everything afterwards.
```ts
import { createRxDatabase } from 'rxdb';
import { getRxStorageMemory } from 'rxdb/plugins/storage-memory';
const db = await createRxDatabase({
name: 'exampledb',
storage: getRxStorageMemory()
});
```
Also notice that the [default memory limit](https://medium.com/geekculture/node-js-default-memory-settings-3c0fe8a9ba1) of Node.js is 4gb (might change of newer versions) so for bigger datasets you might want to increase the limit with the `max-old-space-size` flag:
```bash
# increase the Node.js memory limit to 8GB
node --max-old-space-size=8192 index.js
```
## Hybrid In-memory-persistence-synced storage
If you want to have the performance of an **in-memory database** but require persistency of the data, you can use the [memory-mapped storage](./rx-storage-memory-mapped.md). On database creation it will load all data into the memory and on writes it will first write the data into memory and later also write it to the persistent storage in the background. In the following example the FoundationDB storage is used, but any other RxStorage can be used as persistence layer.
```typescript
import { createRxDatabase } from 'rxdb';
import { getRxStorageFoundationDB } from 'rxdb/plugins/storage-foundationdb';
import { getMemoryMappedRxStorage } from 'rxdb-premium/plugins/storage-memory-mapped';
const db = await createRxDatabase({
name: 'exampledb',
storage: getMemoryMappedRxStorage({
storage: getRxStorageFoundationDB({
apiVersion: 620,
clusterFile: '/path/to/fdb.cluster'
})
})
});
```
While this approach gives you a database with great performance and persistent, it has two major downsides:
- The database size is limited to the memory size
- Writes can be lost when the Node.js process exists between a write to the memory state and the background persisting.
## Share database between microservices with RxDB
Using a local, embedded database in Node.js works great until you have to share the data with another Node.js process or another server at all.
To share the database state with other instances, RxDB provides two different methods. One is [replication](./replication.md) and the other is the [remote RxStorage](./rx-storage-remote.md).
The replication copies over the whole database set to other instances live-replicates all ongoing writes. This has the benefit of scaling better because each of your microservice will run queries on its own copy of the dataset.
Sometimes however you might not want to store the full dataset on each microservice. Then it is better to use the remote RxStorage and connect it to the "main" database. The remote storage will run all operations the main database and return the result to the calling database.
## FAQ
What is the best database to use with Node.js?
You must choose a database based on your project requirements. For simple server-side document storage you use MongoDB or CouchDB. For relational data you use PostgreSQL or MySQL. If you need realtime synchronization between your Node.js backend and client applications you use RxDB. RxDB provides local-first offline support and seamless data replication. You can combine RxDB with storage plugins like SQLite or FoundationDB to achieve high performance.
## Follow up on RxDB+Node.js
- Check out the [RxDB Nodejs example](https://github.com/pubkey/rxdb/tree/master/examples/node).
- If you haven't done yet, you should start learning about RxDB with the [Quickstart Tutorial](./quickstart.md).
- I created [a list of embedded JavaSCript databases](./alternatives.md) that you will help you to pick a database if you do not want to use RxDB.
- Check out the [MongoDB RxStorage](./rx-storage-mongodb.md) that uses MongoDB for the database connection from your Node.js application and runs the RxDB real time database on top of it.
---
## Local First / Offline First
Local-First (aka offline first) is a software paradigm where the software stores data locally on the client's device and must work as well offline as it does online.
To implement this, you have to store data at the client side, so that your application can still access it when the internet connection is lost.
This can be either done with complex caching strategies, or by using a local-first, [offline database](./articles/offline-database.md) (like [RxDB](https://rxdb.info)) that stores the data inside of a local database like [IndexedDB](./rx-storage-indexeddb.md) and replicates it from and to the backend in the background. This makes the local database, not the server, the gateway for all persistent changes in application state.
> **Offline first is not about having no internet connection**
:::note
I wrote a follow-up version of offline/first local first about [Why Local-First Is the Future and what are Its Limitations](./articles/local-first-future.md)
:::
While in the past, internet connection was unstable, things are changing especially for mobile devices.
[Mobile](./articles/mobile-database.md) networks have become better and having no internet becomes less common even in remote locations.
So if we did not care about offline first applications in the past, why should we even care now?
In the following I will point out why offline first applications are better, not because they support offline usage, but because of other reasons.
## UX is better without loading spinners
In 'normal' web applications, most user interactions like fetching, saving or deleting data, correspond to a request to the backend server. This means that each of these interactions require the user to await the unknown latency to and from a remote server while looking at a loading spinner.
In offline-first apps, the operations go directly against the local storage which happens almost instantly. There is no perceptible loading time and so it is not even necessary to implement a loading spinner at all. As soon as the user clicks, the UI represents the new state as if it was already changed in the backend.
## Multi-tab usage just works
Many, even big websites like Amazon, Reddit and Stack Overflow do not handle multi tab usage correctly. When a user has multiple tabs of the website open and does a login on one of these tabs, the state does not change on the other tabs.
On offline first applications, there is always exactly one state of the data across all tabs. Offline first databases (like RxDB) store the data inside of IndexedDb and **share the state** between all tabs of the same origin.
## Latency is more important than bandwidth
In the past, often the bandwidth was the limiting factor on determining the loading time of an application.
But while bandwidth has improved over the years, latency became the limiting factor.
You can always increase the bandwidth by setting up more cables or sending more Starlink satellites to space.
But reducing the latency is not so easy. It is defined by the physical properties of the transfer medium, the speed of light and the distance to the server. All of these three are hard to optimize.
Offline first applications benefit from that because sending the initial state to the client can be done much faster with more bandwidth. And once the data is there, we no longer have to care about the latency to the backend server because you can run near [zero](./articles/zero-latency-local-first.md) latency queries locally.
## Realtime comes for free
Most websites lie to their users. They do not lie because they display wrong data, but because they display **old data** that was loaded from the backend at the time the user opened the site.
To overcome this, you could build a realtime website where you create a websocket that streams updates from the backend to the client. This means work. Your client needs to tell the server which page is currently opened and which updates the client is interested in. Then the server can push updates over the websocket and you can update the UI accordingly.
With offline first applications, you already have a realtime replication with the backend. Most offline first databases provide some concept of changestream or data subscriptions and with [RxDB](https://github.com/pubkey/rxdb) you can even directly subscribe to query results or single fields of documents. This makes it easy to have an always updated UI whenever data on the backend changes.
## Scales with data size, not with the amount of user interaction
On normal applications, each user interaction can result in multiple requests to the backend server which increase its load.
The more users interact with your application, the more backend resources you have to provide.
Offline first applications do not scale up with the amount of user actions but instead they scale up with the amount of data.
Once that data is transferred to the client, the user can do as many interactions with it as required without connecting to the server.
## Modern apps have longer runtimes
In the past you used websites only for a short time. You open it, perform some action and then close it again. This made the first load time the important metric when evaluating page speed.
Today web applications have changed and with it the way we use them. Single page applications are opened once and then used over the whole day. Chat apps, email clients, [PWAs](./articles/progressive-web-app-database.md) and hybrid apps. All of these were made to have long runtimes.
This makes the time for user interactions more important than the initial loading time. Offline first applications benefit from that because there is often no loading time on user actions while loading the initial state to the client is not that relevant.
## You might not need REST
On normal web applications, you make different requests for each kind of data interaction.
For that you have to define a swagger route, implement a route handler on the backend and create some client code to send or fetch data from that route. The more complex your application becomes, the more REST routes you have to maintain and implement.
With offline first apps, you have a way to hack around all this cumbersome work. You just replicate the whole state from the server to the client. The replication does not only run once, you have a **realtime replication** and all changes at one side are automatically there on the other side. On the client, you can access every piece of state with a simple database query.
While this of course only works for amounts of data that the client can load and store, it makes implementing prototypes and simple apps much faster.
## You might not need Redux
Data is hard, especially for UI applications where many things can happen at the same time.
The user is clicking around. Stuff is loaded from the server. All of these things interact with the global state of the app.
To manage this complexity it is common to use state management libraries like Redux or MobX. With them, you write all this lasagna code to wrap the mutation of data and to make the UI react to all these changes.
On offline first apps, your global state is already there in a single place stored inside of the local database.
You do not have to care whether this data came from the UI, another tab, the backend or another device of the same user. You can just make writes to the database and fetch data out of it.
## FAQ
What are the best offline-first mobile databases for unreliable network environments?
RxDB excels as an offline-first mobile database for unreliable network environments. You build mobile applications using the local-first paradigm to store data directly on the device. RxDB ensures full read and write capabilities regardless of current internet connection status. The sync engine automatically resolves conflicts and pushes local changes to the server once connectivity is restored. This approach eliminates loading spinners and guarantees a consistent user experience during network fluctuations.
## Follow up
- Learn how to store and query data with RxDB in the [RxDB Quickstart](./quickstart.md)
- [Downsides of Offline First](./downsides-of-offline-first.md)
- I wrote a follow-up version of offline/first local first about [Why Local-First Is the Future and what are Its Limitations](./articles/local-first-future.md)
---
## Partial Sync with RxDB
Suppose you're building a Minecraft-like voxel game where the world can expand in every direction. Storing the entire map locally for offline use is impossible because the dataset could be massive. Yet you still want a local-first design so players can edit the game world offline and sync back to the server later.
## Idea: One Collection, Multiple Replications
You might define a single RxDB collection called `db.voxels`, where each document represents a block or "voxel" (with fields like id, chunkId, coordinates, and type). With RxDB you can, instead of setting up _one_ replication that tries to fetch _all_ voxels, you create **separate replication states** for each _chunk_ of the world the player is currently near.
When the player enters a particular chunk (say `chunk-123`), you **start a replication** dedicated to that chunk. On the server side, you have endpoints to **pull** only that chunk's voxels (e.g., GET `/api/voxels/pull?chunkId=123`) and **push** local changes back (e.g., POST `/api/voxels/push?chunkId=123`). RxDB handles them similarly to any other offline-first setup, but each replication is filtered to only that chunk's data.
When the player leaves `chunk-123` and no longer needs it, you **stop** that replication. If the player moves to `chunk-124`, you start a new replication for chunk 124. This ensures the game only downloads and syncs data relevant to the player's immediate location. Meanwhile, all edits made offline remain safely stored in the local database until a network connection is available.
```ts
const activeReplications = {}; // chunkId -> replicationState
function startChunkReplication(chunkId) {
if (activeReplications[chunkId]) return;
const replicationId = 'voxels-chunk-' + chunkId;
const replicationState = replicateRxCollection({
collection: db.voxels,
replicationIdentifier: replicationId,
pull: {
async handler(checkpoint, limit) {
const res = await fetch(
`/api/voxels/pull?chunkId=${chunkId}&cp=${checkpoint}&limit=${limit}`
);
/* ... */
}
},
push: {
async handler(changedDocs) {
const res = await fetch(`/api/voxels/push?chunkId=${chunkId}`);
/* ... */
}
}
});
activeReplications[chunkId] = replicationState;
}
function stopChunkReplication(chunkId) {
const rep = await activeReplications[chunkId];
if (rep) {
rep.cancel();
delete activeReplications[chunkId];
}
}
// Called whenever the player's location changes;
// dynamically start/stop replication for nearby chunks.
function onPlayerMove(neighboringChunkIds) {
neighboringChunkIds.forEach(startChunkReplication);
Object.keys(activeReplications).forEach(cid => {
if (!neighboringChunkIds.includes(cid)) {
stopChunkReplication(cid);
}
});
}
```
## Diffy-Sync when Revisiting a Chunk
An added benefit of this multi-replication-state design is checkpointing. Each replication state has a unique "replication identifier," so the next time the player returns to `chunk-123`, the local database knows what it already has and only fetches the differences without the need to re-download the entire chunk.
## Partial Sync in a Local-First Business Application
Though a voxel world is an intuitive example, the same technique applies in enterprise scenarios where data sets are large but each user only needs a specific subset. You could spin up a new replication for each "permission group" or "region," so users only sync the records they're allowed to see. Or in a CRM, the replication might be filtered by the specific accounts or projects a user is currently handling. As soon as they switch to a different project, you stop the old replication and start one for the new scope.
This **chunk-based** or **scope-based** replication pattern keeps your local storage lean, reduces network overhead, and still gives users the offline, instant-feedback experience that local-first apps are known for. By dynamically creating (and canceling) replication states, you retain tight control over bandwidth usage and make the infinite (or very large) feasible. In a production app you would also "flag" the entities (with a `pull.modifier`) by which replication state they came from, so that you can clean up the parts that you no longer need.
---
## React Native Database - Sync & Store Like a Pro
import {Steps} from '@site/src/components/steps';
import {Tabs} from '@site/src/components/tabs';
# React Native Database
If you are looking for a **React Native Database**, you usually want three things:
1. **Persistence**: Store data locally on the device so the app works **[offline](./offline-first.md)**.
2. **Reactivity**: Automatically update the UI when data changes.
3. **Sync**: Replicate data with a backend server in real-time.
RxDB covers all of these requirements out of the box. It is a [local-first](./articles/local-first-future.md) NoSQL database that runs deeply integrated with React Native, giving you the power of a full featured database engine inside your mobile app.
## The Storage Layer: SQLite
React Native does not have a native database engine. To store data persistently and efficiently, RxDB uses **[SQLite](./rx-storage-sqlite.md)** under the hood. SQLite is available on all mobile platforms (iOS, Android) and offers great performance.
RxDB abstracts the complex SQL commands away and provides a simple, **[NoSQL JSON document API](./rx-database.md)** that is easy to use for JavaScript developers.
We recommend different SQLite adapters depending on your environment:
### React Native CLI
For bare React Native projects, use `react-native-quick-sqlite`. It uses JSI (JavaScript Interface) to communicate directly with C++, effectively bypassing the slow React Native Bridge.
**Installation**:
```bash
npm install rxdb rxjs react-native-quick-sqlite
```
**Configuration**:
```ts
import { createRxDatabase } from 'rxdb';
import { getRxStorageSQLite, getSQLiteBasicsQuickSQLite } from 'rxdb/plugins/storage-sqlite';
import { open } from 'react-native-quick-sqlite';
const db = await createRxDatabase({
name: 'mydatabase',
storage: getRxStorageSQLite({
sqliteBasics: getSQLiteBasicsQuickSQLite(open)
}),
multiInstance: false,
ignoreDuplicate: true
});
```
### Expo Go
If you are using Expo, use the official `expo-sqlite` module.
**Installation**:
```bash
npx expo install expo-sqlite
npm install rxdb rxjs
```
**Configuration**:
```ts
import { createRxDatabase } from 'rxdb';
import { getRxStorageSQLite, getSQLiteBasicsExpoSQLiteAsync } from 'rxdb/plugins/storage-sqlite';
import * as SQLite from 'expo-sqlite';
const db = await createRxDatabase({
name: 'mydatabase',
storage: getRxStorageSQLite({
sqliteBasics: getSQLiteBasicsExpoSQLiteAsync(SQLite.openDatabaseAsync)
}),
multiInstance: false,
ignoreDuplicate: true
});
```
## React Integration
RxDB is deeply integrated with React. It provides hooks that make fetching data and subscribing to changes effortless.
### 1. Provide the Database
Wrap your application with the `RxDatabaseProvider`.
```tsx
import { RxDatabaseProvider } from 'rxdb/plugins/react';
export default function App() {
// ... create db instance
return (
);
}
```
### 2. Observe Data
Use the `useRxQuery` hook (or `useLiveRxQuery` shortcut) to fetch data. The component will **automatically re-render** whenever the data in the database changes. You don't need to manually subscriptions or handling event listeners.
```tsx
import { useRxCollection, useLiveRxQuery } from 'rxdb/plugins/react';
function TaskList() {
const collection = useRxCollection('tasks');
// This hook automatically updates 'tasks' whenever the query result changes
const { result: tasks } = useLiveRxQuery(
collection.find({
selector: {
done: { $eq: false }
},
sort: [{ createdAt: 'asc' }]
})
);
return (
{item.title}}
keyExtractor={item => item.id}
/>
);
}
```
### 3. Signals (Performance Mode)
For high-performance applications with frequent data updates, re-rendering the entire React component might be too slow.
RxDB supports **Signals** (via `@preact/signals-react` or similar) to pinpoint updates directly to the DOM nodes.
```tsx
// Enable the signals plugin once
import { addRxPlugin } from 'rxdb';
import { RxDBReactivityPreactSignalsPlugin } from 'rxdb/plugins/reactivity-preact-signals';
addRxPlugin(RxDBReactivityPreactSignalsPlugin);
// ... in your component
const signals = collection.find().$$; // Returns a Signal
```
Using signals allows you to update only the specific text node that changed, keeping your UI running at 60fps even with massive data flux.
## Sync with Backend
A local database is useful, but a synchronized database is powerful.
RxDB provides a robust replication protocol that can sync with **any backend**.
It has dedicated plugins for popular backend solutions:
- **[Supabase / Postgres](./replication-supabase.md)**
- **[Firebase / Firestore](./replication-firestore.md)**
- **[GraphQL](./replication-graphql.md)**
- **[CouchDB](./replication-couchdb.md)**
For custom backends, you can implement the **[simple HTTP replication](./replication-http.md)** protocol.
### Example: Sync with Supabase
Syncing is set-and-forget. You start the replication, and RxDB handles the rest (pulling changes, pushing writes, handling conflict resolution).
```ts
import { replicateSupabase } from 'rxdb/plugins/replication-supabase';
const replicationState = replicateSupabase({
replicationIdentifier: 'my-sync',
collection: db.tasks,
supabaseClient: supabase,
pull: {},
push: {},
});
```
Because RxDB handles the sync layer, you can build your app as if it were a purely local application. All reads and writes happen against the local SQLite database instantly, while the replication happens in the background. This is the essence of **Local-First** development.
## Comparison with Alternatives
How does RxDB compare to other React Native database solutions?
| Feature | **AsyncStorage** | **SQLite** (Raw) | **Realm** | **Firestore** (SDK) | **RxDB** |
|:--- |:--- |:--- |:--- |:--- |:--- |
| **Type** | Key-Value Store | Relational (SQL) | Object Store | Cloud Document Store | **NoSQL Document Store** |
| **Reactivity** | β None | β Manual events | β Local listeners | β Real-time listeners | β **Hooks / Signals / RxJS** |
| **Persistence** | β File (Slow) | β File (Generic) | β Custom File | β οΈ Partial Cache | β **SQLite / File** |
| **Sync** | β Manual | β Manual | β Realm Sync only | β Firebase only | β **Any Backend** |
| **Query Engine** | β None | β SQL Strings | β Custom API | β Limited | β **Mango JSON Query** |
| **Schema** | β None | β SQL Schema | β Class Schema | β Loose | β **[JSON Schema](./rx-schema.md)** |
| **Migration** | β Manual | β Manual SQL | β Migration API | β None | β **Automatic** |
### Summary
- **AsyncStorage**: Good for simple key-value pairs (like settings). Too slow for data.
- **SQLite**: Great foundation, but requires writing raw SQL and manual reactivity/sync.
- **Realm**: Fast object store, but locks you into the MongoDB ecosystem for sync. Realm was deprecated in 2024 ([source](https://github.com/realm/realm-swift/discussions/8680)).
- **Firestore**: Easy networked DB, but poor offline support (cannot start offline) and latency issues.
- **RxDB**: Combines the performance of local SQLite with the ease of NoSQL, automatic reactivity, and backend-agnostic synchronization.
---
**Ready to start?**
Check out the **[React Native Example Project](https://github.com/pubkey/rxdb/tree/master/examples/react-native)** or read the **[Quickstart Guide](./quickstart.md)**.
---
## React
import {Tabs} from '@site/src/components/tabs';
import {Steps} from '@site/src/components/steps';
# React
RxDB provides first-class support for both React and React Native via a dedicated React integration. This integration makes it possible to use RxDB inside functional components using React Context and hooks, without manually subscribing to observables or managing cleanup logic.
The same APIs work in **React** for the web and in [React Native](./react-native-database.md). The only difference between platforms is the [storage](./rx-storage.md) and environment setup. The React integration itself behaves identically in both.
## General concept
RxDB is internally reactive and emits changes via RxJS observables. React and React Native, however, are render-driven frameworks that rely on component state and hooks. The RxDB React integration bridges these two models by exposing a small set of hooks that translate RxDB state into React renders.
Instead of hiding reactivity behind configuration flags, the integration uses explicit hooks. This makes component behavior predictable, avoids accidental subscriptions, and keeps performance characteristics easy to reason about.
## Usage
### Installation
Install RxDB and React as usual:
```bash
npm install rxdb react react-dom
```
### Database creation
Database creation is not part of the React integration itself. RxDB is created in the same way as in non-React applications, including storage selection, plugins, [replication](./replication.md), hooks, and [schema](./rx-schema.md) definitions.
This separation is intentional. React components should never be responsible for creating or configuring the database. They should only consume it.
```ts
import {
createRxDatabase,
addRxPlugin
} from 'rxdb';
import {
getRxStorageLocalstorage
} from 'rxdb/plugins/storage-localstorage';
async function getDatabase() {
const db = await createRxDatabase({
name: 'heroesreactdb',
storage: getRxStorageLocalstorage()
});
await db.addCollections({
heroes: {
schema: myRxSchema
}
});
/**
* Do other stuff here
* like setting up middleware
* or starting replication.
*/
return db;
}
```
### Providing the database
To use RxDB in a React or React Native application, the database instance must be provided via a context. This is done using [RxDatabaseProvider](./rx-database.md).
The database itself is created outside of React, usually in a separate module. The provider is only responsible for making the database available to components once it has been initialized.
```tsx
import React, { useEffect, useState } from 'react';
import { RxDatabaseProvider } from 'rxdb/plugins/react';
import { getDatabase } from './Database';
const App = () => {
const [database, setDatabase] = useState();
useEffect(() => {
const initDb = async () => {
const db = await getDatabase();
setDatabase(db);
};
initDb();
}, []);
if (database == null) {
return
Loading RxDB database...
;
}
return (
{/* your application */}
);
};
export default App;
```
### Accessing collections
```ts
import { useRxCollection } from 'rxdb/plugins/react';
const collection = useRxCollection('heroes');
```
The hook returns the collection once it becomes available. During the initial render, the value may be `undefined`, so components must handle this case.
This hook does not subscribe to any data. It only provides access to the collection instance.
### Queries
To render query results in your component, use the `useRxQuery` hook.
```tsx
import { useRxQuery } from 'rxdb/plugins/react';
const query = {
collection: 'heroes',
query: {
selector: {},
sort: [{ name: 'asc' }]
}
};
const HeroCount = () => {
const { results, loading } = useRxQuery(query);
if (loading) {
return Loading...;
}
return Total heroes: {results.length};
};
```
The query is executed when the component renders. If the component re-renders, the query may be re-executed, but changes in the underlying data will not automatically trigger updates.
This hook is well suited for static views, server-side rendering, and cases where live updates are not required.
### Live queries
Most React applications require views that automatically update when the database changes. For this purpose, RxDB provides the `useLiveRxQuery` hook.
The hook accepts a query description object and returns the current results together with a loading state.
```tsx
import { useLiveRxQuery } from 'rxdb/plugins/react';
const query = {
collection: 'heroes',
query: {
selector: {},
sort: [{ name: 'asc' }]
}
};
const HeroList = () => {
const { results, loading } = useLiveRxQuery(query);
if (loading) {
return Loading...;
}
return (
{results.map(hero => (
{hero.name}
))}
);
};
```
The component automatically re-renders whenever the query result changes. Subscriptions are created when the component mounts and are cleaned up automatically when the component unmounts.
The returned documents are fully reactive RxDB documents and can be modified or removed directly.
## React Native compatibility
All hooks and providers described on this page work the same way in React Native. The React integration does not rely on any browser-specific APIs.
The only platform-specific part of a React Native setup is database creation, where a different storage plugin is typically used. Once the database is created, the React integration behaves identically.
## Signals
In addition to the React hooks shown on this page, RxDB also supports alternative reactivity models such as [signals](./reactivity.md#react). RxDB's core reactivity system can be configured to expose reactive values using different primitives instead of RxJS observables, which makes it possible to integrate RxDB with signal-based approaches in React or other frameworks. This is an advanced capability and is independent of the React integration described here. For more details about how RxDB's reactivity system works and how custom reactivity can be configured, see the [Reactivity documentation](./reactivity.md).
## Follow Up
- RxDB includes a full [React example application](https://github.com/pubkey/rxdb/tree/master/examples/react) that demonstrates the patterns described on this page, including database creation outside of React, usage of `RxDatabaseProvider`, and data access via `useRxQuery`, `useLiveRxQuery`, and `useRxCollection`.
- A corresponding [React Native example](https://github.com/pubkey/rxdb/tree/master/examples/react-native) is also available and shows the same integration concepts applied in a mobile environment, with only the platform-specific storage and setup differing.
---
## RxDB 10.0.0 - Built for the Future
# 10.0.0
One year after version `9.0.0` we now have RxDB version `10.0.0`.
The main goal of version 10 was to change things that make RxDB ready for the future.
Notice that I use major releases to bundle stuff that breaks the RxDB usage in your project, not to add new features.
## The main thing first
In the past, RxDB was build around Pouchdb. Before I started making RxDB I tried to solve the problems of my current project with other existing databases out there. I evaluated all of them and then started using Pouchdb and added many features via plugin. Then I realised it will be easier to create a separate project that wraps around Pouchdb, that was RxDB. Back then pouchdb was the most major browser database out there and it was well maintained and had a big community.
But in the last 5 years, things have changed. A big part of the RxDB users do not use couchdb in the backend and do not need the couchdb replication.
Therefore they do not really need the overhead with revision handling that slows down the performance of pouchdb. Also there where many other problems with using pouchdb. It is not actively developed, many bugs are not fixed and no new features get added. Also there are many unsolved problems like how to finally delete document data or how to replicate more than 6 databases at the same time, how to use replication without attachments data, and so on...
So for this release, I abstracted all parts that we use from pouchdb into the `RxStorage` interface. RxDB works on top of any implementation of the `RxStorage` interface. This means it is now possible to use RxDB together with other underlying storages like SQLite, PostgreSQL, Minimongo, MongoDB, and so on, as long as someone writes the `RxStorage` class for it.
This means, to create a `RxDatabase` you have to pass the storage class instead of pouchdb specific settings:
```ts
// import pouchdb specific stuff and add pouchdb adapters
import {
addPouchPlugin,
getRxStoragePouch
} from 'rxdb/plugins/pouchdb';
// IMPORTANT: Do not use addRxPlugin to add pouchdb adapter, instead use addPouchPlugin
addPouchPlugin(require('pouchdb-adapter-memory'));
import {
addRxPlugin,
createRxDatabase,
randomCouchString,
} from 'rxdb/plugins/core';
// create the database with the storage creator.
const db = await createRxDatabase({
name: 'mydatabase',
storage: getRxStoragePouch('memory'),
});
```
To access the internal `pouch` instance of a collection, you have to go over the `storageInstance`:
```ts
const pouch = myRxCollection.storageInstance.internals.pouch;
```
## Other breaking changes
### Primary key is required
In the past, using a primary key was optional. When no primary key was defined, RxDB filled up the `_id` field with an uuid-like string which was then used as primary. When I researched on github how people use RxDB, I found out that many use a secondary index for what should be the primary key.
Also having the primary key optional, caused much confusing when using RxDB with typescript.
So now the primary key MUST be set when creating a schema for RxDB.
Also the primary key is defined with the `primaryKey` property at the top level of the schema. This ensures that typescript will complain if no `primaryKey` is defined.
```ts
// when using the type `RxJsonSchema` the `DocType` is now required
const mySchema: RxJsonSchema = {
version: 0,
primaryKey: 'passportId',
type: 'object',
properties: {
passportId: {
type: 'string'
}
},
// primaryKey is always required
required: ['passportId']
}
```
### Attachment data must be Blob or Buffer
In the past, an `RxAttachment` could be stored with `Blob`, `Buffer` and `string` data. If a `string` was passed, pouchdb internally transformed the data to a `Blob` or `Buffer`, depending on in which environment it is running.
This behavior caused much trouble and weird edge cases because of how the data is transformed from and to `string`.
So now you can only store `Blob` or `Buffer` as attachment data. `string` is no longer allowed. You can still transform a string to a Blob or Buffer by yourself and then store it.
```ts
import { blobBufferUtil } from 'rxdb';
const attachment = await myDocument.putAttachment(
{
id: 'cat.txt',
data: blobBufferUtil.createBlobBuffer('miau', 'text/plain')
type: 'text/plain'
}
);
```
Also `putAttachment()` now defaults to `skipIfSame=true`. This means when you write attachment data that already is exactly the same in the database, no write will be done.
### Outgoing data is now readonly and deep-frozen
RxDB often uses outgoing data also in the internals. For example the result of a query is not only send to the user, but also used inside of RxDB's query-change-detection. To ensure that mutation of the outgoing data is not changing internal stuff, which would cause strange bugs, outgoing data was always deep-cloned before handing it out to the user. This is a common practice on many javascript libraries.
The problem is that deep-cloning big objects can be very CPU/Memory expensive. So instead of doing a deep-clone, RxDB does now assume that outgoing data is **immutable**. If the users wants to modify that data, it has to be deep-cloned by the user.
To ensure immutability, RxDB runs a [deep-freeze](https://developer.mozilla.org/de/docs/Web/JavaScript/Reference/Global_Objects/Object/freeze) in the dev-mode (about same expensive as deep clone). Also typescript will throw a build-time error because we use `ReadonlyArray` and `readonly` to define outgoing data immutable. In production-mode, there will be nothing besides typescript that ensures immutability to have best performance.
```ts
const data = myRxDocument.toJSON();
data.foo = bar; // This does NOT work!
// instead clone the data before changing it
import { clone } from 'rxjs';
const clonedData = clone(data);
data.foo = bar; // This works!
```
### The in-memory plugin does no longer work.
The in-memory plugin was used to spawn in-memory collections on top of a normal `RxCollection`. The benefit is to have the data replicated into the memory of the javascript runtime, which allows for faster queries.
After doing many tests and observations, I found out that the in-memory plugin was slow. Really slow, even slower then just using the indexeddb adapter in the browser. You can reproduce my observations at the event-reduce testpage. Here you can see that random-writes+query are slower on the [memory-adapter](https://pubkey.github.io/event-reduce/?tech=pouchdb:memory) then on [indexeddb](https://pubkey.github.io/event-reduce/?tech=pouchdb:indexeddb).
The reason for this are the big abstraction layers. Pouchdb uses the adapter system. The memory adapter uses the [leveldown abstraction layer](https://github.com/Level/levelup). Each write/read goes to the [memdown module](https://github.com/Level/memdown).
So the in-memory plugin is not working for now. In the future it will be reimplemented in a custom memory based `RxStorage` class.
:::note
You can of course still use the pouchdb `memory` adapter as usual. It is not affected by this change.
:::
## What else is a breaking change?
- Removed the deprecated `atomicSet()`, use `atomicPatch()` instead.
- Removed the deprecated `RxDatabase.collection()` use `RxDatabase().addCollections()` instead.
- Removed plugin hook `preCreatePouchDb` because it is no longer needed.
- Removed the `watch-for-changes` plugin. We now overwrite pouchdbs `bulkDocs` method to generate events. This is faster and more reliable.
- Removed the `adapter-check` plugin. (The function `adapterCheck` is move to the pouchdb plugin).
- Calling `RxDatabase.server()` now returns a promise that resolves when the server is started up.
- Changed the defaults of `PouchDBExpressServerOptions` from the `server()` method, by default we now store logs in the `tmp` folder and the config is in memory.
- Renamed `replication`-plugin to [replication-couchdb](../replication-couchdb.md) to be more consistent in naming like with `replication-graphql`
- For the same reason, renamed `RxCollection().sync()` to `RxCollection().syncCouchDB()`
- Renamed the functions of the json import/export plugin to be less confusing.
- `dump()` is now `exportJSON()`
- `importDump()` is now `importJSON()`
- `RxCollection` uses a separate pouchdb instance for local documents, so that they can persist during migrations.
- A JsonSchema must have the `required` array at the top level and it must contain the primary key.
## New features
### Composite primary key
You can now use a composite primary key for the schema where you can join different properties of the document data to create a primary key.
```javascript
const mySchema = {
keyCompression: true, // set this to true, to enable the keyCompression
version: 0,
title: 'human schema with composite primary',
primaryKey: {
// where should the composed string be stored
key: 'id',
// fields that will be used to create the composed key
fields: [
'firstName',
'lastName'
],
// separator which is used to concat the fields values.
separator: '|'
}
type: 'object',
properties: {
id: {
type: 'string'
},
firstName: {
type: 'string'
},
lastName: {
type: 'string'
}
},
required: [
'id',
'firstName',
'lastName'
]
};
```
## For the future
With these changes, RxDB is now ready for the future plans:
- I want to replace the `revision` handling of documents with conflict resolution strategies that can always directly resolve conflicts instead of maintaining the revision tree.
- Implement different implementations for `RxStorage`. I will first work on a memory based version. I am in good hope that the community will create other implementations depending on their needs.
## You can help!
There are many things that can be done by **you** to improve RxDB:
- Check the [BACKLOG](https://github.com/pubkey/rxdb/blob/master/orga/BACKLOG.md) for features that would be great to have.
- Check the [breaking backlog](https://github.com/pubkey/rxdb/blob/master/orga/before-next-major.md) for breaking changes that must be implemented in the future but where I did not had the time yet.
- Check the [todos](https://github.com/pubkey/rxdb/search?q=todo) in the code. There are many small improvements that can be done for performance and build size.
- Review the code and add tests. I am only a single dude with a laptop. My code is not perfect and much small improvements can be done when people review the code and help me to clarify undefined behaviors.
- Improve the documentation. In the last user survey many users told me that the documentation is not good enough. But I reviewed the docs and could not find clear flaws. The problem is that I am way to deep into RxDB so that I am not able to understand which documentation a newcomer to the project needs. Likely I assume too much knowledge or focus writing about the wrong parts.
- Update the [example projects](https://github.com/pubkey/rxdb/tree/master/examples) many of them are outdated and need updates.
## Discuss!
Please [discuss here](https://github.com/pubkey/rxdb/issues/3279).
---
## RxDB 11 - WebWorker Support & More
# 11.0.0
The last major release was only about [6 month ago](./10.0.0.md). But to further improve RxDB, it was necessary to make some more breaking changes and release the next major version.
In the last version `10.0.0` the storage layer was abstracted in a way to make it possible to not only use PouchDB as storage, but instead we can use different storage engines like the one based on [LokiJS](../rx-storage-lokijs.md) or any custom implementation of the `RxStorage` interface.
In the new version `11.0.0` the focus is on making it possible to put the RxStorage into a **WebWorker** to take CPU load from the main process into the worker's process. This can improve the perceived performance of your application, especially when you have to handle many documents or when you need the main process CPU cycles to manage the DOM with your frontend framework.
## Worker plugin
Performance was always something RxDB had a struggle with. Not because RxDB itself is slow, but because the underlying storage engine (mostly PouchDB) or [IndexedDB is slow](../slow-indexeddb.md). This never was a problem for 'normal' applications that have to store some documents. But on big applications with much data, there was a bottleneck.
With the Worker plugin, you can move the `RxStorage` out of the main JavaScript process. This makes it pretty easy to utilize more than one CPU core and speed up your application.
```ts
// worker.ts
import { wrappedRxStorage } from 'rxdb/plugins/worker';
import { getRxStorageLoki } from 'rxdb/plugins/storage-lokijs';
wrappedRxStorage({
storage: getRxStorageLoki()
});
```
```ts
// main process
import { createRxDatabase } from 'rxdb/plugins/core';
import { getRxStorageWorker } from 'rxdb/plugins/worker';
const database = await createRxDatabase({
name: 'mydatabase',
storage: getRxStorageWorker(
{
workerInput: 'path/to/worker.js'
}
)
});
```
The whole documentation about the worker plugin can be found [here](../rx-storage-worker.md).
## Transpile `async`/`await` to promises instead of generators
The RxDB source-code is transpiled from TypeScript to es5/es6 JavaScript code via babel.
In the past we transpiled the `async` and `await` keywords with the babel plugin `plugin-transform-async-to-generator`.
Now we use the [babel-plugin-transform-async-to-promises](https://github.com/rpetrich/babel-plugin-transform-async-to-promises) plugin instead.
It transpiles `async`/`await` into native `Promise`s instead of using JavaScript generators. This has shown to decrease build size by about 10% and also improves the performance.
## Removed deprecated `received` methods
In the past there was a typo in all getters and methods that are called `received`.
This was renamed to `received` and all mistyped methods have been deprecated.
We now removed all deprecated methods, so you have to use the correctly spelled methods instead.
[See #3392](https://github.com/pubkey/rxdb/pull/3392)
## All internal events are handled as bulks
All events that are generated from writing to the storage instance are now handled in bulks instead of each event for its own. This has shown to save performance when the events are send over a data layer like a `WebWorker` or the `BroadcastChannel`. This change only affects you if you have created custom RxDB plugins.
## RxStorage interface changes
To make the RxStorage abstraction compatible with Webworkers, we had to do some changes. These will only affect you if you use a custom RxStorage implementation.
- The non async functions `prepareQuery`, `getSortComparator` and `getQueryMatcher` have been moved out of `RxStorageInstance` into the `statics` property of `RxStorage`. This makes it possible to split the code when using the worker plugin. You only need to load the static methods at the main process, and the whole storage engine is only loaded inside of the worker.
- All data communication with the RxStorage now happens only via plain JSON objects. Instead of returning a JavaScript `Map` or `Set`, only [JSON datatypes](https://www.w3schools.com/js/js_json_datatypes.asp) are allowed. This makes it easier to properly serialize the data when transferring it over to or from a WebWorker.
- Events that are created from a write operation, must be emitted **before** the write operation resolves. This ensures that RxDB always knows about all events before it runs another operation. So when you do an insert and a query directly after the insert, the query will return the correct results.
- The meta data `digest` and `length` of attachments is now created by RxDB, not by the RxStorage. [#3548](https://github.com/pubkey/rxdb/issues/3548)
- Added the statics `hashKey` property to identify the used hash function.
## Removed the `no-validate` plugin.
In the past, RxDB required you to add one schema validation plugin. For production, it was useful to not have any schema validation for better performance and a smaller build size. For that, the `no-validate` plugin could be added which was just a dummy plugin that did no do any validation. To remove this unnecessary complexity, RxDB no longer requires you to add a validation plugin. Therefore the no-validate plugin is now removed as it is no longer needed.
## Other changes
- The LokiJS RxStorage no longer uses the `IdleQueue` to determine if the database is idle. Because LokiJS is in-memory, we can just wait for CPU idleness via [requestIdleCallback()](https://developer.mozilla.org/en-US/docs/Web/API/Window/requestIdleCallback)
- Bugfix: Do not throw an error when database is destroyed while a GraphQL replication is running.
- Compound primary key migration throws "Value of primary key(s) cannot be changed" [#3546](https://github.com/pubkey/rxdb/pull/3546)
- Allow `_id` as primaryKey [#3562](https://github.com/pubkey/rxdb/pull/3562) Thanks [@SuperKirik](https://github.com/SuperKirik)
- LokiJS: Remote operations do never resolve when remote instance was leader and died.
## Migration from `10.x.x`
The migration should be pretty easy. Nothing in the datalayer has been changed, so you can use the stored data of v10 together with the new v11 RxDB.
## You can help!
There are many things that can be done by **you** to improve RxDB:
- Check the [BACKLOG](https://github.com/pubkey/rxdb/blob/master/orga/BACKLOG.md) for features that would be great to have.
- Check the [breaking backlog](https://github.com/pubkey/rxdb/blob/master/orga/before-next-major.md) for breaking changes that must be implemented in the future but where I did not had the time yet.
- Check the [todos](https://github.com/pubkey/rxdb/search?q=todo) in the code. There are many small improvements that can be done for performance and build size.
- Review the code and add tests. I am only a single human with a laptop. My code is not perfect and much small improvements can be done when people review the code and help me to clarify undefined behaviors.
- Improve the documentation. In the last user survey many users told me that the documentation is not good enough. But I reviewed the docs and could not find clear flaws. The problem is that I am way to deep into RxDB so that I am not able to understand which documentation a newcomer to the project needs. Likely I assume too much knowledge or focus writing about the wrong parts.
- Update the [example projects](https://github.com/pubkey/rxdb/tree/master/examples) many of them are outdated and need updates.
## Discuss!
Please [discuss here](https://github.com/pubkey/rxdb/issues/3555).
---
## RxDB 12.0.0 - Clean, Lean & Mean
# [RxDB](https://rxdb.info/) 12.0.0
For the last few months, I worked hard on the new RxDB version 12 release. I mostly focused on performance related features and refactored much of the code.
## Removed the `core` plugin
In the past, RxDB exported all bundled plugins when doing `import from 'rxdb';`.
This increased the bundle size, so optionally people could `import from 'rxdb/plugins/core';` to create a custom build that only contains the plugin that they really need.
But very often this lead to accidental imports of `'rxdb'`. For example, when the code editor auto imported methods.
So now, the default `import from 'rxdb';` only exports RxDB core. Every plugin must be imported afterwards if needed.
## Unified the replication primitives and the GraphQL replication plugin
Most of the GraphQL replication code has been replaced by using the replication primitives plugin internally.
This means many bugs and undefined behavior that was already fixed in the replication primitives, are now also fixed in the GraphQL replication.
Also, the GraphQL replication now runs `push` in bulk. This means you either have to update your backend to accept bulk mutations, or set `push.batchSize: 1` and transform the array into a single document inside `push.queryBuilder()`.
## Added the cleanup plugin
To make replication work, and for other reasons, RxDB has to keep deleted documents in storage.
This ensures that when a client is offline, the deletion state is still known and can be replicated with the backend when the client goes online again.
Keeping too many deleted documents in the storage can slow down queries or fill up too much disk space.
With the [cleanup plugin](https://rxdb.info/cleanup.html), RxDB will run cleanup cycles that clean up deleted documents when it can be done safely.
## Allow to set a specific index
By default, the query will be sent to RxStorage, where a query planner will determine which one of the available indexes must be used.
But the query planner cannot know everything and sometimes will not pick the most optimal index.
To improve query performance, you can specify which index must be used, when running the query.
```ts
const queryResults = await myCollection
.find({
selector: {
age: {
$gt: 18
},
gender: {
$eq: 'm'
}
},
/**
* Because the developer knows that 50% of the documents are 'male',
* but only 20% are below age 18,
* it makes sense to enforce using the ['gender', 'age'] index to improve performance.
* This could not be known by the query planner which might have chosen ['age', 'gender'] instead.
*/
index: ['gender', 'age']
}).exec();
```
## Enforce primaryKey in the index
For various performance optimizations, like the [EventReduce](https://github.com/pubkey/event-reduce) algorithm, RxDB needs a **deterministic sort order** for all query results.
To ensure a deterministic sorting, RxDB now automatically adds the primary key as last sort attribute to every query, if it is not there already. This ensures that all documents that have the same attributes on all query relevant fields, still can be sorted in a deterministic way, not depending on which was written first to the database.
In the past, this often lead to slow queries, because indexes where not constructed with that in mind.
Now RxDB will add the `primaryKey` to all indexes that do not contain it already.
If you have any collection with a custom index set, you need to run a [migration](https://rxdb.info/migration-schema.html) when updating to RxDB version `12.0.0` so that RxDB can rebuild the indexes.
## Fields that are used in indexes need some meta attributes
When using a schema with indexes, depending on the field type, you must have set some meta attributes like `maxLength` or `minimum`. This is required so that RxDB
is able to know the maximum string representation length of a field, which is needed to craft custom indexes on several `RxStorage` implementations.
```javascript
const schemaWithIndexes = {
version: 0,
primaryKey: 'id',
type: 'object',
properties: {
id: {
type: 'string',
maxLength: 100 // <- the primary key must set `maxLength`
},
firstName: {
type: 'string',
maxLength: 100 // <- string-fields that are used as an index, must set `maxLength`.
},
active: {
type: 'boolean'
},
balance: {
type: 'number',
// number fields that are used in an index, must set `minimum`, `maximum` and `multipleOf`
minimum: 0,
maximum: 100000,
multipleOf: '0.01'
}
},
required: [
'active' // <- boolean fields that are used in an index, must be required.
],
indexes: [
'firstName',
['active', 'firstName']
]
};
```
## Introduce `_meta` field
In the past, RxDB used a hacky way to mark documents as being from the remote instance during replication.
This is needed to ensure that pulled documents are not sent to the backend again.
RxDB crafted a specific revision string and stored the data with that string.
This meant that it was not possible to replicate with multiple endpoints at the same time.
From now on, all document data is stored with an `_meta` field that can contain various flags and other values.
This makes it easier for plugins to remember stuff that belongs to the document.
**In the future**, the other meta fields like `_rev`, `_deleted` and `_attachments` will be moved from the root level
to the `_meta` field. This is **not** done in release `12.0.0` to ensure that there is a migration path.
## Removed RxStorage RxKeyObjectInstance
In the past, we stored local documents and internal data in a `RxStorageKeyObjectInstance` of the `RxStorage` interface.
In PouchDB, this has a [slight performance](https://pouchdb.com/guides/local-documents.html#advantages-of-local%E2%80%93docs) improvement compared to storing that data in 'normal' documents because it does not have to handle the revision tree.
But this improved performance is only possible because normal document handling on PouchDB is so slow.
For every other RxStorage implementation, it does not really matter if documents are stored in a query-able way or not. Therefore, the whole `RxStorageKeyObjectInstance` is removed. Instead, RxDB now stores local documents and internal data in normal storage instances. This removes complexity and makes things easier in the future. For example, we could now migrate local documents or query them in plugins.
## Refactor plugin hooks
In the past, an `RxPlugin` could add plugins hooks which where always added as last.
This meant that some plugins depended on having the correct order when calling `addRxPlugin()`.
Now each plugin hook can be either defined as `before` or `after` to specify at which position of the current hooks
the new hook must be added.
## Local documents must be activated per RxDatabase/RxCollection
For better performance, the local document plugin does not create a storage for every database or collection that is created.
Instead, you have to set `localDocuments: true` when you want to store local documents in the instance.
```js
// activate local documents on a RxDatabase
const myDatabase = await createRxDatabase({
name: 'mydatabase',
storage: getRxStoragePouch('memory'),
localDocuments: true // <- activate this to store local documents in the database
});
myDatabase.addCollections({
messages: {
schema: messageSchema,
localDocuments: true // <- activate this to store local documents in the collection
}
});
```
## Added Memory RxStorage
The [Memory RxStorage](https://rxdb.info/rx-storage-memory.html) is based on plain in-memory arrays and objects. It can be used in all environments and is made for performance.
## RxDB Premium π
You can now purchase access to additional RxDB plugins that are part of the [RxDB Premium π](/premium/) package.
**If you have [sponsored](https://github.com/sponsors/pubkey) RxDB in the past (before the April 2022), you can get free lifetime access to RxDB Premium π by writing me via [Twitter](https://twitter.com/rxdbjs)**
- [RxStorage IndexedDB](https://rxdb.info/rx-storage-indexeddb.html) a really fast [RxStorage](https://rxdb.info/rx-storage.html) implementation based on **IndexedDB**. Made to be used in browsers.
- [RxStorage SQLite](https://rxdb.info/rx-storage-sqlite.html) a really fast [RxStorage](https://rxdb.info/rx-storage.html) implementation based on **SQLite**. Made to be used on **Node.js**, **Electron**, **React Native**, **Cordova** or **Capacitor**.
- [RxStorage Sharding](https://rxdb.info/rx-storage-sharding.html) a wrapper around any other [RxStorage](https://rxdb.info/rx-storage.html) that improves performance by applying the sharding technique.
- **migrateRxDBV11ToV12** A plugin that migrates data from any RxDB v11 storage to a new RxDB v12 database. Use this when you upgrade from RxDB 11->12 and you have to keep your database state.
## Other changes
- The Dexie.js RxStorage is no longer in beta mode.
- Added `RxDocument().toMutableJSON()`
- Added `RxCollection().bulkUpsert()`
- Added optional `init()` function to `RxPlugin`.
- dev-mode: Add check to ensure all top-level fields in a query are defined in the schema.
- Support for array field based indexes like `data.[].subfield` was removed, as it anyway never really worked.
- Refactored the usage of RxCollection.storageInstance to ensure all hooks run properly.
- Refactored the encryption plugin so no more plugin specific code is in the RxDB core.
- Removed the encrypted export from the json-import-export plugin. This was barely used and made everything more complex. All exports are now non-encrypted. If you need them encrypted, you can still run by encryption after the export is done.
- RxPlugin hooks now can be defined as running `before` or `after` other plugin hooks.
- Attachments are now internally handled as string instead of `Blob` or `Buffer`
- Fix (replication primitives) only drop pulled documents when a relevant document was changed locally.
- Fix dexie.js was not able to query over an index when `keyCompression: true`
Changes to `RxStorageInterface`:
- `RxStorageInstance` must have the `RxStorage` in the `storage` property.
- The `_deleted` field is now required for each data interaction with `RxStorage`.
- Removed `RxStorageInstance.getChangedDocuments()` and added `RxStorageInstance.getChangedDocumentsSince()` for better performance.
- Added `doesBroadcastChangestream()` to `RxStorageStatics`
- Added `withDeleted` parameter to `RxStorageKeyObjectInstance.findLocalDocumentsById()`
- Added internal `_meta` property to stored document data that contains internal document related data like last-write-time and replication checkpoints.
## You can help!
There are many things that can be done by **you** to improve RxDB:
- Check the [BACKLOG](https://github.com/pubkey/rxdb/blob/master/orga/BACKLOG.md) for features that would be great to have.
- Check the [breaking backlog](https://github.com/pubkey/rxdb/blob/master/orga/before-next-major.md) for breaking changes that must be implemented in the future but where I did not have the time yet.
- Check the [todos](https://github.com/pubkey/rxdb/search?q=todo) in the code. There are many small improvements that can be done for performance and build size.
- Review the code and add tests. I am only a single human with a laptop. My code is not perfect and much small improvements can be done when people review the code and help me to clarify undefined behaviors.
- Improve the documentation. In the last user survey, many users told me that the documentation is not good enough. But I reviewed the docs and could not find clear flaws. The problem is that I am way too deep into RxDB so that I am not able to understand which documentation a newcomer to the project needs. Likely I assume too much knowledge or focus writing about the wrong parts.
- Update the [example projects](https://github.com/pubkey/rxdb/tree/master/examples) many of them are outdated and need updates.
- Help the next [PouchDB release](https://github.com/pouchdb/pouchdb/issues/8408) to improve RxDBs performance.
---
## RxDB 13.0.0 - A New Era of Replication
# 13.0.0
So in the last major RxDB versions, the focus was set to **improvements of the storage engine**. This is done. RxDB has now [multiple RxStorage implementations](../rx-storage.md), a better query planner and an improved test suite to ensure everything works correct.
This let to huge improvements in write and query performance and decreased the initial pageload of RxDB based applications.
In the new major version `13.0.0`, the focus was set to improvements to the **replication protocol**.
When I first implemented the GraphQL replication a few years ago, I had a specific use case in mind and designed the whole protocol and replication plugins around that use case.
But the time has shown, that the old replication protocol is a big downside of RxDB:
- The replication relied on the backend to solve all **conflicts**. This was easy to implement into RxDB because the whole responsibility was given away to the person that has to implement a compatible backend.
- In each point in time, the replication did either push or pull documents, but **never in parallel**. This slows done the whole replication process and makes RxDB not usable for the implementation of features like **multi-user-real-time-collaboration** or when many read- and write operations have to happen in a short timespan.
- After each `push`, a `pull` had to be run to check if the backend had changed the state to solve a conflict.
- The replication protocol did not support attachments and was not designed to ever support them.
So in version `13.0.0` I replaced the whole replication plugins with a new replication protocol. The main goals have been:
- Push- and Pull in parallel.
- Use the data in the changestream (optional) to decrease replication latency.
- Implement the conflict resolution into RxDB so that the **client resolves its own conflicts** and does not rely on the backend.
- Decrease the complexity for a compatible backend implementation. The new protocol relies on a *dumb* backend. This will open compatibility with many other use cases like implementing [Offline-First in Supabase](https://github.com/supabase/supabase/discussions/357) or using CouchDB but having a faster replication compared to the native CouchDB replication.
- Make it possible to use [CRDTs](https://en.wikipedia.org/wiki/Conflict-free_replicated_data_type) instead of a conflict resolution.
- Design a the protocol in a way to make it possible to add attachments replication in the future.
On the RxDocument level, the replication works like git, where the fork/client contains all new writes and must be merged with the master/server before it can push its new state to the master/server.
```
A---B1---C1---X master/server state
\ /
B1---C2 fork/client state
```
For more details, read the [documentation about the new RxDB Sync Engine](../replication.md).
Backends that have been compatible with the previous RxDB versions `12` and older, will not work with the new replication protocol. To learn how to do that, either read the [docs](../replication.md) or check out the [GraphQL example](https://github.com/pubkey/rxdb/tree/master/examples/graphql).
## Other breaking changes
- RENAMED the `ajv-validate` plugin to `validate-ajv` to be in equal with the other validation plugins.
- The `is-my-json-valid` validation is no longer supported until [this bug](https://github.com/mafintosh/is-my-json-valid/pull/192) is fixed.
- REFACTORED the [schema validation plugins](https://rxdb.info/schema-validation.html), they are no longer plugins but now they get wrapped around any other RxStorage.
- It allows us to run the validation inside of a [Worker RxStorage](../rx-storage-worker.md) instead of running it in the main JavaScript process.
- It allows us to configure which `RxDatabase` instance must use the validation and which does not. In production it often makes sense to validate user data, but you might not need the validation for data that is only replicated from the backend.
- REFACTORED the [key compression plugin](../key-compression.md), it is no longer a plugin but now a wrapper around any other RxStorage.
- It allows to run the key-compression inside of a [Worker RxStorage](../rx-storage-worker.md) instead of running it in the main JavaScript process.
- REFACTORED the encryption plugin, it is no longer a plugin but now a wrapper around any other RxStorage.
- It allows to run the encryption inside of a [Worker RxStorage](../rx-storage-worker.md) instead of running it in the main JavaScript process.
- It allows do use asynchronous crypto function like [WebCrypto](https://developer.mozilla.org/en-US/docs/Web/API/Web_Crypto_API)
- Store the password hash in the same write request as the database token to improve performance.
- REMOVED support for temporary documents [see here](https://github.com/pubkey/rxdb/pull/3777#issuecomment-1120669088)
- REMOVED RxDatabase.broadcastChannel The broadcast channel has been moved out of the RxDatabase and is part of the RxStorage. So it is no longer exposed via `RxDatabase.broadcastChannel`.
- Removed the `liveInterval` option of the replication plugins. It was an edge case feature with wrong defaults. If you want to run the pull replication on interval, you can send a `RESYNC` event manually in a loop.
- REPLACED `RxReplicationPullError` and `RxReplicationPushError` with normal `RxError` like in the rest of the RxDB code.
- REMOVED the option to filter out replication documents with the push/pull modifiers [#2552](https://github.com/pubkey/rxdb/issues/2552) because this does not work with the new replication protocol.
- CHANGE default of replication `live` to be set to `true`. Because most people want to do a live replication, not a one time replication.
- RENAMED the `server` plugin is now called `server-couchdb` and `RxDatabase.server()` is now `RxDatabase.serverCouchDB()`
- CHANGED Attachment data is now always handled as `Blob` because Node.js does support `Blob` since version 18.0.0 so we no longer have to use a `Buffer` but instead can use Blob for browsers and Node.js
- REFACTORED the layout of `RxChangeEvent` to better match the RxDB requirements and to fix the 'deleted-document-is-modified-but-still-deleted' bug.
When used with Node.js, RxDB now requires Node.js version `18.0.0` or higher.
## Other non breaking or internal changes
- REMOVED many unused plugin hooks because they decreased the performance.
- REMOVE RxStorageStatics `.hash` and `.hashKey`
- CHANGE removed default usage of `md5` as default hashing. Use a faster non-cryptographic hash instead.
- ADD option to pass a custom hash function when calling `createRxDatabase`.
- CHANGE use `Float` instead of `Int` to represent timestamps in GraphQL.
- FIXED multiple problems with encoding attachments data. We now use the `js-base64` library which properly handles utf-8/binary/ascii transformations.
- In the RxDB internal `_meta.lwt` field, we now use 2 decimals number of the unix timestamp in milliseconds.
- ADDED `checkpointSchema` to the `RxStorage.statics` interface.
## New Features
- ADDED the [websocket replication plugin](../replication-websocket.md)
- ADDED the [FoundationDB RxStorage](../rx-storage-foundationdb.md)
## Migration to the new version
Stored data of the previous RxDB versions is not compatible with RxDB `13.0.0`. So if you want to keep that data, you have to migrate it in a way. For most use cases you might want to just drop the data from the client and re-sync it again from the backend. To keep the data locally, you might want to use the [storage migration plugin](../migration-storage.md).
---
## RxDB 14.0 - Major Changes & New Features
# 14.0.0
The release [14.0.0](https://rxdb.info/releases/14.0.0.html) is used for major refactorings and API changes.
The replication or the storage layer have only been touched marginally.
Notice that only the major changes are listed here. All minor changes can be found in the [changelog](https://github.com/pubkey/rxdb/blob/master/CHANGELOG.md).
## Removing deprecated features
The PouchDB RxStorage was deprecated in 13 and has now finally been removed, see [here](../rx-storage-pouchdb.md).
Also the old `replication-couchdb` plugin was removed. Instead we had the `replication-couchdb-new` plugin which was now renamed to `replication-couchdb`.
## API changes
### RxDocument objects are now immutable
At the previous version of RxDB, RxDocuments mutate themself when they receive ChangeEvents from the database. For example when you have a document where `name = 'foo'` and some update changes the state to `name = 'bar'` in the database, then the previous JavaScript object changed its own property to the have `doc.name === 'bar'`.
This feature was great when you use a RxDocument with some change-detection like in angular or vue templates. You can use document properties directly in the template and all updates will be reflected in the view, without having to use observables or subscriptions.
However this behavior is also confusing many times. When the state in the database is changed, it is not clear at which exact point of time the objects attribute changes. Also the self mutating behavior created some problem with vue- and react-devtools because of how they clone objects.
In RxDB v14, all RxDocuments are immutable. When you subscribe to a query and the same document is returned in the results, this will always be a new JavaScript object.
Also `RxDocument.$` now emits `RxDocument` instances instead of the plain document data.
### Refactor `findByIds()`
In the past, the functions `findByIds` and `findByIds$` directly returned the result set. This was confusing, instead they now return a `RxQuery` object that works exactly like any other database query.
```ts
const results = await myRxCollection.findByIds(['foo', 'bar']).exec();
const results$ = await myRxCollection.findByIds(['foo', 'bar']).$;
```
### Rename to RxDocument update/modify functions
Related issue [#4180](https://github.com/pubkey/rxdb/issues/4180).
In the past the naming of the document mutation methods is confusing.
For example `update()` works completely different to `atomicUpdate()` and so on.
The naming of all functions was unified and all methods do now have an incremental and a non-incremental version (previously known as `atomic`):
- RENAME `atomicUpdate()` to `incrementalModify()`
- RENAME `atomicPatch()` to `incrementalPatch()`
- RENAME `atomicUpsert()` to `incrementalUpsert()`
- ADD `RxDocument().incrementalUpdate()`
- ADD `RxDocument.incrementalRemove()`
- ADD non-incremental `RxDocument` methods `patch()` and `modify()`
### Replication is started with a pure function
In the past, to start a replication, the replication plugin was added to RxDB and a method on the RxCollection was called like `myRxCollection.syncGraphQL()`.
This caused many problems with tree shaking bailouts and having the correct typings.
So instead of having class method on the RxCollection, the replications are now started like:
```ts
import { replicateCouchDB } from 'rxdb/plugins/replication-couchdb';
const replicationState = replicateCouchDB({ /* ... */ });
```
### Storage plugins are prefixed with `storage-`
For better naming, all storage plugins have been prefixed with `storage-` so the imports have been changed:
```ts
// Instead of
import { getRxStorageDexie } from 'rxdb/plugins/dexie';
// it is now
import { getRxStorageDexie } from 'rxdb/plugins/storage-dexie';
```
### Encryption plugin was renamed to `encryption-crypto-js`
To make it possible to have alternative encryption plugins, the `encryption` plugin was renamed to `encryption-crypto-js`.
```ts
// Instead of
import { wrappedKeyEncryptionStorage } from 'rxdb/plugins/encryption';
// it is now
import { wrappedKeyEncryptionCryptoJsStorage } from 'rxdb/plugins/encryption-crypto-js';
```
### Rewrite the `worker` plugin
In the past, the [worker plugin](../rx-storage-worker.html) was based on the [threads](https://www.npmjs.com/package/threads) library. It was completely rewritten and uses the plain JavaScript API together with the [remote storage plugin](../rx-storage-remote.md). **BUT** notice that the worker plugin has moved into the [RxDB Premium π](/premium/) package.
## Performance improvements
### Do not use hash for revisions
In the past, the `_rev` field of a RxDocument data was filled with a hash of the documents data. This was not the best solution because:
- Hashing in JavaScript is slow, not running hashes on insert improves performance by about 33%
- When 2 clients do the exact same write to the document, it is not clear from comparing the document states because they will have the exact same hash which makes some conflict resolution strategies impossible to implement.
Instead we now use just use the RxDatabase.token together with the revision height.
### Batch up incremental operations
When making multiple writes to different (or the same) document, in the past RxDB made one write call to the storage. When doing fast writes, like when you writhe the current mouse position to a document, the writes could have queued up to the point where the users recognizes performance lag.
In RxDB v14, pending document updates are batched up into single storage writes for better performance.
### Improved tree shaking
Many changes have been made to improve [tree shakability](https://webpack.js.org/guides/tree-shaking/) of RxDB which results in smaller bundle sizes and a faster application startup.
### Refactor the document cache
The whole RxDocument cache was refactored. It now runs based on the [WeakRef API](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/WeakRef) and automatically cleans up cached documents that are no longer referenced. This reduces the use memory and makes RxDB more suitable for being used in Node.js on the server side.
Notice that the WeakRef API is only featured in [modern browsers](https://caniuse.com/?search=weakref) so RxDB will no longer run on ie11.
### No longer transpile some modern JavaScript features
To reduce the bundle size and improve performance, the following JavaScript features will no longer be transpiled because they are natively supported in modern browsers anyway:
- [async/await](https://caniuse.com/async-functions)
- [Arrow functions](https://caniuse.com/arrow-functions)
- [for...of](https://caniuse.com/?search=for...of)
- [shorthand properties](https://caniuse.com/mdn-javascript_operators_object_initializer_shorthand_property_names)
- [Spread operator](https://caniuse.com/?search=spread%20operator)
- [destructuring](https://caniuse.com/?search=destructuring)
- [default parameters](https://caniuse.com/?search=default%20parameters)
- [object spread](https://caniuse.com/?search=Object%20spread)
All these optimizations together reduced the [test-bundle](https://github.com/pubkey/rxdb/blob/master/config/bundle-size.js) size from `74148` bytes down to `36007` bytes.
## Other changes
- ADD `push/pull.initialCheckpoint` to start a replication from a given checkpoint.
- The following plugins are out of beta mode:
- [RxStorage Sharding](../rx-storage-sharding.md)
- [RxStorage Remote](../rx-storage-remote.md)
- [RxStorage FoundationDB](../rx-storage-foundationdb.md)
- [New Replication CouchDB](../replication-couchdb.md)
- [Cleanup Plugin](../cleanup.md)
## Bugfixes
- CHANGE (memory RxStorage) do not clean up database state on closing of the storage, only on `remove()`.
- FIX CouchDB replication: Use correct default fetch method.
- FIX schema hashing should respect the sort order [#4005](https://github.com/pubkey/rxdb/pull/4005)
- FIX replication does not provide a `._rev` to the storage write when a conflict is resolved.
- FIX(remote storage) ensure caching works properly even on parallel create-calls
- FIX(replication) Composite Primary Keys broken on replicated collections [#4190](https://github.com/pubkey/rxdb/pull/4190)
- FIX(sqlite) $in Query not working SQLite [#4278](https://github.com/pubkey/rxdb/issues/4278)
- FIX CouchDB push is throwing error because of missing revision [#4299](https://github.com/pubkey/rxdb/pull/4299)
- ADD dev-mode shows a `console.warn()` to ensure people do not use it in production.
- Remove the usage of `Buffer`. We now use `Blob` everywhere.
- FIX import of socket.io [#4307](https://github.com/pubkey/rxdb/pull/4307)
- FIX Id length limit reached with composite key [#4315](https://github.com/pubkey/rxdb/issues/4315)
- FIX `$regex` query not working on remote storage.
- FIX SQLite must store attachments data as Blob instead of base64 string to reduce the database size.
- FIX CouchDB replication conflict handling
- CHANGE Encryption plugin was renamed to `encryption-crypto-js`
- FIX replication state meta data must also be encrypted.
## Migration from 13.x.x to 14.0.0
The way RxDB hashes and normalizes a schema has changed. To migrate stored data between the RxDB versions, therefore you have to increase your schema version by `1` and add a migration strategy.
For some storages, the stored data of the previous RxDB versions is not compatible with RxDB `14.0.0`. So if you want to keep that data, you have to migrate it in a way. For most use cases you might want to just drop the data from the client and re-sync it again from the backend. To keep the data locally, you might want to use the [storage migration plugin](../migration-storage.md).
## You can help!
There are many things that can be done by **you** to improve RxDB:
- Check the [BACKLOG](https://github.com/pubkey/rxdb/blob/master/orga/BACKLOG.md) for features that would be great to have.
- Check the [breaking backlog](https://github.com/pubkey/rxdb/blob/master/orga/before-next-major.md) for breaking changes that must be implemented in the future but where I did not have the time yet.
- Check the [todos](https://github.com/pubkey/rxdb/search?q=todo) in the code. There are many small improvements that can be done for performance and build size.
- Review the code and add tests. I am only a single human with a laptop. My code is not perfect and much small improvements can be done when people review the code and help me to clarify undefined behaviors.
- Update the [example projects](https://github.com/pubkey/rxdb/tree/master/examples) many of them are outdated and need updates.
---
## RxDB 15.0.0 - Major Migration Overhaul
# 15.0.0
The release [15.0.0](https://rxdb.info/releases/15.0.0.html) is used for major refactorings in the migration plugins and performance improvements of the RxStorage implementations.
## LinkedIn
Stay connected with the latest updates and network with professionals in the RxDB community by following RxDB's [official LinkedIn page](https://www.linkedin.com/company/rxdb)!
## Performance
Performance has improved a lot. Much work has been done to reduce the CPU footprint of RxDB so that when writes and reads are performed on the database, the JavaScript process can start updating the UI much faster.
Also there have been a lot of improvements to the [IndexedDB RxStorage](../rx-storage-indexeddb.md) which now runs in a **Write-Ahead Logging (WAL)** mode which makes write operations about 4x as fast.
The whole RxStorage interface has been optimized so that it has to run less operations which improves overall performance. [Read more](../rx-storage-performance.md)
## Replication
Replication options (like url), are no longer used as replication state identifier. Changing the url without having to restart the replication, is now possible. This is useful if you change your replication url (like `path/v3/`) on schema changes and you do not want to restart the replication from scratch. You can even swap the replication plugin while still keeping the replication state. The [couchdb replication](../replication-couchdb.md) now also requires an `replicationIdentifier`.
The replication meta data is now also compressed when the [KeyCompression Plugin](../key-compression.md) is used.
The replication-protocol does now support attachment replication. This clears the path to add the attachment replication to the other RxDB replication plugins.
## Rewrite schema version migration
The [schema migration plugin](../migration-schema.md) has been fully rewritten from scratch.
From now on it internally uses the replication protocol to do a one-time replication from the old collection to the new one. This makes the code more simple and ensures that canceled migrations (when the user closes the browser), can continue from the correct position.
Replication states from the [RxReplication](../replication.md) are also migrated together with the normal data.
Previously a migration dropped the replication state which required a new replication of all data from scratch, even if the
client already had the same data as the server. Now the `assumedMasterState` and `checkpoint` are also migrated so that
the replication will continue from where it was before the migration has run.
Also it now handles multi-instance runtimes correctly. If multiple browser tabs are open, only one of them (per RxCollection) will run the migration.
Migration state events are propagated across browser tabs.
Documents with `_deleted: true` will also be migrated. This ensures that non-pushed deletes are not dropped during migrations and will
still be replicated if the client goes online again.
## Set `eventReduce:true` as default
The [EventReduce algorithm](https://github.com/pubkey/event-reduce) is now enabled by default.
## Use `crypto.subtle.digest` for hashing
Using `crypto.subtle.digest` from the native WebCrypto API is much faster, so RxDB now uses that as a default. If the API is not available, like in React-Native, the [ohash](https://github.com/unjs/ohash) module is used instead. Also any custom `hashFunction` can be provided when creating the [RxDatabase](../rx-database.md). The `hashFunction` must now be async and return a Promise.
## Fix attachment hashing
Hashing of attachment data to calculate the `digest` is now done from the RxDB side, not the RxStorage. If you set a custom `hashFunction` for the database, it will also be used for attachments `digest` meta data.
## Requires at least typescript version 5.0.0
We now use `export type * from './types';` so RxDB will not work on typescript versions older than 5.0.0.
## Require string based `$regex`
Queries with a `$regex` operator must now be defined as strings, not with `RegExp` objects. You can still pass RegExp's flags in `$options` parameter. `RegExp` are mutable objects, which was dangerous and caused hard-to-debug problems.
Also stringification of the $regex had bad performance but is required to send queries from RxDB to the RxStorage.
## Refactor dexie.js RxStorage
The [dexie.js storage](../rx-storage-dexie.md) was refactored to add some missing features:
- [Attachment](../rx-attachment.md) support
- Support for boolean indexes
## RxLocalDocument.$ emits a document instance, not the plain data
This was changed in [v14](./14.0.0.md) for a normal RxDocument.$ which emits RxDocument instances. Same is now also done for [local documents](../rx-local-document.md).
## Fix return type of .bulkUpsert
Equal to other bulk operations, `bulkUpsert` will now return an `error` and a `success` array. This allows to filter for validation errors and handle them properly.
## Add dev-mode check for disallowed $ref fields
RxDB cannot resolve `$ref` fields in the schema because it would have a negative performance impact.
We now have a dev-mode check to throw a helpful error message if $refs are used in the schema.
## Improve RxDocument property access performance
We now use the Proxy API instead of defining getters on each nested property. Also fixed [#4949](https://github.com/pubkey/rxdb/pull/4949)
`patternProperties` is now allowed on the non-top-level of a schema [#4951](https://github.com/pubkey/rxdb/pull/4951)
## Add deno support
The RxDB test suite now also runs in the [deno](https://deno.com/) runtime. Also there is a [DenoKV](https://rxdb.info/rx-storage-denokv.html) based RxStorage to use with Deno Deploy.
## Memory RxStorage
Rewrites of the [Memory RxStorage](../rx-storage-memory.md) for better performance.
- Writes are 3x faster
- Find-by id is 2x faster
## Memory-Synced storage no longer supports replication+migration
The [memory-synced storage](../rx-storage-memory-synced.md) itself does not support replication and migration. This was allowed in the past, but dangerous so now there is an error that throws.
Instead you should replicate the underlying parent storage. Notice that this is only for the [memory-synced storage](../rx-storage-memory-synced.md), NOT for the normal [memory storage](../rx-storage-memory.md). There the replication works like before.
## Added Logger Plugin
I added a [logger plugin](../logger.md) to detect performance problems and errors.
## Documentation is now served by docusaurus
In the past we used gitbook which is no longer maintained and had some major issues. Now the documentation of RxDB is rendered and served with [docusaurus](https://docusaurus.io/) which has a better design and maintenance.
## Replaced `modfijs` with `mingo` package
In the past, the [modifyjs](https://github.com/lgandecki/modifyjs) was used for the `update` plugin. This was replaced with the [mingo library](https://github.com/kofrasa/mingo) which is more up to date and already used in RxDB for the query engine.
## Changes to the RxStorage interface
We no longer have `RxStorage.statics.prepareQuery()`. Instead all storages get the same prepared query as input for the `.query()` method. If a storage requires some transformations, it has to do them by itself before running the query. This change simplifies the whole RxDB code base a lot and the previous assumption of having a better performance by pre-running the query preparation, turned out to be not true because the query planning is quite fast.
Removed the `RxStorage.statics` property. This makes configuration easier especially for the remote storage plugins.
The RxStorage itself will now return `_deleted=true` documents on the `.query()` method. This is required for upcoming plugins like the server plugin where it must be able to run queries on deleted documents.
Notice that this is only for the RxStorage itself, RxDB queries will run like normal and NOT contain deleted documents in their results.
Changed the response type of RxStorageInstance.bulkWrite() from indexed (by id) objects to arrays for better performance.
## Other changes
- Added `RxCollection.cleanup()` to manually call the [cleanup functions](../cleanup.md).
- Rename send$ to sent$: `myRxReplicationState.send$.subscribe` works only if the sending is successful. Therefore, it is renamed to `sent$`, not `send$`.
- We no longer ship `dist/rxdb.browserify.js` and `dist/rxdb.browserify.min.js`. If you need these, build them by yourself.
- The example project for vanilla javascript was outdated. I removed it to no longer confuse new users.
- REPLACE `new Date().getTime()` with `Date.now()` which is [2x faster](https://stackoverflow.com/questions/12517359/performance-date-now-vs-date-gettime).
- Renamed replication-p2p to replication-webrtc. I will add more p2p replication plugins in the future, which are not based on [WebRTC](../replication-webrtc.md).
- REMOVED `RxChangeEvent.eventId`. If you really need a unique ID, you can craft your own one based on the document `_rev` and `primary`.
- REMOVED `RxChangeEvent.startTime` and `RxChangeEvent.endTime` so we do not have to call `Date.now()` once per write row.
- ADDED `EventBulk.startTime` and `EventBulk.endTime`.
- FIX `database.remove()` does not work on databases with encrypted fields.
- FIX [react-native: replaceAll is not a function](https://github.com/pubkey/rxdb/pull/5187)
- FIX Throttle calls to forkInstance on push-replication to not cause memory spikes and lagging UI
- FIX PushModifier applied to pre-change legacy document, resulting in old document sent to endpoint [#5256](https://github.com/pubkey/rxdb/issues/5256)
- [Attachment compression](../rx-attachment.md#attachment-compression) is now using the native `Compression Streams API`.
- FIX [#5311](https://github.com/pubkey/rxdb/issues/5311) URL.createObjectURL is not a function in a browser plugin environment(background.js)
- FIX `structuredClone` not available in ReactNative [#5046](https://github.com/pubkey/rxdb/issues/5046#issuecomment-1827374498)
- The following things moved out of beta:
- [Firestore replication](../replication-firestore.md)
- [WebRTC replication](../replication-webrtc.md)
- [NATS replication](../replication-nats.md)
- [OPFS RxStorage](../rx-storage-opfs.md)
## Changes to the π Premium Plugins
### storage-migration plugin moved from premium to open-core
The storage migration plugin can be used to migrate data between different RxStorage implementation or to migrate data between major RxDB versions. This previously was a π premium plugin, but now it is part of the open-core. Also the params syntax changed a bit [read more](../migration-storage.md).
### Changes in pricing
The pricing of the premium plugins was changed. This makes it cheaper for smaller companies and single individuals.
### Added perpetual license option
By default you are not allowed to use the premium plugins after the
license has expired and you will no longer be able to install them. But
you can choose the **Perpetual license** option. With the perpetual
license option, you can still use the plugins even after the license is
expired. But you will no longer get any updates from newer RxDB
versions.
## You can help!
There are many things that can be done by **you** to improve RxDB:
- Check the [BACKLOG](https://github.com/pubkey/rxdb/blob/master/orga/BACKLOG.md) for features that would be great to have.
- Check the [breaking backlog](https://github.com/pubkey/rxdb/blob/master/orga/before-next-major.md) for breaking changes that must be implemented in the future but where I did not have the time yet.
- Check the [todos](https://github.com/pubkey/rxdb/search?q=todo) in the code. There are many small improvements that can be done for performance and build size.
- Review the code and add tests. I am only a single human with a laptop. My code is not perfect and much small improvements can be done when people review the code and help me to clarify undefined behaviors.
- Update the [example projects](https://github.com/pubkey/rxdb/tree/master/examples) some of them are outdated and need updates.
---
## RxDB 16.0.0 - Efficiency Redefined
# 16.0.0
The release [16.0.0](https://rxdb.info/releases/16.0.0.html) is used for major refactorings. We did not change that much, mostly renaming things and fixing confusing implementation details.
Data stored in the previous version `15` is compatible with the code of the new version `16` for most RxStorage implementation. So migration will be easy.
Only the following RxStorage implementations are required to migrate the data itself with the [storage migration plugin](../migration-storage.md):
- SQLite RxStorage
- NodeFilesystem RxStorage
- OPFS RxStorage
## Breaking Changes
- CHANGE [RxServer](https://rxdb.info/rx-server.html) is no longer in beta mode.
- CHANGE [Fulltext Search](https://rxdb.info/fulltext-search.html) is no longer in beta mode.
- CHANGE [Custom Reactivity](https://rxdb.info/reactivity.html) is no longer in beta mode.
- CHANGE [initialCheckpoint in replications](https://rxdb.info/replication.html) is no longer in beta mode.
- CHANGE [RxState](https://rxdb.info/rx-state.html) is no longer in beta mode.
- CHANGE [MemoryMapped RxStorage](https://rxdb.info/rx-storage-memory-mapped.html) is no longer in beta mode.
- CHANGE rename `randomCouchString()` to `randomToken()`
- FIX (GraphQL replication) datapath must be equivalent for pull and push [#6019](https://github.com/pubkey/rxdb/pull/6019)
- REMOVE fallback to the `ohash` package when `crypto.subtle` does not exist. All modern runtimes (also react-native) now support `crypto.subtle`, so we do not need that fallback anymore.
### Removed deprecated LokiJS RxStorage
The [LokiJS RxStorage](https://rxdb.info/rx-storage-lokijs.html) was deprecated because LokiJS itself is no longer maintained. Therefore it will be removed completely. If you still need that, you can fork the code of it and publish it in an own package and link to it from the [third party plugins page](https://rxdb.info/third-party-plugins.html).
### Renamed `.destroy()` to `.close()`
Destroy was adapted from PouchDB, but people often think this deletes the written data. `close` is a better name for that functionality.
- Also renamed similar functions/attributes:
- `.destroy()` to `.close()`
- `.onDestroy()` to `.onClose()`
- `postDestroyRxCollection` to `postCloseRxCollection`
- `preDestroyRxDatabase` to `preCloseRxDatabase`
### `ignoreDuplicate: true` on `createRxDatabase()` must only be allowed in dev-mode.
The `ignoreDuplicate` flag is only useful for tests and should never be used in production. We now throw an error if it is set to `true` in non-dev-mode.
### When dev-mode is enabled, a schema validator must be used.
Many reported issues come from people storing data that is not valid to their schema.
To fix this, in dev-mode it is now required that at least one schema validator is used.
### Removed the memory-synced storage
The memory-synced RxStorage was removed in RxDB version 16. Please use the `memory-mapped` storage instead which has better trade-offs and is easier to configure.
### Split conflict handler functionality into `isEqual()` and `resolve()`.
Because the handler is used in so many places it becomes confusing to write a proper conflict handler.
Also having a handler that requires user interaction is only possible by hackingly using the context param.
By splitting the functionalities it is easier to learn where the handlers are used and how to define them properly.
## Full rewrite of the OPFS and Filesystem-Node RxStorages
The [OPFS](../rx-storage-opfs.md) and [Filesystem-Node](../rx-storage-filesystem-node.md) RxStorage had problems with storing emojis and other special characters inside of indexed fields.
I completely rewrote them and improved performance especially on initial load when a lot of data is stored already and when doing many small writes/reads at the same time on in series.
## Internal Changes
- CHANGE (internal) migration-storage plugin: Remove catch from cleanup
- CHANGE (internal) rename RX_PIPELINE_CHECKPOINT_CONTEXT to `rx-pipeline-checkpoint`
- CHANGE (internal) remove `conflictResultionTasks()` and `resolveConflictResultionTask()` from the RxStorage interface.
- REMOVED (internal) do not check for duplicate event bulks, all RxStorage implementations must guarantee to not emit the same events twice.
- REFACTOR (internal) Only use event-bulks internally and only transform to single emitted events if actually someone has subscribed to the eventstream.
## Bugfixes
- Having a lot of documents pulled in the replication could in some cases slow down the database initialization because `upstreamInitialSync()` did not set a checkpoint and each time checked all documents if they are equal to the master.
- If the handler of a [RxPipeline](../rx-pipeline.md) throws an error, block the whole pipeline and emit the error to the outside.
- Throw error when dexie.js RxStorage is used with optional index fields [#6643](https://github.com/pubkey/rxdb/pull/6643#issuecomment-2505310082).
- Fix IndexedDB bug: Some people had problems with the IndexedDB RxStorage that opened up collections very slowly. If you had this problem, please try out this new version.
- Add check to ensure remote instances are build with the same RxDB version. This is to ensure if you update RxDB and forget to rebuild your workers, it will throw instead of causing strange problems.
- When the pulled documents in the replication do not match the schema, do not update the checkpoint.
- When the pushed document conflict results in the replication do not match the schema, do not update the checkpoint.
## Other
- Added more [performance tests](https://rxdb.info/rx-storage-performance.html)
- The amount of collections in the open source version has been limited to `16`.
- Moved RxQuery checks into dev-mode.
- RxQuery.remove() now internally does a bulk operation for better performance.
- Lazily process bulkWrite() results for less CPU usage.
- Only run interval cleanup on the storage of a collection if there actually have been writes to it.
- Schema validation errors (code: 422) now include the `RxJsonSchema` for easier debugging.
- Added an interval to prevent browser tab hibernation while a replication is running.
## You can help!
There are many things that can be done by **you** to improve RxDB:
- Check the [BACKLOG](https://github.com/pubkey/rxdb/blob/master/orga/BACKLOG.md) for features that would be great to have.
- Check the [breaking backlog](https://github.com/pubkey/rxdb/blob/master/orga/before-next-major.md) for breaking changes that must be implemented in the future but where I did not have the time yet.
- Check the [todos](https://github.com/pubkey/rxdb/search?q=todo) in the code. There are many small improvements that can be done for performance and build size.
- Review the code and add tests. I am only a single human with a laptop. My code is not perfect and much small improvements can be done when people review the code and help me to clarify undefined behaviors.
- Update the [example projects](https://github.com/pubkey/rxdb/tree/master/examples) some of them are outdated and need updates.
## LinkedIn
Stay connected with the latest updates and network with professionals in the RxDB community by following RxDB's [official LinkedIn page](https://www.linkedin.com/company/rxdb)!
---
## RxDB 17.0.0
# RxDB 17.0.0 - Local-First to the Moon (beta)
RxDB 17 focuses on **better reactivity**, **improved debugging**, and **important storage fixes**, while also graduating several long-standing plugins out of beta.
Most applications can upgrade easily, but **users of OPFS and filesystem-based storages must review the migration notes carefully**.
:::note
RxDB version 17 is currently in **beta**. For testing, please install the latest beta version [from npm](https://www.npmjs.com/package/rxdb?activeTab=versions) and provide feedback or report issues [on GitHub](https://github.com/pubkey/rxdb/issues/7574).
:::
## Migration
RxDB 17 is mostly backward-compatible, but please review the following points before upgrading:
### Storage migration required
If you use any of the following storages, **you must migrate your data** using the [storage migrator](../migration-storage.md):
- OPFS RxStorage
- Filesystem RxStorage (Node)
- IndexedDB RxStorage (if you use attachments)
For all other RxStorages, data created with RxDB v16 can be reused directly in RxDB v17.
### Other migration notes
- The GitHub repository **no longer contains prebuilt `dist` files**.
Install RxDB from npm or run the build scripts locally.
- `toggleOnDocumentVisible` now defaults to **`true`**. [#6810](https://github.com/pubkey/rxdb/issues/6810)
- Schema fields marked as `final` **no longer need to be explicitly listed as `required`**.
- Some integrations now use **optional peer dependencies**. You may need to install them manually:
- `firebase`
- `mongodb`
- `nats`
## Features
### Improvements for our Vibe-Coders π»β¨
To improve vide-coding with RxDB, we made the following DX-for-LLM improvements:
- **ADD** [https://rxdb.info/llms.txt](https://rxdb.info/llms.txt) for LLM-friendly documentation access.
- **ADD** `ERROR-MESSAGES.md` to the root of the RxDB package so your LLM knows about all possible errors.
- **ADD** Errors now contain a `cause`, `fix` and `docs` property which tells your LLM what went wrong and where to find information about possible fixes.
- **ADD** `@example` tags to TypeScript comments in many parts of the typings.
To improve vibe-coding when working with RxDB directly we:
- ADD [.aiexclude](https://developer.android.com/studio/gemini/aiexclude?hl=de) file to reduce context.
- ADD [.agent Skill](https://antigravity.google/docs/skills) to verify code changes.
- ADD `.claudeignore` file to reduce context.
- ADD `.claude/settings.json` to improve post-generation testing.
- ADD `CLAUDE.md`
### Better/More Integrations
- **ADD** [WebMCP Plugin](../webmcp.md) to expose RxDB collections to AI Agents via the Model Context Protocol.
- **ADD** [Google Drive Replication](../replication-google-drive.md) plugin to replicate client data to a clients Google Drive folder without any server.
- **ADD** [Microsoft OneDrive Replication](../replication-microsoft-onedrive.md) plugin to replicate client data to a clients OneDrive folder without any server.
- **ADD** [react-hooks plugin](../react.md).
### π Reactivity & APIs
- **ADD** `RxDatabase.collections$` observable for reactive access to collections
- **ADD** support for the JavaScript `using` keyword to automatically remove databases in tests
- **ADD** new `reactivity-angular` package
- **ADD** correctly typed `$$` properties and `get$$()` via HKT-based `ReactivityLambda` pattern. Signals now carry the document data type (e.g. `Signal` instead of `Signal`).
- **CHANGE** moved `reactivity-vue` and `reactivity-preact-signals` from premium to core
- **FIX** query results becoming incorrect when changes occur faster than query update [#7067](https://github.com/pubkey/rxdb/issues/7067)
### π§ Debugging & Developer Experience
- **ADD** `context` field to all RxDB write errors for easier debugging
- **ADD** improved OPFS RxStorage error logging in `devMode`, including:
- strict `TextDecoder` mode
- detailed decoding error output
- **ADD** internal `WeakRef` TypeScript types so users no longer need to enable `ES2021.WeakRef`
- **CHANGE** `toggleOnDocumentVisible` now defaults to `true` [#6810](https://github.com/pubkey/rxdb/issues/6810)
- **ADD** docs page about [testing with RxDB](../testing.md).
### ποΈ Storage, Queries & Replication Fixes
- **FIX** OPFS RxStorage memory and cleanup leaks
- **FIX** memory-mapped storage not purging deleted documents
- **FIX** `RxCollection.cleanup()` ignoring `minimumDeletedTime`
- **FIX** short primary key lengths not matching replication schema [#7587](https://github.com/pubkey/rxdb/issues/7587)
- **FIX** retry DenoKV commits when encountering `"database is locked"` errors
- **FIX** close broadcast channels and leader election **after** database shutdown, not during
- **FIX**: migration memory leak: old collection meta doc not deleted [#7792](https://github.com/pubkey/rxdb/pull/7792)
- **CHANGE**: Store data as binary in IndexedDB to use less disc space.
- **CHANGE**: Pull-Only replications no longer store the server metadata on the client.
- **FIX**: downstream replication issues [#7804](https://github.com/pubkey/rxdb/pull/7804)
- **FIX** add event guard to count and findByIds in _execOverDatabase [#7864](https://github.com/pubkey/rxdb/pull/7864)
- **FIX**: memory leak in migratePromise() [#7787](https://github.com/pubkey/rxdb/pull/7787)
- **FIX** `exclusiveMinimum` and `exclusiveMaximum` TypeScript types corrected from `boolean` to `number` to match JSON Schema Draft 6+ [#7962](https://github.com/pubkey/rxdb/pull/7962)
- **FIX** `find()` with `$in` on the primary key now correctly uses the fast `findDocumentsById` path instead of falling back to a full storage query [#7993](https://github.com/pubkey/rxdb/pull/7993)
- **FIX** Dexie and DenoKV RxStorages throwing errors (`DataError` / "Start key is greater than end key") on query ranges that yield no results (e.g., `$gt` and `$lt` at the same value).
- **ADD** `waitBeforePersist` option to `ReplicationPushOptions` to delay upstream persistence cycles, enabling write batching across collections and CPU-idle deferral [#7872](https://github.com/pubkey/rxdb/issues/7872)
- **ADD** enforce maximum length for indexes and primary keys (`maxLength: 2048`)
- **CHANGE** `final` schema fields no longer need to be marked as `required`
- **UPDATE** [z-schema](https://github.com/zaggino/z-schema/blob/main/CHANGELOG.md) to version 9.
- **CHANGE** replace `appendToArray()` with `Array.concat()` for better browser-level optimizations
- **ADD** ensure indexes and primary keys are validated consistently across replication schemas
- **CHANGE**: Reduce `NON_PREMIUM_COLLECTION_LIMIT` from `16` to `13`.
- **ADD** Support `extendedLifetime` when using the [SharedWorker](../rx-storage-shared-worker.md).
### π Plugins Graduating from Beta
The following plugins are **no longer in beta** and are now considered production-ready:
- Replication Appwrite
- Replication Supabase
- Replication MongoDB
- RxStorage MongoDB
- RxStorage Filesystem (Node)
- RxStorage DenoKV
- Attachment replication
- CRDT Plugin
- RxPipeline
## You can help!
There are many things that can be done by **you** to improve RxDB:
- Check the [BACKLOG](https://github.com/pubkey/rxdb/blob/master/orga/BACKLOG.md) for features that would be great to have.
- Check the [breaking backlog](https://github.com/pubkey/rxdb/blob/master/orga/before-next-major.md) for breaking changes that must be implemented in the future but where I did not have the time yet.
- Check the [todos](https://github.com/pubkey/rxdb/search?q=todo) in the code. There are many small improvements that can be done for performance and build size.
- Review the code and add tests. I am only a single human with a laptop. My code is not perfect and much small improvements can be done when people review the code and help me to clarify undefined behaviors.
- Update the [example projects](https://github.com/pubkey/rxdb/tree/master/examples) some of them are outdated and need updates.
## LinkedIn
Stay connected with the latest updates and network with professionals in the RxDB community by following RxDB's [official LinkedIn page](https://www.linkedin.com/company/rxdb)!
---
## Meet RxDB 8.0.0 - New Defaults & Performance
# 8.0.0
When I created RxDB, two years ago, there where some things that I did not consider and other things that I just decided wrong.
With the breaking version `8.0.0` I rewrote some parts of RxDB and changed the API a bit. The focus laid on **better defaults** and **better performance**.
## disableKeyCompression by default
Because the keyCompression was confusing and most users do not use it, it is now disabled by default. Also the naming was bad, so `disableKeyCompression` is now renamed to `keyCompression` which defaults to `false`.
## collection() now only accepts a RxJsonSchema as schema
In the past, it was allowed to set an `RxSchema` or an `RxJsonSchema` as `schema`-field when creating a collection. This was confusing and so it is now only allowed to use `RxJsonSchema`.
## required fields have to be set via array
In the past it was allowed to set a field as required by setting the boolean value `required: true`.
This is against the [json-schema-standard](https://json-schema.org/understanding-json-schema/reference/object.html#required-properties) and will make problems when switching between different schema-validation-plugins. Therefore required fields must now be set via `required: ['fieldOne', 'fieldTwo']`.
## Setters are only callable on temporary documents
To be similar to mongoosejs, there was the possibility to set a documents value via `myDoc.foo = 'bar'` and later call `myDoc.save()` to persist these changes.
But there was the problem that we have no weak pointers in javascript and therefore we have to store all document-instances in a cache to run change-events on them and make them 'reactive'. To save memory-space, we reuse the same documents when they are used multiple times. This made it hard to determine what happens when multiple parts of an application used the setters at the same time. Also there was an undefined behavior what should happen when a field is changed via setter and also via replication before `save()` was called.
- `myDoc.age = 50;`, `myDoc.set('age', 50);` and `myDoc.save()` is no more allowed on non-temporary-documents
- Instead, to change document-data, use `RxDocument.atomicUpdate()` or `RxDocument.atomicSet()` or `RxDocument.update()`.
- The following document-methods no longer exist: `synced$`, `resync()`
## middleware-hooks contain plain json as first parameter and RxDocument as second
When the middleware-hooks where created, the goal was to work equal then [mongoose](http://mongoosejs.com/docs/middleware.html). But this is not possible because mongoose is more 'static' and its documents never change their attributes. RxDB is a reactive database and when the state on disc changes, the attributes of the document also change. This caused some undefined behavior especially with async middleware-hooks. To solve this, hooks now can only modify the plain data, not the `RxDocument` itself.
```javascript
myCollection.preSave(function(data, rxDocument) {
// to set age to 50 before saving, change the first parameter
data.age = 50;
}, false);
```
## multiInstance is now done via broadcast-channel
Because the [BroadcastChannel-API](https://developer.mozilla.org/en-US/docs/Web/API/Broadcast_Channel_API) is not usable in all browsers and also not in NodeJs, multiInstance-communication was hard. The hacky workaround was to use a pseudo-socket and regularly check if an other instance has emitted a `RxChangeEvent`.
This was expensive because even when the database did nothing, we wasted disk-IO and CPU by handling this.
To solve this waste, I spend one month creating [a module that polyfills the broadcast-channel-api](https://github.com/pubkey/broadcast-channel) so it works on old browsers, new browsers and even NodeJs. This does not only waste less resources but also has a lower latency. Also the module is used by more projects than RxDB which allows use the fix bugs and improve performance together.
## Set QueryChangeDetection via RxDatabase-option
In the past, the QueryChangeDetection had to be enabled by importing the QueryChangeDetection and calling a function on it. This was strange and also did not allow to toggle the QueryChangeDetection on specific databases.
Now we set the QueryChangeDetection by adding the boolean field `queryChangeDetection: true` when creating the database.
```javascript
const db = await RxDB.create({
name: 'heroesdb',
adapter: 'idb',
queryChangeDetection: false // <- queryChangeDetection (optional, default: false)
});
console.dir(db);
```
## Reuse an RxDocument-prototype per collection instead of adding getters/setters to each document
Because the fields of an `RxDocument` are defined dynamically by the schema and we could not use the `Proxy`-Object because it is not supported in IE11, there was one workaround used: Each time a document is created, all getters and setters where applied on it. This was expensive and now we use a different approach.
Once per collection, a custom RxDocument-prototype and constructor is created and each RxDocument of this collection is created with the constructor. This change is a rewrite internally and should not change anything for RxDB-users. If you use RxDB with vuejs, you might get some problems that can be fixed by upgrading vuejs to the latest version.
## Rewritten the inMemory-plugin
The inMemory-plugin was written with some wrong estimations. I rewrote it and added much more tests. Also `awaitPersistence()` can now be used to check if all writes have already been replicated into the parent collection. Also `RxCollection.watchForChanges()` got split out from the `replication`-plugin into its own `watch-for-changes`-plugin because it is used in the inMemory and the replication functionality.
## Some comparisons
(Tested with node 10.6.0 and the memory-adapter, lower is better)
Reproduce with `npm run build:size` and `npm run test:performance`
| | 7.7.1 | 8.0.0 |
| :-------------------------------------------: | :-----: | ---------- |
| Bundle-Size (Webpack+minify+gzip) | 110 kb | 101.962 kb |
| Spawn 1000 databases with 5 collections each | 8744 ms | 9906 ms |
| insert 2000 documents | 8006 ms | 5781 ms |
| find 10.000 documents | 3202 ms | 1312 ms |
---
## RxDB 9.0.0 - Faster & Simpler
# 9.0.0
So I was working hard the past month to prepare everything for the next major release of RxDB. The last major release was 1,5 years ago by the way.
When I started [listing up the planned changes](https://github.com/pubkey/rxdb/issues/1636) I had big ambitions about basically rewriting everything. But I found out this time has not come yet. There is some work to be done first. So version `9.0.0` was more about fixing all these small things that made improving the codebase difficult. Much has been refactored and moved. Some API-parts have been changed to have a more simple project with a cleaner codebase.
Notice that I use major releases to bundle stuff that breaks the RxDB usage in your project. By having only few major releases you can be sure that you can upgrade in one big block instead of changing stuff each few months. Big features are released in non-major releases because they mostly can be implemented without side effects.
## Breaking changes
You have to apply these changes to your codebase when upgrading RxDB.
### All default exports have been removed
Using default exports and imports can be helpful when you want to write code fast.
But using them also disabled the tree-shaking of your bundler which means you added much code to your bundle that was not even used. To prevent this common behavior, I removed all default exports and renamed functions so that they are more unlikely to clash with other non-RxDB function names.
Instead of doing
```typescript
import RxDB from 'rxdb';
RxDB.plugin(/* ... */);
await RxDB.create({/* ... */});
```
You now do
```typescript
import { createRxDatabase, addRxPlugin } from 'rxdb';
addRxPlugin(/* ... */);
await createRxDatabase({/* ... */});
```
Also `removeDatabase()` is renamed to `removeRxDatabase()` and `plugin()` is now `addRxPlugin()`.
Same goes for all previous default exports of the plugins.
### Indexes are specified at the top level of the schema definition
[related issue](https://github.com/pubkey/rxdb/issues/1655)
In the past the indexes of a collection had to be specified at the field level of the schema like
```json
{
"firstName": {
"type": "string",
"index": true
}
}
```
This made it complex to list up the index fields which had a bad performance on startup.
To fix this the indexes are now specified at the top level of the schema like
```json
{
"title": "my schema",
"version": 0,
"type": "object",
"properties": {},
"indexes": [
"firstName",
["compound", "index"]
]
}
```
### Encrypted fields at the top level of the schema
Same as the indexes, encrypted fields are now also defined in the top level like
```json
{
"title": "my schema",
"version": 0,
"type": "object",
"properties": {},
"encrypted": [
"password"
]
}
```
### New dev-mode plugin
In the past we had stuff that is only wanted for development in the two plugins `error-messages` and `schema-check`.
Now we have a single plugin `dev-mode` that contains all these checks and development helper functions. I also moved many other checks out of the core-module into dev-mode.
```typescript
import { RxDBDevModePlugin } from 'rxdb/plugins/dev-mode';
addRxPlugin(RxDBDevModePlugin);
```
### New migration plugin
The data migration was not used by all users of this project. Often it was easier to just wipe the local data store and let the client sync everything from the server. Because the migration has so much code, it is now in a separate plugin so that you do not have to ship this code to the clients if not necessary.
```typescript
import { RxDBMigrationPlugin } from 'rxdb/plugins/migration';
addRxPlugin(RxDBMigrationPlugin);
```
### Rewritten key-compression
The key-compression logic was fully coded only for RxDB. This was a problem because it was not usable for other stuff and also badly tested. We had a known problem with nested arrays that caused much confusion because some queries did not find the correct documents.
I now created a npm-package [jsonschema-key-compression](https://github.com/pubkey/jsonschema-key-compression) that has cleaner code, better tests and can also be used for non-RxDB stuff.
If you used the key-compression in the past and have clients out there with old data, you have to find a way to migrate that data by using the json-import or other solutions depending on your project.
### Rewritten query-change-detection to event-reduce
One big benefit of having a [realtime database](../articles/realtime-database.md) is that big performance optimizations can be done when the database knows a query is observed and the updated results are needed continuously. In the past this optimization was done by the internal `queryChangeDetection` which was a big tree of if-else-statements that hopefully worked. This was also the reason why queryChangeDetection was in beta mode and not enabled by default.
After months of research and testing I was able to create [Event-Reduce: An algorithm to optimize database queries that run multiple times](https://github.com/pubkey/event-reduce). This JavaScript module contains an algorithm that is able to optimize realtime queries in a way that was not possible before. The algorithm is not RxDB specific and also heavily tested.
Instead of setting `queryChangeDetection` when creating a `RxDatabase`, you now set `eventReduce` which defaults to `true`.
### find() and findOne() now accepts the full mango query
In the past, only the selector of a query could be passed to `find()` and `findOne()` if you wanted to also do `sort`, `skip` or `limit`, you had to call additional functions like
```typescript
const query = myRxCollection.find({
age: {
$gt: 10
}
}).sort('name').skip(5).limit(10);
```
Now you can pass the full query to the function call like
```typescript
const query = myRxCollection.find({
selector: {
age: {
$gt: 10
}
},
sort: [{name: 'asc'}],
skip: 5,
limit: 10
});
```
### moved query builder to own plugin
The query builder that allowed to create queries like `.where('foo').eq('bar')` etc. was not really used by many people. Most of the time it is better to just pass the full query as simple json object. Also the code for the query builder was big and increased the build size much more than its value added. Only some edge-cases where recursive query modification was needed made the query builder useful. If you still want to use the query builder, you have to import the plugin.
```typescript
import { RxDBQueryBuilderPlugin } from 'rxdb/plugins/query-builder';
addRxPlugin(RxDBQueryBuilderPlugin);
```
### Refactored RxChangeEvent
The whole data structure of `RxChangeEvent` was way more complicated than it had to be. I refactored the whole class to be more simple. If you directly use `RxChangeEvent` in your project you have to adapt to these changes. Also the stream of `RxDatabase().$` will no longer emit the `COLLECTION` event when a new collection is created.
### Internal hash() is now using a salt
The internal hash function was used to store hashes of database passwords to compare them and directly throw errors when the wrong password was used with an existing data set. This was dangerous because you could use rainbow tables or even [just google](https://www.google.com/search?q=e5e9fa1ba31ecd1ae84f75caaa474f3a663f05f4) the hash to find out the plain password. So now the internal hashing is using a salt to prevent these attacks. If you have used the encryption in the past, you have to migrate your internal schema storage.
### Changed default of RxDocument.toJSON()
By default `RxDocument.toJSON()` always returned also the `_rev` field and the `_attachments`. This was confusing behavior which is why I changed the default to `RxDocument().toJSON(withRevAndAttachments = false)`
### Typescript 3.8.0 or newer is required
Because RxDB and some subdependencies extensively use `export type ...` you now need typescript `3.8.0` or newer.
### GraphQL replication will run a schema validation of incoming data
In dev-mode, the GraphQL-replication will run a schema validation of each document that comes from the server before it is saved to the database.
## Internal and other changes
I refactored much internal stuff and moved much code out of the core into the specific plugins.
* Renamed `RxSchema.jsonID` to `RxSchema.jsonSchema`
* Moved remaining stuff of leader-election from core into the plugin
* Merged multiple internal databases for metadata into one `internalStore`
* Removed many runtime type checks that now should be covered by typescript in buildtime
* The GraphQL replication is now out of beta mode
* Removed documentation examples for `require()` CommonJS loading
* Removed `RxCollection.docChanges$()` because all events are from the docs
## Help wanted
RxDB is an open source project an heavily relies on the contribution of its users. There are some things that must be done, but I have no time for them.
### Refactor data-migrator
The current implementation has some flaws and should be completely rewritten.
* It does not use pouchdb's bulkDocs which is much faster
* It could have been written without rxjs and with less code that is easier to understand
* It does not migrate the revisions of documents which causes a problem when replication is used
### Add e2e tests to the react example
The [react example](https://github.com/pubkey/rxdb/tree/master/examples/react) has no end-to-end tests which is why the CI does not ensure that it works all the time. We should add some basic tests like we did for the other example projects.
### Fix pouchdb bug so we can upgrade pouchdb-find
There is a [bug in pouchdb](https://github.com/pouchdb/pouchdb/issues/7810) that prevents the upgrade of `pouchdb-find`. This is why RxDB relies on an old version of `pouchdb-find` that also requires different sub-dependencies. This increases the build size a lot because for example we ship multiple version of `spark-md5` and others.
## About the future of RxDB
At the moment RxDB is a realtime database based on pouchdb. In the future I want RxDB to be a wrapper around pull-based databases that also works with other source-dbs like mongoDB or PostgreSQL. As a start I defined the `RxStorage` interface and created a `RxStoragePouchdb` class that implements it and contains all pouchdb-specific logic. I want to move every direct storage usage into that interface so that later we can create other implementations of it for other source databases.
---
## Seamless P2P Data Sync
# The RxDB Plugin `replication-p2p` has been renamed to `replication-webrtc`
The new documentation page has been moved to [here](./replication-webrtc.md)
---
## PouchDB RxStorage - Migrate for Better Performance
# RxStorage PouchDB
The PouchDB RxStorage is based on the [PouchDB](https://github.com/pouchdb/pouchdb) database. It is the most battle proven RxStorage and has a big ecosystem of adapters. PouchDB does a lot of overhead to enable CouchDB replication which makes the PouchDB RxStorage one of the slowest.
:::warning
The PouchDB RxStorage is removed from RxDB and can no longer be used in new projects. You should switch to a different [RxStorage](./rx-storage.md).
:::
## Why is the PouchDB RxStorage deprecated?
When I started developing RxDB in 2016, I had a specific use case to solve.
Because there was no client-side database out there that fitted, I created
RxDB as a wrapper around PouchDB. This worked great and all the PouchDB features
like the query engine, the adapter system, CouchDB-replication and so on, came for free.
But over the years, it became clear that PouchDB is not suitable for many applications,
mostly because of its performance: To be compliant to CouchDB, PouchDB has to store all
revision trees of documents which slows down queries. Also purging these document revisions [is not possible](https://github.com/pouchdb/pouchdb/issues/802)
so the database storage size will only increase over time.
Another problem was that many issues in PouchDB have never been fixed, but only closed by the issue-bot like [this one](https://github.com/pouchdb/pouchdb/issues/6454). The whole PouchDB RxStorage code was full of [workarounds and monkey patches](https://github.com/pubkey/rxdb/blob/285c3cf6008b3cc83bd9b9946118a621434f0cff/src/plugins/pouchdb/pouch-statics.ts#L181) to resolve
these issues for RxDB users. Many these patches decreased performance even further. Sometimes it was not possible to fix things from the outside, for example queries with `$gt` operators return [the wrong documents](https://github.com/pouchdb/pouchdb/pull/8471) which is a no-go for a production database
and hard to debug.
In version [10.0.0](./releases/10.0.0.md) RxDB introduced the [RxStorage](./rx-storage.md) layer which
allows users to swap out the underlying storage engine where RxDB stores and queries documents from.
This allowed to use alternatives from PouchDB, for example the [IndexedDB RxStorage](./rx-storage-indexeddb.md) in browsers
or even the [FoundationDB RxStorage](./rx-storage-foundationdb.md) on the server side.
There where not many use cases left where it was a good choice to use the PouchDB RxStorage. Only replicating with a
CouchDB server, was only possible with PouchDB. But this has also changed. RxDB has [a plugin](./replication-couchdb.md) that allows
to replicate clients with any CouchDB server by using the [RxDB Sync Engine](./replication.md). This plugins work with any RxStorage so that it is not necessary to use the PouchDB storage.
Removing PouchDB allows RxDB to add many awaited features like filtered change streams for easier replication and permission handling. It will also free up development time.
If you are currently using the PouchDB RxStorage, you have these options:
- Migrate to another [RxStorage](./rx-storage.md) (recommended)
- Never update RxDB to the next major version (stay on older 14.0.0)
- Fork the [PouchDB RxStorage](./rx-storage-pouchdb.md) and maintain the plugin by yourself.
- Fix all the [PouchDB problems](https://github.com/pouchdb/pouchdb/issues?q=author%3Apubkey) so that we can add PouchDB to the RxDB Core again.
## Pros
- Most battle proven RxStorage
- Supports replication with a CouchDB endpoint
- Support storing [attachments](./rx-attachment.md)
- Big ecosystem of adapters
## Cons
- Big bundle size
- Slow performance because of revision handling overhead
## Usage
```ts
import { createRxDatabase } from 'rxdb';
import { getRxStoragePouch, addPouchPlugin } from 'rxdb/plugins/pouchdb';
addPouchPlugin(require('pouchdb-adapter-idb'));
const db = await createRxDatabase({
name: 'exampledb',
storage: getRxStoragePouch(
'idb',
{
/**
* other pouchdb specific options
* @link https://pouchdb.com/api.html#create_database
*/
}
)
});
```
## Polyfill the `global` variable
When you use RxDB with **angular** or other **webpack** based frameworks, you might get the error:
```html
Uncaught ReferenceError: global is not defined
```
This is because pouchdb assumes a nodejs-specific `global` variable that is not added to browser runtimes by some bundlers.
You have to add them by your own, like we do [here](https://github.com/pubkey/rxdb/blob/master/examples/angular/src/polyfills.ts).
```ts
(window as any).global = window;
(window as any).process = {
env: { DEBUG: undefined },
};
```
## Adapters
[PouchDB has many adapters for all JavaScript runtimes](./adapters.md).
## Using the internal PouchDB Database
For custom operations, you can access the internal PouchDB database.
This is dangerous because you might do changes that are not compatible with RxDB.
Only use this when there is no way to achieve your goals via the RxDB API.
```javascript
import {
getPouchDBOfRxCollection
} from 'rxdb/plugins/pouchdb';
const pouch = getPouchDBOfRxCollection(myRxCollection);
```
---
## RxDB Tradeoffs - Why NoSQL Triumphs on the Client
# RxDB Tradeoffs
[RxDB](https://rxdb.info) is client-side, [offline first](./offline-first.md) Database for JavaScript applications.
While RxDB could be used on the server side, most people use it on the client side together with an UI based application.
Therefore RxDB was optimized for client side applications and had to take completely different tradeoffs than what a server side database would do.
## Why not SQL syntax
When you ask people which database they would want for browsers, the most answer I hear is *something SQL based like SQLite*.
This makes sense, SQL is a query language that most developers had learned in school/university and it is reusable across various database solutions.
But for RxDB (and other client side databases), using SQL is not a good option and instead it operates on document writes and the JSON based **[Mango-query](https://github.com/cloudant/mango)** syntax for querying.
```ts
// A Mango Query
const query = {
selector: {
age: {
$gt: 10
},
lastName: 'foo'
},
sort: [{ age: 'asc' }]
};
```
### SQL is made for database servers
SQL is made to be used to run operations against a database server. You send a SQL string like ```SELECT SUM(column_name)...``` to the database server and the server then runs all operations required to calculate the result and only send back that result.
This saves performance on the application side and ensures that the application itself is not blocked.
But RxDB is a client-side database that runs **inside** of the application. There is no performance difference if the `SUM()` query is run inside of the database or at the application level where a `Array.reduce()` call calculates the result.
### Typescript support
SQL is `string` based and therefore you need additional IDE tooling to ensure that your written database code is valid.
Using the Mango Query syntax instead, TypeScript can be used validate the queries and to autocomplete code and knows which fields do exist and which do not. By doing so, the correctness of queries can be ensured at compile-time instead of run-time.
### Composeable queries
By using JSON based Mango Queries, it is easy to compose queries in plain JavaScript.
For example if you have any given query and want to add the condition `user MUST BE 'foobar'`, you can just add the condition to the selector without having to parse and understand a complex SQL string.
```ts
query.selector.user = 'foobar';
```
Even merging the selectors of multiple queries is not a problem:
```ts
queryA.selector = {
$and: [
queryA.selector,
queryB.selector
]
};
```
## Why Document based (NoSQL)
Like other NoSQL databases, RxDB operates data on document level. It has no concept of tables, rows and columns. Instead we have collections, documents and fields.
### Javascript is made to work with objects
### Caching
### EventReduce
### Easier to use with typescript
Because of the document based approach, TypeScript can know the exact type of the query response while a SQL query could return anything from a number over a set of rows or a complex construct.
## Why no transactions
- Does not work with offline-first
- Does not work with multi-tab
- Easier conflict handling on document level
-- Instead of transactions, rxdb works with revisions
## Why no relations
- Does not work with easy replication
## Why is a schema required
- migration of data on clients is hard
- Why jsonschema
##
---
## Why NoSQL Powers Modern UI Apps
import {VideoBox} from '@site/src/components/video-box';
# Why UI applications need NoSQL
[RxDB](https://rxdb.info), a client side, offline first, JavaScript database, is now several years old.
Often new users appear in the chat and ask for that one simple feature:
They want to store and query **relational data**.
> So why not just implement SQL?
All these client databases out there have on some kind of document based, NoSQL like, storage engine. PouchDB, Firebase, AWS Datastore, RethinkDB's [Horizon](https://github.com/rethinkdb/horizon), Meteor's [Minimongo](https://github.com/mWater/minimongo), [Parse](https://parseplatform.org/), [Realm](https://realm.io/). They all do not have real relational data.
They might have some kind of weak relational foreign keys like the [RxDB Population](./population.md)
or the [relational models](https://docs.amplify.aws/lib/datastore/relational/q/platform/js/) of AWS Datastore.
But these relations are weak. The foreign keys are not enforced to be valid like in PostgreSQL, and you cannot query
the rows with complex subqueries over different tables or collections and then make mutations based on the result.
There must be a reason for that. In fact, there are multiple of them and in the following I want to show you why you can neither have, nor want real relational data when you have a client-side database with replication.
## Transactions do not work with humans involved
On the server side, transactions are used to run steps of logic inside of a self contained `unit of work`. The database system ensures that multiple transactions do not run in parallel or interfere with each other.
This works well because on the server side you can predict how longer everything takes. It can be ensured that one transaction does not block everything else for too long which would make the system not responding anymore to other requests.
When you build a UI based application that is used by a real human, you can no longer predict how long anything takes.
The user clicks the edit button and expects to not have anyone else change the document while the user is in edit mode.
Using a transaction to ensure nothing is changed in between, is not an option because the transaction could be open for a long
time and other background tasks, like replication, would no longer work.
So whenever a human is involved, this kind of logic has to be implemented using other strategies. Most NoSQL databases like [RxDB](./) or [CouchDB](./replication-couchdb.md) use a system based on [revision and conflicts](./transactions-conflicts-revisions.md) to handle these.
## Transactions do not work with offline-first
When you want to build an [offline-first](./offline-first.md) application, it is assumed that the user can also read and write data, even when the device has lost the connection to the backend.
You could use database transactions on writes to the client's database state, but enforcing a transaction boundary across other instances like clients or servers, is not possible when there is no connection.
On the client you could run an update query where all `color: red` rows are changed to `color: blue`, but this would not guarantee that there will still be other `red` documents when the client goes online again and restarts the replication with the server.
```sql
UPDATE docs
SET docs.color = 'red'
WHERE docs.color = 'blue';
```
## Relational queries in NoSQL
What most people want from a relational database, is to run queries over multiple tables.
Some people think that they cannot do that with NoSQL, so let me explain.
Let's say you have two tables with `customers` and `cities` where each city has an `id` and each customer has a `city_id`. You want to get every customer that resides in `Tokyo`. With SQL, you would use a query like this:
```sql
SELECT *
FROM city
WHERE city.name = 'Tokyo'
LEFT JOIN customer ON customer.city_id = city.id;
```
With **NoSQL** you can just do the same, but you have to write it manually:
```typescript
const cityDocument = await db.cities.findOne().where('name').equals('Tokyo').exec();
const customerDocuments = await db.customers.find().where('city_id').equals(cityDocument.id).exec();
```
So what are the differences? The SQL version would run faster on a remote database server because it would aggregate all data there and return only the customers as result set. But when you have a local database, it is not really a difference. Querying the two tables by hand would have about the same performance as a JavaScript implementation of SQL that is running locally.
The main benefit from using SQL is, that the SQL query runs inside of a **single transaction**. When a change to one of our two tables happens, while our query runs, the SQL database will ensure that the write does not affect the result of the query. This could happen with NoSQL, while you retrieve the city document, the customer table gets changed and your result is not correct for the dataset that was there when you started the querying. As a workaround, you could observe the database for changes and if a change happened in between, you have to re-run everything.
## Reliable replication
In an offline first app, your data is replicated from your backend servers to your users and you want it to be reliable.
The replication is **reliable** when, no matter what happens, every online client is able to run a replication
and end up with the **exact same** database state as any other client.
Implementing a reliable replication protocol is hard because of the circumstances of your app:
- Your users have unknown devices.
- They have an unknown internet speed.
- They can go offline or online at any time.
- Clients can be offline for a several days with un-synced changes.
- You can have many users at the same time.
- The users can do many database writes at the same time to the same entities.
Now lets say you have a SQL database and one of your users, called Alice, runs a query that mutates some rows, based on a condition.
```sql
# mark all items out of stock as inStock=FALSE
UPDATE
Table_A
SET
Table_A.inStock = FALSE
FROM
Table_A
WHERE
Table_A.amountInStock = 0
```
At first, the query runs on the local database of Alice and everything is fine.
But at the same time Bob, the other client, updates a row and sets `amountInStock` from `0` to `1`.
Now Bob's client replicates the changes from Alice and runs them. Bob will end up with a different database state than Alice because on one of the rows, the `WHERE` condition was not met. This is not what we want, so our replication protocol should be able to fix it. For that it has to reduce all mutations into a deterministic state.
Let me loosely describe how "many" SQL replications work:
Instead of just running all replicated queries, we remember a list of all past queries. When a new query comes in that happened `before` our last query, we roll back the previous queries, run the new query, and then re-execute our own queries on top of that. For that to work, all queries need a timestamp so we can order them correctly. But you cannot rely on the clock that is running at the client. Client side clocks drift, they can run in a different speed or even a malicious client modifies the clock on purpose. So instead of a normal timestamp, we have to use a [Hybrid Logical Clock](https://jaredforsyth.com/posts/hybrid-logical-clocks/) that takes a client generated id and the number of the clients query into account. Our timestamp will then look like `2021-10-04T15:29.40.273Z-0000-eede1195b7d94dd5`. These timestamps can be brought into a deterministic order and each client can run the replicated queries in the same order.
While this sounds easy and realizable, we have some problems:
This kind of replication works great when you replicate between multiple SQL servers. It does not work great when you replicate between a single server and many clients.
1. As mentioned above, clients can be offline for a long time which could require us to do many and heavy rollbacks on each client when someone comes back after a long time and replicates the change.
2. We have many clients where many changes can appear and our database would have to roll back many times.
3. During the rollback, the database cannot be used for read queries.
4. It is required that each client downloads and keeps the whole query history.
With **NoSQL**, replication works different. A new client downloads all current documents and each time a document changes, that document is downloaded again. Instead of replicating the query that leads to a data change, we just replicate the changed data itself. Of course, we could do the same with SQL and just replicate the affected rows of a query, like WatermelonDB [does it](https://youtu.be/uFvHURTRLxQ?t=1133). This was a clever way to go for WatermelonDB, because it was initially made for React Native and did want to use the fast SQLite instead of the slow [AsyncStorage](https://medium.com/@Sendbird/extreme-optimization-of-asyncstorage-in-react-native-b2a1e0107b34). But in a more general view, it defeats the whole purpose of having a replicating relational database because you have transactions locally, but these transactions become **meaningless** as soon as the data goes through the replication layer.
## Server side validation
Whenever there is client-side input, it must be validated on the server.
On a NoSQL database, validating a changed document is trivial. The client sends the changed document to the server, and the server can then check if the user was allowed to modify that one document and if the applied changes are ok.
Safely validating a SQL query is up to impossible.
- You first need a way to parse the query with all this complex SQL syntax and keywords.
- You have to ensure that the query does not DOS your system.
- Then you check which rows would be affected when running the query and if the user was allowed to change them
- Then you check if the mutation to that rows are valid.
For simple queries like an insert/update/delete to a single row, this might be doable. But a query with 4 `LEFT JOIN` will be hard.
## Event optimization
With NoSQL databases, each write event always affects exactly one document. This makes it easy to optimize the processing of events at the client. For example instead of handling multiple updates to the same document, when the user comes online again, you could skip everything but the last event.
Similar to that you can optimize observable query results. When you query the `customers` table you get a query result of 10 customers. Now a new customer is added to the table and you want to know how the new query results look like. You could analyze the event and now you know that you only have to add the new customer to the previous results set, instead of running the whole query again. These types of optimizations can be run with all NoSQL queries and even work with `limit` and `skip` operators. In RxDB this all happens in the background with the [EventReduce algorithm](https://github.com/pubkey/event-reduce) that calculates new query results on incoming changes.
These optimizations do not really work with relational data. A change to one table could affect a query to any other tables. and you could not just calculate the new results based on the event. You would always have to re-run the full query to get the updated results.
## Migration without relations
Sooner or later you change the layout of your data. You update the schema and you also have to migrate the stored rows/documents. In NoSQL this is often not a big deal because all of your documents are modeled as self containing piece of data. There is an old version of the document and you have a function that transforms it into the new version.
With relational data, nothing is self-contained. The relevant data for the migration of a single row could be inside any other table. So when changing the schema, it will be important which table to migrate first and how to orchestrate the migration or relations.
On client side applications, this is even harder because the client can close the application at any time and the migration must be able to continue.
## Everything can be downgraded to NoSQL
To use an offline first database in the frontend, you have to make it compatible with your backend APIs.
Making software things compatible often means you have to find the **lowest common denominator**.
When you have SQLite in the frontend and want to replicate it with the backend, the backend also has to use SQLite. You cannot even use PostgreSQL because it has a different SQL dialect and some queries might fail. But you do not want to let the frontend dictate which technologies to use in the backend just to make replication work.
With NoSQL, you just have documents and writes to these documents. You can build a document based layer on top of everything by **removing** functionality. It can be built on top of SQL, but also on top of a graph database or even on top of a key-value store like [levelDB](./adapters.md#leveldown) or [FoundationDB](./rx-storage-foundationdb.md).
With that document layer you can build a [Sync Engine](./replication.md) that serves documents sorted by the last update time and there you have a realtime replication.
## Caching query results
Memory is limited and this is especially true for client side applications where you never know how much free RAM the device really has. You want to have a fast realtime UI, so your database must be able to cache query results.
When you run a SQL query like `SELECT ..` the result of it can be anything. An `array`, a `number`, a `string`, a single row, it depends on how the query goes on. So the caching strategy can only be to keep the result in memory, once for each query.
This scales very bad because the more queries you run, the more results you have to store in memory.
When you make a query to a NoSQL collection, you always know how the result will look like. It is a list of documents, based on the collection's schema (if you have one). The result set is stored in memory, but because you get similar documents for different queries to the same collection, we can de-duplicated the documents. So when multiple queries return the same document, we only have it in the cache **once** and each query caches point to the same memory object. So no matter how many queries you make, your cache maximum is the collection size.
## TypeScript support
Modern web apps are build with TypeScript and you want the transpiler to know the types of your query result so it can give you build time errors when something does not match. This is quite easy on document based systems. The typings of for each document of a collection can be generated from the schema, and all queries to that collection will always return the given document type. With SQL you have to manually write the typings for each query by hand because it can contain all these aggregate functions that affect the type of the query's result.
## What you lose with NoSQL
- You can not run relational queries across tables inside a single transaction.
- You can not mutate documents based on a `WHERE` clause, in a single transaction.
- You need to resolve replication conflicts on a per-document basis.
## But there is database XY
Yes, there are SQL databases out there that run on the client side or have replication, but not both.
- WebSQL / [sql.js](https://github.com/sql-js/sql.js/): In the past there was **WebSQL** in the browser. It was a direct mapping to SQLite because all browsers used the SQLite implementation. You could store relational data in it, but there was no concept of replication at any point in time. **sql.js** is an SQLite complied to JavaScript. It has not replication and it has (for now) no persistent storage, everything is stored in memory.
- WatermelonDB is a SQL databases that runs in the client. WatermelonDB uses a document-based replication that is not able to replicate relational queries.
- Cockroach / Spanner/ PostgreSQL etc. are SQL databases with replication. But they run on servers, not on clients, so they can make different trade offs.
# Further read
- Cockroach Labs: [Living Without Atomic Clocks](https://www.cockroachlabs.com/blog/living-without-atomic-clocks/)
- [Transactions, Conflicts and Revisions in RxDB](./transactions-conflicts-revisions.md)
- [Why MongoDB, Cassandra, HBase, DynamoDB, and Riak will only let you perform transactions on a single data item](https://dbmsmusings.blogspot.com/2015/10/why-mongodb-cassandra-hbase-dynamodb_28.html)
- `Make a PR to this file if you have more interesting links to that topic`