10.0.0

One year after version 9.0.0 we now have RxDB version 10.0.0. The main goal of version 10 was to change things that make RxDB ready for the future.

Notice that I use major releases to bundle stuff that breaks the RxDB usage in your project, not to add new features.

The main thing first

In the past, RxDB was build around Pouchdb. Before I started making RxDB I tried to solve the problems of my current project with other existing databases out there. I evaluated all of them and then started using Pouchdb and added many features via plugin. Then I realised it will be easier to create a separate project that wraps around Pouchdb, that was RxDB. Back then pouchdb was the most major browser database out there and it was well maintained and had a big community. But in the last 5 years, things have changed. A big part of the RxDB users do not use couchdb in the backend and do not need the couchdb replication. Therefore they do not really need the overhead with revision handling that slows down the performance of pouchdb. Also there where many other problems with using pouchdb. It is not actively developed, many bugs are not fixed and no new features get added. Also there are many unsolved problems like how to finally delete document data or how to replicate more then 6 databases at the same time, how to use replication without attachments data, and so on...

So for this release, I abstracted all parts that we use from pouchdb into the RxStorage interface. RxDB works on top of any implementation of the RxStorage interface. This means it is now possible to use RxDB together with other underlying storages like SQLite, PostgreSQL, Minimongo, MongoDB, and so on, as long as someone writes the RxStorage class for it.

This means, to create a RxDatabase you have to pass the storage class instead of pouchdb specific settings:

// import pouchdb specific stuff and add pouchdb adapters
import {
    addPouchPlugin,
    getRxStoragePouch
} from 'rxdb/plugins/pouchdb';

// IMPORTANT: Do not use addRxPlugin to add pouchdb adapter, instead use addPouchPlugin
addPouchPlugin(require('pouchdb-adapter-memory'));

import {
    addRxPlugin,
    createRxDatabase,
    randomCouchString,
} from 'rxdb/plugins/core';

// create the database with the storage creator.
const db = await createRxDatabase({
    name: 'mydatabase',
    storage: getRxStoragePouch('memory'),
});

To access the internal pouch instance of a collection, you have to go over the storageInstance:

const pouch = myRxCollection.storageInstance.internals.pouch;

Other breaking changes

Primary key is required

In the past, using a primary key was optional. When no primary key was defined, RxDB filled up the _id field with an uuid-like string which was then used as primary. When I researched on github how people use RxDB, I found out that many use a secondary index for what should be the primary key. Also having the primary key optional, caused much confusing when using RxDB with typescript.

So now the primary key MUST be set when creating a schema for RxDB. Also the primary key is defined with the primaryKey property at the top level of the schema. This ensures that typescript will complain if no primaryKey is defined.

// when using the type `RxJsonSchema<DocType>` the `DocType` is now required
const mySchema: RxJsonSchema<MyDocumentData> = {
    version: 0,
    primaryKey: 'passportId',
    type: 'object',
    properties: {
        passportId: {
            type: 'string'
        }
    },
    // primaryKey is always required
    required: ['passportId']
}

Attachment data must be Blob or Buffer

In the past, an RxAttachment could be stored with Blob, Buffer and string data. If a string was passed, pouchdb internally transformed the data to a Blob or Buffer, depending on in which environment it is running. This behavior caused much trouble and weird edge cases because of how the data is transformed from and to string. So now you can only store Blob or Buffer as attachment data. string is no longer allowed. You can still transform a string to a Blob or Buffer by yourself and then store it.

import { blobBufferUtil } from 'rxdb';

const attachment = await myDocument.putAttachment(
    {
        id: 'cat.txt',
        data: blobBufferUtil.createBlobBuffer('miau', 'text/plain')
        type: 'text/plain'
    }
);

Also putAttachment() now defaults to skipIfSame=true. This means when you write attachment data that already is exactly the same in the database, no write will be done.

Outgoing data is now readonly and deep-frozen

RxDB often uses outgoing data also in the internals. For example the result of a query is not only send to the user, but also used inside of RxDB's query-change-detection. To ensure that mutation of the outgoing data is not changing internal stuff, which would cause strange bugs, outgoing data was always deep-cloned before handing it out to the user. This is a common practice on many javascript libraries.

The problem is that deep-cloning big objects can be very CPU/Memory expensive. So instead of doing a deep-clone, RxDB does now assume that outgoing data is immutable. If the users wants to modify that data, it has be be deep-cloned by the user. To ensure immutability, RxDB runs a deep-freeze in the dev-mode (about same expensive as deep clone). Also typescript will throw a build-time error because we use ReadonlyArray and readonly to define outgoing data immutable. In production-mode, there will be nothing besides typescript that ensures immutability to have best performance.

const data = myRxDocument.toJSON();
data.foo = bar; // This does NOT work!

// instead clone the data before changing it
import { clone } from 'rxjs';
const clonedData = clone(data);
data.foo = bar; // This works!

The in-memory plugin does no longer work.

The in-memory plugin was used to spawn in-memory collections on top of a normal RxCollection. The benefit is to have the data replicated into the memory of the javascript runtime, which allows for faster queries.

After doing many tests and observations, I found out that the in-memory plugin was slow. Really slow, even slower then just using the indexeddb adapter in the browser. You can reproduce my observations at the event-reduce testpage. Here you can see that random-writes+query are slower on the memory-adapter then on indexeddb. The reason for this are the big abstraction layers. Pouchdb uses the adapter system. The memory adapter uses the leveldown abstraction layer. Each write/read goes to the memdown module.

So the in-memory plugin is not working for now. In the future it will be reimplemented in a custom memory based RxStorage class.

note

You can of course still use the pouchdb memory adapter as usual. It is not affected by this change.

What else is a breaking change?

• Removed the deprecated atomicSet(), use atomicPatch() instead.
• Removed the deprecated RxDatabase.collection() use RxDatabase().addCollections() instead.
• Removed plugin hook preCreatePouchDb because it is no longer needed.
• Removed the watch-for-changes plugin. We now overwrite pouchdbs bulkDocs method to generate events. This is faster and more reliable.
• Removed the adapter-check plugin. (The function adapterCheck is move to the pouchdb plugin).
• Calling RxDatabase.server() now returns a promise that resolves when the server is started up.
• Changed the defaults of PouchDBExpressServerOptions from the server() method, by default we now store logs in the tmp folder and the config is in memory.
• Renamed replication-plugin to replication-couchdb to be more consistent in naming like with replication-graphql
  • For the same reason, renamed RxCollection().sync() to RxCollection().syncCouchDB()
• Renamed the functions of the json import/export plugin to be less confusing.
  • dump() is now exportJSON()
  • importDump() is now importJSON()
• RxCollection uses a separate pouchdb instance for local documents, so that they can persist during migrations.
• A JsonSchema must have the required array at the top level and it must contain the primary key.

New features

Composite primary key

You can now use a composite primary key for the schema where you can join different properties of the document data to create a primary key.

const mySchema = {
  keyCompression: true, // set this to true, to enable the keyCompression
  version: 0,
  title: 'human schema with composite primary',
  primaryKey: {
      // where should the composed string be stored
      key: 'id',
      // fields that will be used to create the composed key
      fields: [
          'firstName',
          'lastName'
      ],
      // separator which is used to concat the fields values.
      separator: '|'
  }
  type: 'object',
  properties: {
      id: {
          type: 'string'
      },
      firstName: {
          type: 'string'
      },
      lastName: {
          type: 'string'
      }
  },
  required: [
    'id', 
    'firstName',
    'lastName'
  ]
};

For the future

With these changes, RxDB is now ready for the future plans:

• I want to replace the revision handling of documents with conflict resolution strategies that can always directly resolve conflicts instead of maintaining the revision tree.
• Implement different implementations for RxStorage. I will first work on a memory based version. I am in good hope that the community will create other implementations depending on their needs.

You can help!

There are many things that can be done by you to improve RxDB:

• Check the BACKLOG for features that would be great to have.
• Check the breaking backlog for breaking changes that must be implemented in the future but where I did not had the time yet.
• Check the TODOs in the code. There are many small improvements that can be done for performance and build size.
• Review the code and add tests. I am only a single dude with a laptop. My code is not perfect and much small improvements can be done when people review the code and help me to clarify undefined behaviors.
• Improve the documentation. In the last user survey many users told me that the documentation is not good enough. But I reviewed the docs and could not find clear flaws. The problem is that I am way to deep into RxDB so that I am not able to understand which documentation a newcomer to the project needs. Likely I assume too much knowledge or focus writing about the wrong parts.
• Update the example projects many of them are outdated and need updates.

Discuss!

Please discuss here.