Simple Denormalization for Meteor

Simple Denormalization for Meteor

In this readme, parent always refers to the documents in which the cache is stored, while child refers to the documents that will be cached.

Denormalize

Simple denormalization for Meteor

Introduction

meteor add herteby:denormalize

In this readme, parent always refers to the documents in which the cache is stored, while child refers to the documents that will be cached.

Example: You have two collections - Users and Roles. The Users store the id of any Roles they have been assigned. If you want each User to cache information from any Roles that are assigned to it, the Users would be the _parents and the Roles would be the children, and it would be either a one or many relationship, depending on if a User can have multiple Roles. If you wanted each Role to store a list of all Users which have that role, the Roles would be the parents and the Users would be the children, and it would be an inverse or many-inverse relationship.

Collection.cache(options)

Posts.cache({
  type:'one',
  collection:Meteor.users,
  fields:['username', 'profile.firstName', 'profile.lastName'],
  referenceField:'author_id',
  cacheField:'author'
})
Property Valid values Description
type 'one', 'many', 'inverse' or 'many-inverse' one: The parent stores a single child _id many: The parent stores an array of child _ids inverse: Each child stores a single parent _id many-inverse: Each child stores an array of parent _ids
collection Mongo.Collection The "child collection", from which docs will be cached
fields Array of Strings or Object The fields to include in the cache. It can either look like ['username', 'profile.email'] or {username:1, profile:{email:1}}. For "many", "inverse" and "many-inverse", _id will always be included.
referenceField String For "one" and "many", the field on the parent containing _id of children. For "inverse" and "many-inverse", the field on the children containing the _id of the parent.
cacheField String The field on the parent where children are cached. Can be a nested field, like 'caches.field', but it can not be in the same top level field as the referenceField. For type:'one', cacheField will store a single child. For all others, it will store an array of children.
bypassSchema Boolean (optional) If set to true, it will bypass any collection2 schema that may exist. Otherwise you must add the cacheField to your schema.
Notes and clarification:
  • "one" and "inverse" are many-to-one relationships (with "one", a parent can only have one child, but many parents could have the same child). "many" and "many-inverse" are many-to-many relationships
  • When cacheField is an array (all types except "one"), the order of the children is not guaranteed.
  • When referenceField is an array, if it contains duplicate _ids, they will be ignored. The cacheField will always contain unique children.

Collection.cacheCount(options)

TodoLists.cacheCount({
  collection:Todos,
  referenceField:'list_id',
  cacheField:'counts.important',
  selector:{done:null, priority:{$lt:3}}
})

cacheCount() can be used on "inverse" and "many-inverse" relationships

Property Valid values Description
collection Mongo.Collection The collection in which docs will be counted
referenceField String The field on counted docs which must match the parent _id
cacheField String The field where the count is stored. Can be a nested field like 'counts.all'
selector Mongo selector (optional) Can be used to filter the counted documents. [referenceField]:parent._id will always be included though.
bypassSchema Boolean (optional) If set to true, it will bypass any collection2 schema that may exist. Otherwise you must add the cacheField to your schema.

Collection.cacheField(options)

Meteor.users.cacheField({
  fields:['profile.firstName', 'profile.lastName'],
  cacheField:'fullname',
  transform(doc){
    return doc.profile.firstName + ' ' + doc.profile.lastName
  }
})
Property Valid values Description
fields Array of Strings or Object The fields to watch for changes. It can either look like ['username', 'profile.email'] or {username:1, profile:{email:1}}
cacheField String Where the result is stored. Can be nested like 'computed.fullName'
transform Function (optional) The function used to compute the result. If not defined, the default is to return a string of all watched fields concatenated with ', '
The document provided to the function only contains the fields specified in fields
bypassSchema Boolean (optional) If set to true, it will bypass any collection2 schema that may exist. Otherwise you must add the cacheField to your schema.

Note: The transform function could also fetch data from other collections or through HTTP if you wanted, as long as it's done synchronously.

Migration

If you decide to add a new cache or change the cache options on a collection that already contains documents, those documents need to be updated. There are two options for this:

migrate(collectionName, cacheField, [selector])

import {migrate} from 'meteor/herteby:denormalize'
migrate('users', 'fullName')
migrate('users', 'fullAddress', {fullAddress:{$exists:false}})

This updates the specified cacheField for all documents in the collection, or all documents matching the selector. Selector can also be an _id.

autoMigrate()

import {autoMigrate} from 'meteor/herteby:denormalize'
autoMigrate() //should be called last in your server code, after all caches have been declared

When autoMigrate() is called, it checks all the caches you have declared against a collection (called _cacheMigrations in the DB) to see wether they need to be migrated. If any do, it will run a migration on them, and then save the options to _cacheMigrations, so that it won't run again unless you change any of the options. If you later for example decide to add another field to a cache, it will rerun automatically.

One thing it does not do is remove the old cacheField, if you were to change the name or remove the cache. That part you have to do yourself.

Note: it does not check the documents, it just checks each cache declaration, so it won't thrash your DB on server start going through millions of records (unless something needs to be updated).

Nested referenceFields

For "one" and "inverse", nested referenceFields are simply declared like referenceField:'nested.reference.field'

For "many" and "many-inverse", if the referenceField is an Array containing objects, a colon is used to show where the Array starts.

Example:

If the parent doc looks like this:

{
  //...
  references:{
    users:[{_id:'user1'}, {_id:'user2'}]
  }
}

The referenceField string should be 'references.users:_id'

Recursive caching

You can use the output (the cacheField) of one cache function as one of the fields to be cached by another cache function, or even as the referenceField. They will all be updated correctly. This way you can create "chains" connecting three or more collections.

In the examples below, all cache fields start with _, which may be a good convention to follow for all your caches.

Use cacheField() to cache the sum of all cached items from a purchase
Bills.cacheField({
  fields:['_items'],
  cacheField:'_sum',
  transform(doc){
    return _.sum(_.map(doc._items, 'price'))
  }
})
Caching the cacheFields of another cache
Bills.cache({
  cacheField:'_items',
  collection:Items,
  type:'many',
  referenceField:'item_ids',
  fields:['name', 'price']
})
Customers.cache({
  cacheField:'_bills',
  collection:Bills,
  type:'inverse',
  referenceField:'customer_id',
  fields:['_sum', '_items']
})
Using the cacheField of another cache as referenceField
Customers.cache({
  cacheField:'_bills2',
  collection:Bills,
  type:'inverse',
  referenceField:'customer_id',
  fields:['item_ids', '_sum']
})
Customers.cache({
  cacheField:'_items',
  collection:Items,
  type:'many',
  referenceField:'_bills2:item_ids',
  fields:['name', 'price']
})
Incestuous relationships

With this fun title I'm simply referring to caches where the parent and child collections are the same.

Meteor.users.cache({
  cacheField:'_friends',
  collection:Meteor.users,
  type:'many',
  referenceField:'friend_ids',
  fields:['name', 'profile.avatar']
})

This works fine, but there is one thing you can not do - cache the cacheField of a document in the same collection - in this example it would be caching the friends of a users friends. This would lead to an infinite loop and infinitely growing caches.

When are the caches updated?

The caches for cache() and cacheCount() are updated immediately and synchronously.

Posts.cache({
  cacheField:'_author',
  //...
})
Posts.insert({_id:'post1', author_id:'user1'})
Posts.findOne('post1')._author //will contain the cached user

cache() uses 5 hooks: parent.after.insert, parent.after.update, child.after.insert, child.after.update and child.after.remove. There are then checks done to make sure it doesn't do unnecessary updates.

Basically you should always be able to rely on the caches being updated. If they're not, that should be considered a bug.

However, to avoid a complicated issue with "recursive caching", the update of cacheField() is always deferred.

Meteor.users.cacheField({
  fields:['address', 'postalCode', 'city'],
  cacheField:'_fullAddress',
})
Meteor.users.insert({_id:'user1', ...})
Meteor.users.findOne('user1')._fullAddress //will not contain the cached address yet
Meteor.setTimeout(()=>{
  Meteor.users.findOne('user1')._fullAddress //now it should be there
}, 50)

Note: Since this package relies on collection-hooks, it won't detect any updates you do to the DB outside of Meteor. To solve that, you can call the migrate() function afterwards.

Testing the package

meteor test-packages packages/denormalize --driver-package=practicalmeteor:mocha

(Then open localhost:3000 in your browser) The package currently has over 120 tests Note: The "slowness warnings" in the results are just due to the asynchronous tests

Download Details:

Author: Herteby

Source Code: https://github.com/Herteby/denormalize

meteor javascript

Bootstrap 5 Complete Course with Examples

Bootstrap 5 Tutorial - Bootstrap 5 Crash Course for Beginners

Nest.JS Tutorial for Beginners

Hello Vue 3: A First Look at Vue 3 and the Composition API

Building a simple Applications with Vue 3

Deno Crash Course: Explore Deno and Create a full REST API with Deno

How to Build a Real-time Chat App with Deno and WebSockets

Convert HTML to Markdown Online

HTML entity encoder decoder Online

JavaScript Shopping Cart - Javascript Project for Beginners

JavaScript Shopping Cart - javascript shopping cart tutorial for beginnersBuy me a coffee 🍺 https://www.paypal.com/paypalme/ziddahSource Code: https://bit....

The essential JavaScript concepts that you should understand

The essential JavaScript concepts that you should understand - For successful developing and to pass a work interview

Data Types In JavaScript

JavaScript data types are kept easy. While JavaScript data types are mostly similar to other programming languages; some of its data types can be unique. Here, we’ll outline the data types of JavaScript.

Introduction With Basic JavaScript

Introduction With Basic JavaScript - Unlike most programming languages, the JavaScript language has no concept of input or output. It is designed to run as a scripting language in a host environment, and it is up to the host environment to provide mechanisms for communicating with the outside world.

JavaScript Memory Management System

The main goal of this article is help to readers to understand that how memory management system performs in JavaScript. I will use a shorthand such as GC which means Garbage Collection. When the browsers use Javascript, they need any memory location to store objects, functions, and all other things. Let’s deep in dive that how things going to work in GC.