Data Consistency Approaches for Rails Apps

I worked for a year on a consistency system for contract calculations. It went over multiple invoices, changes in contract prices, invoice runs and discounts.

During this time we figured out some best practices that saved our a** more than a few times.

How internal inconsistency and sloppiness can break a Rails app

Simple example: imagine you have a User model but you want to move some data out of it and create a Person model. That’s a good idea when you want to separate concerns, or you might need a Person who doesn’t need to log in.

Have you heard of the devise gem by now? Ask you devs about it, you are likely to use it. It’s often a good idea to separate data here.

Now you want to be GPDR compliant and have users delete their accounts. A User of course destroys their Person object along with it - but now the Event model refers to a Person that doesn’t exist any more. ⚡️

Possible side effects happening now:

0. Other users looking at events will get a 500 error
1. Jobs that create reports will fail

A pragmatic approach to ensuring internal consistency

This post was inspired by Chris from Stockholm who argued that consistency is never 100% reachable. I do agree with it for external consistency (described below).

However, for internal consistency there’s a simple remedy that you can deploy into your app right now. Take this code and adapt it to your needs:

class CheckRecordValidityJob < ApplicationJob
  queue_as :default

  def perform
    invalid_records = {}
    ActiveRecord::Base.descendants.each do |model|
      next if model.table_name.nil?
      # TODO: this is not performant, but you get the idea:
      invalid_records[model.table_name.to_sym] = model.all.select { |record| !record.valid? }
    end

    invalid_records_count = invalid_records.values.sum(&:count)

    if invalid_records_count > 0
      # Send email with list of invalid records
      email = AdminMailer.invalid_record_alert(invalid_records)
      email.deliver_now
    end
  end
end

Tougher cases

The above example only shows relational inconsistency that can easily be found, and even prevented on the database level (null: false on a foreign key in the database schema).

Now, there are many more cases where you cannot enforce integrity. I’ll give an abstract example which almost happened like this in a production system of a client.

Imagine you run a software like JIRA, where you create invoices based on usage. Every invoice is a sum of the number of users in the client’s team. If you have an invoice that says “5 users, $10 each”, and a user deletes their record, the invoice might say $50, but the database says “only 4 users”.

“Of course people think about this” you might say - but imagine you are bootstrapping a SaaS, then it’s very likely you need to focus on other features. Now, this invoicing system comes with some technical debt, and that’s good - for now.

For bigger systems with thousands of users, this flaw now becomes a liability. And if it’s never fixed a customer might come along and request the user statement of a year ago. For some industries this can be a huge problem.

Tip: Store as much data as you can

Example: Document generation. How this usually works is you have a template rendering service which gathers data from your database, populates a template, renders a PDF and then stores it somewhere.

What we found is crucial: always store the template data in the database. Just a json field with key-value works for template variables before you create the PDF.

we failed at building that 2 times before we got it right. In one case, we found inconsistencies in the database and wanted to double-check with the PDFs. Problem was that we didn’t follow this best-practice yet.

The result?

We were forced to parse PDFs and store the info back into the DB. That took us days of writing a parser and then executing it over night for hundreds of thousands of PDFs.

External consistency need caution

Then there’s consistency that’s external to a system. […] there is no way to be 100% correct using only the tools that are inside the system. You need external audits, reality checks, periodic reconciliation.

I almost forgot about external consistency until I read the above lines in Daniel’s post. Yes, there’s always uncertainty when using external systems, it’s better not to assume they always tell the truth.

Tip: How to approach for APIs

Make a table called api_messages and store all requests and responses in it. That way you can check if a request suddenly deviates from the same request a year ago.

It might also help you fix problems in your own database later, as you always have the raw data saved.

Tests only go so far

I’ve worked with system that had 100% test coverage and they still were inconsistent. You cannot ever get to all edge cases:

100% code coverage only says that the code is covered, not that all possible inputs are dealt with

TODO: Look up the definition of code coverage. I’d bet that some tools check for edge cases in input data

A big enough system will always deal with it

Large software systems have one thing in common: they grew to that point, and growth also means change. Ideally the underlying data also changes (that’s why we use data migrations), but there’s always an error.

This error eventually compounds over the time a software system changes, it’ll also compound over the number of users in the system.

Guideline: Never let the customer pay for your mistakes

Vidar points out on HN

Customers won’t complain about being under-charged, so a loss of data for a system like that is fine[…]

In the system we were maintaining we gave discounts at one point. Customer happiness was worth more than the loss in not-invoiced quota.

“Why did you invoice me this?”

We sometimes had customers asking for explanations. Almost always these customers were happy when getting a discount. That cost the system money, but it was better than a possible complaint on Google Reviews.

Do you havea story about data inconsistency? Let me know!