How to Save Deterministic Time Series - No Database Needed
I just saw a case where I strongly disagree with using a database: For time series, in this case. In the end I show a better way to deal with deterministic time series data. This blog will show:
- What it means to have deterministic data
- Why it made sense for us to keep it in memory
- How to approach a problem simple first, and optimize later
Let me get out my iPad…
So here’s the situation we’re dealing with: We have some input data, an array of numbers. In our case, it was about 10 different numbers coming from three different models. Pretty complex calculations.
The developer decided to put all of this into a database. The input data was already there, just form fields that users fill out. But then they also stored the resulting time series - essentially a compound interest calculation.
Every year (or in our example, every day), the data changes a bit. The important thing is that the input data completely determines the whole time series. If the input changes, the entire series gets rewritten into the database, projecting into the future.
Now, storing this in a database creates some problems:
- You need to delete all the old data when input changes
- You have to recreate the whole time series
- You must update the graph based on new sums
While this works, it’s not the best approach. Let me show you a better way: store the whole thing in memory.
Why does this make sense? Let’s say we have 100 years, 10 values each, all floats. We’re talking about roughly 10 kilobytes of data. Sure, you can store that in a database, but you can just as easily recreate it on the fly.
The key insight is this: the time series doesn’t change after the initial calculation. One small change in the input propagates through the entire series. This means the time series can be fully generated just from the input data.
We don’t need to store it in the database at all. Instead, we need objects and classes that contain the data and perform the calculations at the right point. We can iterate over an array of objects, sum up the values, and generate our chart data.
This approach is simpler, faster, and cleaner:
- No database overhead
- No need to keep state (it’s all in the input data)
- No migrations necessary
- No data cleanup required
If we need to cache for performance (say, for millions of years), we have options:
- Cache in a single database field
- Use Redis if it’s already in our stack
- Consider a specialized time series database for extreme cases
But for a simple web app with a graph? We don’t need a database at all.
This is the power of thinking through your data structures and understanding the nature of your data. Sometimes, the simplest solution is the best one.
If you like this kind of engineering content for CTOs, hit that like button and subscribe to my channel. Got questions? Drop them in the comments. And if you want to learn more about this then watch full video, check out my video on breadboards.
