Chapter 2: The Second Database Tax

The Sync Pipeline

Most teams running both databases have constructed a synchronization mechanism between them. MongoDB handles the document writes. PostgreSQL handles the reporting, the cross-entity joins, the transactional workflows. Between the two sits a pipeline that keeps them approximately in agreement. I say approximately because precision would be misleading.

The most common implementation uses Debezium with Kafka. Debezium's MongoDB connector uses Change Streams — the recommended capture mode since MongoDB 4.x — to stream document changes into Kafka topics. From there, a consumer writes those changes into PostgreSQL. Other teams use Change Streams directly with a custom consumer, or a scheduled ETL job, or — and I mention this without judgment — a script that someone wrote during an incident and nobody has revisited since. Each of these approaches works. Each has costs that the database invoice does not itemize.

The infrastructure cost. Kafka and Debezium constitute their own cluster to deploy, monitor, and maintain. Kafka requires brokers, ZooKeeper or KRaft, topic configuration, retention policies, and its own alerting. Debezium requires connector configuration, offset management, and restart handling. A custom ETL requires application code to maintain, test, and debug. Whichever path you have chosen, the sync pipeline is a third system — a system that exists for the sole purpose of moving data between two databases that would not need to communicate if the data lived in one place.

The lag. Under normal load, Debezium plus Kafka introduces one to three seconds of latency on incremental changes. Under heavy load, during connector restarts, or during catch-up after an outage, that lag extends to minutes or hours. During that window, the two databases disagree. A report built from PostgreSQL will not reflect the writes that MongoDB accepted moments ago. Two sources of truth, each telling a slightly different version of the present moment. A database that disagrees with itself is not, I would respectfully suggest, a source of truth at all.

The failure modes. The pipeline can fall behind. It can crash and restart. It can lose its position entirely. Debezium stores a resume token that points to a position in MongoDB's oplog. If the oplog rolls over before the connector catches up — which happens when write volume exceeds the oplog's size-based retention — the resume token becomes invalid. Events between the last processed position and the current oplog window are not delayed. They are lost. The connector cannot resume, and the data in PostgreSQL is now silently incomplete.

When a sync pipeline fails, the symptom is not an error message. It is data that quietly stops arriving. Reports become stale. Queries return yesterday's answers to today's questions. The first person to notice is usually someone in product or finance, asking why the numbers look wrong. By the time the question is asked, the answer has been wrong for hours.

The Backup Problem

A backup that restores MongoDB to 2:00 AM and PostgreSQL to 2:03 AM has a three-minute consistency gap. Every document written in those three minutes exists in one database but not the other. Every operation that spanned both — a user registration in MongoDB, a welcome email queued in PostgreSQL — is partially restored. A backup that covers half your data is not half a backup. It is a confidence you have not yet been asked to test.

No tooling exists to coordinate point-in-time recovery across MongoDB and PostgreSQL simultaneously. MongoDB uses oplog-based PITR. PostgreSQL uses WAL-based PITR. Each has its own timestamps, its own recovery mechanisms, its own backup formats. Restoring both to a consistent shared point in time is a manual timestamp-matching exercise, performed under pressure, with the sync pipeline's state at the time of backup adding a variable that no restore procedure fully accounts for.

The practical result: two backup schedules to maintain, two retention policies to configure, two restore procedures to document and rehearse, and two RPO/RTO calculations that must account for the gap between the two restored states. Your effective RPO is not the better of the two schedules. It is the worse of the two, plus the sync pipeline's lag at the moment of the backup.

The alternative, in a single-database architecture, is pg_dump or pg_basebackup. One command. One point in time. Every relational table, every document collection, every index, every function, every permission. One restore procedure that your team can practice, document, and execute with confidence when the phone rings at 3 AM. There is a certain peace in knowing that your backup is complete — not approximately complete, not complete pending pipeline state, but complete.

The Connection Tax

Two databases means two sets of infrastructure to operate. None of it is individually difficult. All of it doubles the surface area for the kind of small operational mistakes that become large operational incidents.

Two connection pools, each with different configuration models, different timeout semantics, and different behavior under exhaustion. MongoDB's pool and PostgreSQL's — often PgBouncer or pgcat — tune differently, fail differently, and recover differently. When one pool exhausts during a traffic spike, the symptoms depend on which database ran out of connections first, and the fix depends on knowing which pool's configuration to adjust. An engineer who has diagnosed connection exhaustion in PostgreSQL may find that the same intuition leads them astray in MongoDB, where the pool behavior, the error messages, and the recovery pattern are all different.

Two sets of credentials to rotate on two schedules, in two places, with two revocation procedures when something is compromised. A credential rotation that updates PostgreSQL but misses MongoDB — or vice versa — produces a partial outage that is functional in some parts of the application and broken in others. The debugging starts with "which database can't the application reach?" and the answer is not always obvious from the error logs.

Two connection string formats — MongoDB's mongodb+srv:// with its DNS SRV discovery and automatic TLS, PostgreSQL's postgres:// with its explicit hosts, explicit sslmode, and explicit failover syntax. Different parameter names. Different TLS behavior. Different failover mechanisms. A developer who has memorized one will reach for the documentation when configuring the other.

Two monitoring dashboards. Two sets of alerts. Two on-call runbooks. Two sets of metrics — connection counts, query latency, replication lag, storage utilization — each reported in different units, with different semantics, on different graphs.

I do not list these to alarm you. I list them because operational complexity is not measured in any single item. It is measured in the total number of things that must go right simultaneously. Each additional item is small. The sum is not. And the sum is paid not once, but every day your architecture includes a second database.

The Knowledge Tax

Your team must be fluent in two database paradigms. Not casually familiar — fluent. Fluent enough to diagnose production issues under time pressure, in the middle of the night, without reaching for the documentation.

MongoDB has its own query language: find() with filter documents, aggregate() with pipeline stages, update operators like $set and $push and $addToSet. PostgreSQL has SQL. The two share no syntax. A developer who can construct a complex aggregation pipeline in MongoDB cannot transfer that skill to a PostgreSQL window function. The knowledge does not translate. It must be learned separately, maintained separately, and applied under pressure separately.

The sync pipeline adds a third body of knowledge. Kafka topics, consumer groups, offsets, connector configuration, Debezium's specific behavior around resume tokens and schema changes — or, if you built a custom ETL, whatever conventions and edge cases that code embodies. Someone on your team understands this system. I would encourage you to confirm that it is more than one person.

Hiring narrows. Your job posting asks for experience with both MongoDB and PostgreSQL. You have either limited your candidate pool to developers who know both, or accepted that every new hire spends their first weeks learning whichever one they do not. Neither outcome is catastrophic. Both are costs. And both compound over time — every team member who leaves takes their dual-database knowledge with them, and every replacement starts the learning curve again.

The application code itself bears the tax. Your codebase has two database clients, two query builders, two sets of error handling patterns, and conditional logic that decides which database to talk to for which operation. The data access layer — the part of your application that should be the simplest — becomes the most complex, because it must speak two languages and route traffic between two systems that have different connection semantics, different error codes, and different retry behavior. Code that serves two masters is code that serves neither particularly well.

Incident response is where the knowledge tax collects its highest rate. When something breaks at 3 AM, the on-call engineer needs to determine which database the problem is in, how to query it for diagnostic information, and how the sync pipeline connects the two. The debugging surface area is not additive — it is multiplicative. The failure could be in MongoDB, in PostgreSQL, in the pipeline, or in the interaction between any two of them. Expertise that must span three systems under pressure is expensive expertise. It is also, in a single-database architecture, unnecessary expertise.

The Consistency Gap

Two databases that are eventually consistent with each other create a class of bugs that live in no codebase. They live in the architecture. They are invisible to your test suite, because your test suite does not model sync pipeline lag. They surface in production, in the space between a write and a read, and they look like someone else's mistake.

The everyday version is the one your team encounters most. A user updates their profile. The write goes to MongoDB. A downstream service reads that profile to send a notification — a welcome email, a confirmation, an alert. That service reads from PostgreSQL, through the sync pipeline. The pipeline is three seconds behind. The notification goes out with the old name, the old address, the old preferences. The user sees stale data in an email that your application sent correctly, from a database that answered truthfully, using data that was accurate three seconds ago. No code is wrong. Every system behaved as designed. The architecture is the bug.

This is not an edge case. It is the default behavior of any system where one database accepts writes and another serves reads with a sync pipeline between them. Every read from the lagging database returns data from the recent past. Most of the time, the recent past is close enough. When it is not — when a user sees their old name in a notification they triggered thirty seconds ago — the result is a support ticket that no engineer can reproduce, because by the time they look, the pipeline has caught up.

The more consequential version involves transactions that span both databases. An order placed in MongoDB. A payment recorded in PostgreSQL. There is no cross-database transaction — MongoDB and PostgreSQL cannot participate in the same BEGIN/COMMIT. You can approximate it with saga patterns and compensating transactions, which means you have added distributed systems coordination to your application logic because your data lives in two places. This is solvable. It is also complexity that would not exist if the data shared a single transactional boundary.

And there is the subtle version that costs nothing in engineering time and everything in organizational trust. A product manager pulls a report from PostgreSQL. A developer queries MongoDB directly. The numbers disagree. Both are correct — at different points in time. The conversation about which number is "right" is a conversation that should not need to happen. Data that is correct at two different points in time is, for the purposes of any decision made from it, correct at neither.