Let me take you back to a project we were all excited about.
We were building a key capability for a super app — an Offer Engine — and like many modern-day projects, we decided to go with microservices. The architecture made sense. We identified the core capabilities: one to lookup offers based on context, and the other to execute and apply the offer during a transaction.
We had a tight deadline. And in those moments, when timelines stare you down, you cut corners you know you shouldn’t.
We bundled both lookup and execution into a single microservice. One service. Multiple endpoints. One common backend integration. It worked flawlessly on our local machines, and even in the early stages of system testing.
So far, so good.
When the Real World Hit Us
Then came the performance tests — and like a storm hitting a poorly built tent, everything came crashing.
This wasn’t just any app. This was a component of a massive super app, and we were testing concurrency at scale — thousands of lookups per second, simultaneous execution calls, and real-world access patterns.
Here’s what went wrong:
CPU and memory spiked out — we had shared logic for DB access, and it couldn’t keep up with the number of threads.
The DB started gasping — both read-heavy lookups and write-heavy executions were hitting the same tables and same queries.
No caching layer — Every lookup was a fresh DB hit. Rookie mistake? Maybe. But when you’re rushing, even the best of us overlook the basics.
Horizontal scaling wasn’t effective — because a single service was doing too many things. The services were not independently scalable.
It was a disaster unfolding in slow motion.
Back to the Drawing Board
There was no shortcut out of this one. So we paused, took a deep breath, and went back to basics.
We split the responsibilities:
A Read-Optimized Lookup Microservice was carved out. This handled high-volume, low-latency offer lookups. We added a caching layer (more on this in another post), tuned DB queries, and optimized it for horizontal scaling.
A separate Execution-Focused Microservice was developed. It was responsible for validating and applying the offer during transaction.
Each service had its own scaling profile, its own performance strategy, and its own infrastructure config.
The results? Night and day.
The system started to breathe again. Latency dropped. Memory stabilized. We passed the load tests with confidence. The learnings though? Priceless.
Key Takeaways: Don’t Fall into This Trap
This wasn’t just a technical glitch — it was an architectural lesson. Here are the core lessons that I hope every developer, tech lead, and architect carries forward:
Design for Load, Not Just for Features : Don’t stop at functionality. Understand the nature of the endpoints — read-heavy vs. write-heavy — and design accordingly.
Granularity of Microservices Matters: A single microservice doing too much will choke under scale. Isolate responsibilities — even if it means spinning up more services.
Plan for Scale from Day 1 : Especially if your service is part of a high-traffic system. It’s easy to say “we’ll optimize later.” But sometimes, there’s no time for that.
Use Caching for High-Frequency Reads : Lookup services are perfect candidates for caching. DBs are not meant to handle thousands of reads per second without help.
Separate DB Access Paths Where Possible : When reads and writes contend for the same resources, you’ll hit a bottleneck fast. Optimize queries. Use replicas. Segment access
Test with Real-World Patterns, Early : Don’t wait till the end to simulate production traffic. Build it into your CI/CD pipeline. Be ruthless with your tests.
Final Thoughts
Every architecture decision is a bet. Sometimes we win, sometimes we learn. In our case, we learned — and then rebuilt.
So if you’re starting out with microservices, ask yourself not just what the service does, but how often, how intensely, and how independently each responsibility behaves under load.
That clarity, I can promise you, is worth its weight in gold.