When I started working on the Quant Stack at 108 Capital, one of the first things I built was a small data collection service in Python.
The system was part of a larger effort to build a research platform that could acquire, process, and analyze information from multiple external sources. Initially, we were still testing whether those datasets would be useful, so there wasn't much reason to build anything particularly elaborate.
A Python application, a few scheduled jobs, and a database were enough to get us started.
But as the project grew, the collector began taking on responsibilities that didn't fit particularly well together.
It needed to communicate with external services, manage authentication resources, coordinate concurrent requests, detect failures, and sometimes run recovery operations that couldn't finish within a normal request.
We eventually split it into several services, primarily written in Go, with Python retained for browser automation.
That solved some problems, but introduced another one.
How should all these services communicate?
The problem with having multiple services
Our first major separation was between the code that communicated with external data sources and the code that coordinated access to them.
The integration service was stateless. It handled external requests, interpreted responses, and provided a normalized interface.
The coordination service managed the resources needed to execute those requests. This included resource selection, rotation, distributed locking, and handling resources that became temporarily unavailable.
We also had a Python service responsible for recovery operations that required browser automation.
These were reasonable boundaries. Each service had a different responsibility, and the implementation could evolve without requiring changes everywhere else.
But they didn't all communicate in the same way.
Sometimes the coordination service needed to call the integration service and immediately receive a response.
Other times, it needed to initiate background work without waiting for completion.
And some operations could take considerably longer than a normal API request. Recovery was one example. The service initiating the operation needed to continue processing other requests while the recovery work happened independently.
We needed three basic communication patterns:
1. Synchronous request/reply between services.
2. Asynchronous event delivery.
3. Durable background work that could be processed independently of the caller.
We could have implemented these separately using HTTP or gRPC, a message broker, and a task queue.
But I wasn't particularly interested in operating three communication systems if one could reasonably handle the job.
Why we chose NATS
We evaluated the communication requirements and eventually chose NATS.
The main reason was simplicity.
Core NATS supported request/reply and publish/subscribe messaging. JetStream added persisted streams and durable consumption, which we could use for background workloads.
That covered the communication patterns we needed without introducing an additional broker or task queue.
It also fit the services we were writing. Both Go and Python had NATS clients, so the communication layer didn't force us to standardize on a single programming language.
We could keep the main services in Go and retain Python for the automation workloads where its ecosystem was more useful.
The resulting architecture was fairly straightforward.
NATS became the common internal communication layer, while the services retained responsibility for their own business logic.
It wasn't a replacement for every protocol in the system. We still used REST and gRPC where they made sense.
The point was to avoid building and maintaining separate communication infrastructure for each internal workflow.
Request/reply for callable operations
One of the most useful NATS features for us was request/reply.
The coordination service needed to invoke operations on the stateless integration service and receive the results. These were ordinary callable operations from the application's perspective.
NATS request/reply allowed us to implement them through the messaging layer rather than introducing another direct service-to-service HTTP dependency.
The caller sent a message to a subject, and the receiving service processed the request and published its response to the reply subject.
From the caller's perspective, it still behaved like a remote operation.
This was particularly convenient because the integration service was stateless. We could run multiple instances without requiring the coordination service to know which particular instance should process a request.
Core NATS queue groups allowed subscribers to share work on a subject, with one member of the group receiving a given message.
There were limits to this model, of course.
A request/reply call still depended on a responding service. We needed appropriate timeouts and error handling. And Core NATS didn't persist messages simply because we used request/reply.
For operations that required durable processing, we used a different pattern.
JetStream for background work
Not every operation could, or should, finish within a request.
Consider resource recovery.
The coordination service might discover that one of its authentication resources had become unavailable. Recovering it could require a separate workflow involving browser automation.
Waiting for that workflow to finish would tie up the original request unnecessarily.
Instead, the coordination service could submit the work asynchronously.
JetStream gave us persistent messaging and durable consumers for these jobs. Recovery workers could consume the requests independently, execute their tasks, and report the outcomes.
The coordination service remained responsible for the resource state. The recovery service was responsible only for carrying out the recovery operation.
This separated the lifetime of the recovery workflow from the request that initiated it.
It also meant that recovery workers could be unavailable temporarily without requiring the coordination service to wait for them.
However, persistent messaging didn't eliminate the need to reason about failures.
JetStream consumers can receive messages again if processing isn't acknowledged as expected. Applications therefore need to consider duplicate delivery and make sure repeated processing doesn't corrupt state.
For our architecture, that responsibility belonged to the services consuming the messages, rather than to NATS alone.
Another important distinction was that the message broker wasn't the source of truth for resource state. PostgreSQL and Redis continued to serve their respective persistence and coordination roles.
NATS moved work between services. It didn't replace the databases or distributed locks.
Keeping service ownership separate
One decision I think helped the architecture was giving each service ownership of a specific part of the system.
The integration service didn't manage authentication resources.
The recovery workers didn't decide which resources were eligible for ordinary operations.
The coordination service didn't implement browser automation or external response normalization.
This mattered once we introduced asynchronous communication.
It's easy for a messaging system to become a place where services publish arbitrary events and other services modify state in response, eventually making it difficult to determine which component actually owns what.
We tried to keep those responsibilities clear.
For example, a recovery worker could report that an operation succeeded or failed, but the coordination service remained responsible for updating the account or resource lifecycle.
The messages represented requests and outcomes. They didn't transfer ownership of the underlying state.
Adding more services
One of the more useful consequences of choosing NATS became apparent when we needed to add another service.
Our data acquisition platform eventually needed to manage collections of entities that could change dynamically. These collections had provider-specific limits, ownership requirements, and background maintenance work.
We decided to move that functionality into a dedicated management service.
It exposed an API for updating collection requirements, persisted the desired configuration in PostgreSQL, and used background workers to reconcile that configuration with the external provider.
It also needed to call the existing coordination service to perform operations.
By then, the communication infrastructure was already in place.
We could introduce the new service, give it the appropriate messaging responsibilities, and reuse the existing request/reply and background processing patterns.
That didn't mean adding a microservice was literally free. It still needed deployment, configuration, monitoring, failure handling, and maintenance.
But we didn't have to introduce a new communication system to support it.
This was probably where I appreciated the decision to use NATS the most.
Why not use NATS for everything?
Once you have a messaging system that supports several patterns, it's tempting to use it for every interaction.
I don't think that's necessarily a good idea.
Request/reply over NATS is still a remote call. It can time out, fail, or return an error. Moving it onto a message broker doesn't make it asynchronous in the architectural sense if the caller still waits for the result.
Likewise, using JetStream for a job doesn't automatically make that job safe to retry.
You still need to think about idempotency, duplicate delivery, acknowledgments, partial completion, and how the service recovers after a failure.
We also continued using Redis for caching and distributed locking, and PostgreSQL for persistent state.
NATS solved the communication problem. It wasn't intended to solve every distributed systems problem we encountered.
Keeping that distinction clear helped prevent the messaging layer from taking on responsibilities that belonged elsewhere.
Looking back
The original Python collector was sufficient for what we needed at the beginning.
As the system grew, we separated the external integration, resource coordination, and recovery operations because they had different responsibilities and failure modes.
NATS made that separation easier to manage.
We could use request/reply for ordinary service calls, publish/subscribe for asynchronous communication, and JetStream for durable background work, all without maintaining a collection of unrelated messaging products.
I would choose NATS again for a system with similar requirements.
Not because every distributed application needs a message broker, or because microservices are inherently better than a monolith.
But once we had a real need for independent services with different communication patterns, NATS gave us a relatively simple way to connect them.
And simplicity was exactly what we were looking for.