Token Inflation is Real
How Zalando built a client-side load balancing, Token inflation and when event sourcing is not a good choice
Client-Side Load Balancing at a Million Requests Per Second
Load balancing is a standard approach used for both high-availability and scalability, allowing the traffic to be distributed among a fleet of services (application/persistence).
The “traditional” approach is to implement this at the serving side, with some sort of infrastructure receiving all the requests and forwarding to the available servers based on some logic (load, round robin, etc).
It is interesting to see when (and why) we would need to use a different approach, and that is what Zalando did and described here.
Their product read API serves millions of requests per second and has a low latency budget. It is used both by customers and internal services via the same ingress load balancer.
It uses a consistent-hashing mechanism so the same product IDs are routed to the same pods to leverage pod-local caches.
The motivation was two-fold:
Harder troubleshooting - is the problem at the API or at the load balancer
Higher coupling - any issues with the load balancer directly affected the API, especially the batch endpoint
To understand the batch fan-out issue, we can see the following situation
A single batch call triggers N “get product” calls that need to be routed/served by the same ingress load balancer. This adds to the latency and the dependency, as if the load balancer slightly misbehaves, the entire batch will have to wait for the slowest of the requests.
The solution? Introduce a CSLB (Client-side load balancing) and achieve the following behavior
One interesting constraint of developing this situation was that they needed to achieve the same consistent hashing as the load balancer; Otherwise, they would route the same product ID request to a different pod, with consequences on the latency (and cost).
They implemented the same algorithm and tied both implementations via a series of tests that confirm that the same endpoint generates the same hash rings. Those tests are run at every build to catch any drift from being missed.
Next challenge? K8S pod discovery!
For the load balancer to work, it needs to know which pods exist, and the initial approach is the same I see abused elsewhere: polling. While simple to implement, the problem is that it has the hidden potential of DoSing your own service as it scales out.
In their case, the reactive approach was used, tapping into a stream that informs changes in real time.
The result was not only a reduction of the latency and more visibility on the causes of issues, but also a more stable latency and reduced costs at the LB (as fewer instances were needed).
The takeaways for me (and maybe for you as well) after going through the article:
Fan-out is a real issue and easy to miss during development
Owned observability exposes what shared infrastructure hides
Deployment velocity is key for safe experimentation
Token Economy
We have been used to a trend where the same thing costs less and less over time, and AI was supposed to be the same, right?
For a while, every new model was presented as more capable at the same, or reduced cost. This aligned with our Moore’s Law expectations.
In recent months, I have noticed a change in this behavior, where at least Anthropic models are also using a new tokenizer that generates up to 30% more tokens when compared to previous versions.
This makes the token maxxing approach even more dangerous than before.
Event Sourcing Is Not a Silver Bullet
I find event sourcing an awesome pattern that is often underutilized due to its perceived complexity. I wrote extensively about it in the past (my first article on the topic is from 2020!) and how it can be used to bring meaningful value to many scenarios.
With that being said, it is not a silver bullet, so it is equally important to recognize when it may not be the best solution for you.
Technical reasons are often mentioned, but as Oskar Dudycz pointed out here, they are not the only ones.
He listed three categories where the social aspect plays a role:
The team is doing well with the current approach.
The team could benefit from Event Sourcing, but thinks they don’t need it.
The team thinks it needs Event Sourcing, but it doesn’t have the competence and experience.
As someone responsible for making the decision, you have to recognize if any of those apply to your team and project.
Remember that it is not only about being right, but also about making it work within your context.
While categories 1 and 2 could mean bigger problems (apathy within the team, company culture), forcing event sourcing without managing how to overcome those challenges can become a fool’s errand.
The third one is tricky, as you want to keep the enthusiasm, but introducing a new way of thinking, developing, testing, and operating to a team that is not mature enough to take it is a sure recipe for problems.
If you want to know more about how to adopt the architect mindset and scale the practices within your company, check here.