We’re announcing our first open source project on github.

It’s a sharding service in the same family as Google’s Slicer. Or kind of like DataBrick’s Dicer, except it natively offers topology aware routing, exclusive grants, and other great features.

Splitter is the foundation for some of our consistent multi-regional primitives. In the upcoming months, we’ll open source other projects that are built on top of Splitter.

When we released a blog post where we discussed the architecture and philosophy behind Splitter, we got a number of requests to open source the project. So we’re excited to finally do it.

Blob storage (S3, GCS, etc) is an amazing technology that is the gold standard for large scale data storage. Modern Machine Learning is hard for us to imagine without blob storage feeding the data into GPU fleets. We are heavily reliant on blob storage.

We moved all our blob storage onto Azure Blob Storage (see first post about Cloudless Blob), and observed that our scale exceeded what Azure could provide out of the box. This post focuses on the technology that enabled us to scale past Azure’s bottlenecks.

Hitting the limits of Azure

After moving our data analytics stack to Azure it became clear that Azure Blob Storage was not nearly as elastic and auto-scaling as our previous cloud provider. We started hitting the throughput limits on data in and out of our buckets as well as limits on the number of reads and writes per second.

Our initial design

First we will take a short look at our initial design for Blob Storage on Azure. When building on Azure Blob Storage, there are inherent scaling limitations and quotas that can impact performance and growth. Here are the most important limits we had to consider:

1. Request Throughput per Storage Account

• Default QPS (Requests per Second): between 20,000 and 40,000 requests/sec depending on the region

2. Bandwidth Limits per Storage Account

• Maximum Ingress (Data In to Blob): 60 Gbps (~7.5 GB/sec)

• Maximum Egress (Data Out of Blob): 200 Gbps (~25 GB/sec)

3. Storage Capacity per Storage Account

• Maximum Data Stored: 5 PiB (Pebibytes)

4. Number of Storage Accounts

• Maximum Storage Accounts per Subscription: 200 accounts

As many of the limits are on Storage Accounts, we initially provisioned a handful of those, and spread the buckets we were migrating across them as fairly as possible. But, we started experiencing noisy neighbours on our Blob Storage Accounts. One example of this was an incident where writes and reads on a single Iceberg table saturated the limits for an entire Storage Account.

Analytics batch jobs broke our frontends

A large SQL job in Trino, loading many different parquet files from an analytics bucket would use all our bandwidth, and Azure would throttle all buckets on the same Storage Account. Resulting in frontends that did not load as the requests for HTML and JavaScript files were getting dropped by Azure.

Unreliable Storage Accounts

In general Azure Blob Storage SLAs looks like this

Read requests SLA

99.9%: For Locally-Redundant Storage (LRS), Zone-Redundant Storage (ZRS), and Geo-Redundant Storage (GRS), and for RA-GRS if retries aren’t used.

Write requests SLA

99.9%: For all standard redundancy options (LRS, ZRS, GRS, and RA-GRS).

This meant we often had failing requests towards Azure Blob Storage. Having to live with the risk of incurring up to 45 minutes of unavailability every month was completely unacceptable to us.

Latencies

We identified the latencies for the ListObjects operation as a significant area of concern. This particular operation has notably poor performance both in average and at the 99th percentile. This mainly impacted the overall reliability and user experience of our analytics stack. The high latencies often lead to cascading issues, including timeouts, degraded user interface responsiveness.

Cost

We pay a lot for Head and List requests because we make a lot of Head and List requests, and we would like this to be cheaper!

Designing a scalable solution

Ask Azure to scale more horizontally?

The initial solution we explored to help fix scaling problems was aimed at Azure. Asking for higher limits, and trying to understand why the infrastructure did not scale the same way as we were used to and hoped for.

This attempt worked a bit, and we got a few of the limits raised temporarily. But in the end we had to accept the fact that the cloud was not magical and in fact just someone else’s computer. With the temporary bump in limits to stop the bleeding we started looking at which possibilities we could find to solve our scaling challenges ourselves.

Just use less blob storage?

Another straightforward solution was to just use less blob storage. Ask our stakeholders to do less requests per second, store less data, and read/write fewer bytes. This solution is clearly the cheapest one, less usage of blob storage means less money spent on blob storage. At least as long as no one decides to store the data anywhere else instead. Unfortunately, during high rate limit periods we saw multiple proposals to move use cases from the cheap blob storage to much more expensive tech, such as SSDs, Redis Key-Value databases, or CockroachDB clusters.

Create a new scalable infrastructure on top of Azure Blob Storage?

We could not really provision a single Storage Account per bucket, as we have many more buckets that the Azure limit of 200 Storage Accounts allows. Even if we could just allocate a new Storage Account per bucket, this would not solve the throughput or storage limit problems for our analytics buckets. It would also not help with the low SLA on Storage Accounts.

Implement a logical sharding of our buckets

We ended up proposing a transparent sharding layer on top of Azure Blob Storage. Spreading blobs from different buckets across many Storage Accounts. This approach would:

• Remove the QPS limit for a single Storage Account (20k-40K requests/sec) by sharding requests across multiple Storage Accounts

• Remove the throughput limit for a single Storage Account ( 60-200 Gbps) by sharding reads and writes across multiple Storage Accounts

• Remove the size limit for a single Storage Account (5 PiB) by sharding data across multiple Storage Accounts

This approach included a new inhouse Metadata Layer to keep track of which blobs were stored on which Azure Storage Accounts. Using this Metadata Layer resulted in:

• Being able to respond to HeadObject and ListObject requests without having to query Azure Blob.

  • Making both request types much cheaper.

  • Making ListObjects much faster.

• Being able to store multiple replicas of the same blob, consistently, on multiple Storage Accounts, in different regions.

  • Removing the reliability problems we saw with single Storage Accounts.

As crazy as it sounds, we propose to implement a more scalable blob storage on top of a less scalable blob storage.

Architecture and Metadata

To address the scaling and cost issues, we added the option for any bucket to use a simple metadata storage layer. This layer keeps track of which blobs are stored on which Storage Accounts and what the current version, size, eTag, and custom metadata is. Whenever someone downloads an object or lists blobs in the bucket, the Bucket-Gateway checks the metadata database first. If possible, it answers right away - without needing to call Azure at all for HeadObject and ListObject operations, as well as for non-existing blobs. This is a huge improvement, since these types of requests are frequent and expensive.

For storing data, we shard blobs across multiple Storage Accounts—spreading the load and sidestepping Azure’s limits on requests, capacity, and bandwidth. The Metadata Layer records these assignments, which makes it easy to add new Storage Accounts when existing Storage Accounts are getting close to the quotas, and helps minimize disruption when any given account is slow or unavailable.

Read flow

When reading blobs from buckets that are using the sharding Metadata Layer the Bucket Gateway follows the following steps:

1. Authorizes the request

   1. Can the given user read the requested blob in the bucket?

2. Looks up where the blob lives

   1. The metadata for any given blob is stored in the Metadata Storage Layer.

   2. The actual blobs may be stored in any region, and Storage Account.

3. If the blob is found in the Metadata Layer, the Bucket Gateway reads the actual blob content from the identified Storage Account and versioned blob name.

Write Flow

The write flow is a bit more involved. The process detailed below will ensure that the blobs are written atomically.

1. Authorizes the request

   1. Can the given user write the requested blob in the bucket?

2. Validate the request is valid with the current metadata.

   1. E.g the If-None-Match header, that requires that the blob does not already exist.

3. Use the Placement Algorithm detailed below to find the target Storage Account Shard for the new/updated blob content

4. Upload the content under a new unique versioned name in the shard.

   1. If the upload fails, due to a problem with the Storage Account (rate limiting, throughput limiting, unavailability etc), use the Placement Algorithm again to choose an alternative Storage Account and retry.

5. Commits the new version of the blob to the Metadata Layer with the new backing location, size, Custom S3 Metadata etc.

   1. If any step fails, we can safely return an error to the client without risking any externally visible inconsistency.

   2. Any uploaded content that fails to be added to the metadata store will be cleaned up asynchronously

6. Async cleanup of old versions

Placement algorithm

The algorithm used to assign a blob to one of the many shared Storage Accounts is fairly simple.

1. From all Storage Accounts. filter out all that are:

   1. Not marked as electable for new blobs (accounts that are too close to different limits)

   2. Marked by the circuit breaker as currently un-stable (has had multiple failed read/writes within the last 5 minutes)

      1. If all circuit breaker as open, we pick a random to try anyway

2. We are now left with all accounts that will work. We choose the destination by

   1. Picking a random account in the current region, if one exists

   2. Picking any random account

This approach ensures that we shard blobs fairly across all active Storage Accounts.

Scaling process

When a single Storage Account gets close to full utilization, it’s removed from the set of electable accounts. When the current set of electable accounts gets too small, a handful of new ones are added. The Placement Algorithm will ensure that new blobs only land on accounts with free capacity.

The results

- Speed, Money, and Fame

Sharding

Looking back at the initial four pain points the Metadata Layer has improved our blob storage on all dimensions. The Metadata Layer has helped us scale beyond Azure’s built-in limits, slash costs for common operations, and keep our system responsive for users.

Scaling and Reliability

Scaling and reliability has improved for both reads and writes.

The main problems with hot buckets hitting the Storage Accounts limits are gone because of the new sharding strategy the Bucket Gateway uses to shard buckets across many Storage Accounts. Failed writes are much less frequent due to the Bucket Gateway being able to retry writes on alternative Storage Accounts when the initial one is degraded, and failed reads are gone as Azure no longer has to throttle the Accounts.

Latencies

Allows Bucket-Gateway to answer all Head and List requests without querying Azure Blob Storage.

Cost

Keeping all metadata in-house allows Bucket-Gateway to answer all Head and List requests without querying Azure Blob Storage.

The Metadata Layer with versioned blob names also allows us to implement a consistent in-cluster cache to read from. But that’s for another blog post.

Cover photo by https://unsplash.com/@frankiefoto

When we set out to sync state across 10+ devices per restaurant without a leader, we started where most teams do: off-the-shelf conflict-free replicated data type (CRDT) libraries. They seemed to work as advertised, but then we measured performance on the low-end Android tablets our customers actually use and we were shocked: database operations were too slow, memory overhead was too high, and device performance tanked.

So we went back to first principles and built something different. By rethinking how version metadata relates to business data, we achieved a 90% reduction in database latency and 4x less memory overhead compared to the container-based approach that dominates the CRDT landscape.

This post walks through the CRDT architecture that most off-the-shelf libraries use, where it breaks down for structured data, and the insight that lets us dramatically outperform it. If you’re syncing protobuf messages between distributed nodes, this approach may work for you too.

Why CRDTs?

A burger joint cannot stop selling burgers just because the internet goes down. The countertop point-of-sale, the kitchen display system, the self-ordering kiosk at the front–all of these devices need to be in sync, regardless of network connectivity. Requiring backend server coordination is not an option. We evaluated other approaches:

On-prem server: Additional hardware at each location creates a single point of failure and operational complexity that doesn’t scale.

Primary device coordination: Restaurant environments are hostile: devices drop from WiFi constantly, tablets overheat, someone spills liquid near a power supply. When your coordinator goes down during the dinner rush, everything stops.

Consensus algorithms (Raft, Paxos): Many restaurants run 2-3 devices. Two devices are clearly unworkable for quorum-based algorithms.

With CRDTs, we remove the need for coordination by embracing concurrent updates across replicas, leaning on an algorithm that deterministically resolves inconsistencies. Devices can update their local state independently, regardless of connectivity status, and then synchronize their data when they reconnect. There’s no single source of truth, no single point of failure, and data in the mesh is guaranteed to eventually converge.

The Standard Approach: Container-Based CRDTs

Most CRDT libraries follow a common architectural pattern: wrap every value in a container that implements a commutative, associative, and idempotent “merge” function.

For “last-write-wins” (LWW) data types, this means wrapping both the data and its version metadata in each container. This is the approach used by Ditto, Automerge, Y-CRDT, and most open-source implementations.

// The standard approach: wrap every value in a ‘mergeable’ data type
interface Mergeable {
    /** Deterministically merges a remote state with this one. */
    fun merge(remote: Mergeable): MergeResult
}

enum class MergeResult {
    OURS, // Our data was unchanged (all remote data was ignored)
    THEIRS, // The new data is now equal to the remote data
    NEW, // The new data is not equal to ours or remote.
}

class LWWRegister<T>(
    val value: T,
    val version: Long
) : Mergeable

class LWWMap<K, V>(
    val entries: Map<K, LWWRegister<V>>
): MutableMap<K, V>, Mergeable

// Your data becomes deeply nested containers
class Order(
    val customerId: LWWRegister<String>,
    val status: LWWRegister<OrderStatus>,
    val items: LWWMap<String, OrderItem>,
    // ... every field wrapped
) : Mergeable

This pattern is conceptually elegant, each value carries its own merge logic. But elegance doesn’t always equate to performance.

The Hidden Costs

When we deployed a container-based CRDT solution in production, the costs became clear.

Memory explosion: A 1KB protobuf blob became 4-5KB with containers. Each field needed a wrapper with type metadata, value container, and version info, even if they were identical across multiple fields. Average orders were 4-5KB, but complex orders reached 100KB before applying CRDT metadata leading to memory spikes on low-end devices.

Mapping overhead: For type safety and interoperability with other systems, our data models are defined using Protocol Buffers. This allows us to interact with the data in a natural way, provides clear schema update rules, and enables the reuse of several pieces of infrastructure already available around this technology.

However, this choice also means that we had to implement adapters to map domain object fields into generic container types. As a result, every read and write operation to the database required an O(F x D) transformation (where F is the number of fields and D is the depth).

Version redundancy: When an actor updates multiple related fields, they share the same version. Yet container-based systems store identical version info for each field.

For applications syncing small documents or text, these costs are acceptable. But our devices aren’t just syncing—they’re simultaneously generating bitmaps for receipt printing, fetching menus, processing payments, and rendering content across multiple screens. The CRDT layer competes for limited CPU and memory with everything else.

Our Approach: Separate Version from Data

Traditional CRDT libraries embed versions within values. Our key insight: separate them instead. Maintain a parallel version tree that mirrors your data structure. Remove redundant nodes in the version tree by only storing entries for fields that differ from the base version.

For example, in Figure 4 all fields were set at once, so they inherit the root version. No per-field version storage needed.

When a device updates only the status field, then only that field gets a version entry. The other fields continue inheriting from the base version. This gives us O(m) space complexity where m = modified fields, instead of O(F) for all fields.

The data layer remains a clean protobuf domain object, free of any version semantics. The parallel version layer mirrors the data layer, tracking field-level modifications in a sparse tree. Fields without explicit entries inherit the base version from their parent node, providing massive memory savings for typical usage patterns like data that has many fields but only a few change over time.

Complexity Comparison

With the container approach you must wrap each field of your data type in the CRDT container wrappers when you merge data types or read the value of a given field, incurring an operational complexity proportional to the number of fields times the depth of your structure.

In our approach, because version metadata is sparse and stored separately from the business data, reads are constant and merging is proportional only to the number of modified fields.

How It Works

Let’s walk through concrete examples using a restaurant order schema. First, the protobuf definition with CRDT options:

import "com/css/protobuf/crdt/data/options/message_options.proto";
import "com/css/protobuf/crdt/data/options/field_options.proto";

message Order {
    string id = 1;
    string customer_name = 2;
    OrderStatus status = 3;

    // Nested message: field-level merge by default
    PaymentInfo payment = 4;

    // Nested message with atomic replacement
    Receipt receipt = 5 [
        (com.css.protobuf.crdt.data.options.crdt_merge_strategy) = REPLACE
    ];

    // Map: per-key versioning with tombstone TTL
    map<string, string> metadata = 6 [
	(com.css.protobuf.crdt.data.options.crdt_tombstone_ttl) = 3600
    ];

    // Repeated with ID field: element-level merge
    repeated LineItem items = 7 [
        (com.css.protobuf.crdt.data.options.crdt_id_field) = 1
    ];

    // Counter: concurrent increments merge correctly
    int64 modification_count = 8 [
        (com.css.protobuf.crdt.data.options.crdt_merge_strategy) = COUNTER
    ];
}

Last-write-wins doesn’t work for every data type. We still need to support custom per-field merge strategies. We leaned into a schema-driven approach using protobuf options. We’ll go through the default merge case, as well as some of the custom scenarios below.

Scenario 1: Basic Field Merge

Assume we have two devices making concurrent modifications to different fields of the order protobuf. Device A updates the customer field, while device B updates the status field.

Both devices apply a local write to the order. When a device receives the other’s update a commutative merge is performed leveraging both version trees to reach a last-write-wins merge result.

// Device A: Update customer name
val orderA = existingOrder.copy(customer_name = "Jane Doe")
val deltaA = orderResolver.applyLocalWrite(/* ... */, timestamp = 1000)

// Device B: Update status (at the same time, different field)
val orderB = existingOrder.copy(status = OrderStatus.CONFIRMED)
val deltaB = orderResolver.applyLocalWrite(/* ... */, timestamp = 1001)

// When B receives A's update, both changes merge using the version trees
val merged = orderResolver.resolveIncoming(
    deviceB.value,
    deviceB.versionTree,
    deviceA.value,
    deviceA.versionTree
)

// Result: customer_name="Jane Doe", status=CONFIRMED
// Both updates preserved - no data loss

Scenario 2: Map Field with Concurrent Updates

Map fields require correct handling of deleted elements. Assume two devices update different keys in a map field, where one device also deletes a key.

// Device A: Add a metadata entry
val orderA = existingOrder.copy(
    metadata = existingOrder.metadata + ("source" to "mobile_app")
)

// Device B: Add different entry and delete another
val orderB = existingOrder.copy(
    metadata = (existingOrder.metadata - "old_key") + ("priority" to "high")
)

// After merge: all additions preserved, deletion applied
// metadata = { "source": "mobile_app", "priority": "high", ... }
// "old_key" removed (tombstoned)
val merged = orderResolver.resolveIncoming(/* ... */)

Deleted entries persist in the version tree as tombstones allowing the resolvers to merge map changes correctly across devices. The crdt_tombstone_ttl option allows you to define a time-to-live (TTL) for tombstone entries to bound memory growth.

Scenario 3: Repeated Field with ID-Based Merge

Lists present a unique challenge during merge resolution given the sorting of elements could change across devices, even if the data is the same. Assume two devices modify different items in the order’s line items list.

// Existing order has items: [{ id: "item-1", quantity: 1 }, { id: "item-2", quantity: 2 }]

// Device A: Update quantity on item-1
val itemsA = existingOrder.items.map { item ->
    if (item.id == "item-1") item.copy(quantity = 3) else item
}
val orderA = existingOrder.copy(items = itemsA)

// Device B: Update price on item-2
val itemsB = existingOrder.items.map { item ->
    if (item.id == "item-2") item.copy(price_cents = 999) else item
}
val orderB = existingOrder.copy(items = itemsB)

// After merge: both item updates preserved
// items = [{ id: "item-1", quantity: 3 }, { id: "item-2", quantity: 2, price_cents: 999 }]
val merged = orderResolver.resolveIncoming(/* ... */)

The crdt_id_field option tells the resolver which field identifies each element. Without it, the entire list would need to be replaced atomically (i.e. last-write-wins on the entire list).

Scenario 4: Counter Field

When an integer or long field is marked with the COUNTER merging strategy, a G-Counter structure is stored in the version tree. Counter fields use per-actor tracking internally so that each device’s contribution is recorded separately. The counter value is the sum of all contributions, ensuring concurrent increments never lose updates, whereas a naive last-write-wins approach would lose all but one of them.

If two devices increment a counter concurrently, both increments are preserved.

// Both devices start with modification_count = 5

// Device A: Increment by 1
val orderA = existingOrder.copy(modification_count = 6) // 5 + 1
val deltaA = orderResolver.applyLocalWrite(/* ... */)

// Device B: Increment by 2
val orderB = existingOrder.copy(modification_count = 7) // 5 + 2
val deltaB = orderResolver.applyLocalWrite(/* ... */)

// After merge: modification_count = 8 (5 + 1 + 2)
// Counter tracks per-actor contributions and sums them

Scenario 5: Atomic Replacement vs Field Merge

Different fields might require different merge strategies. For example, you may want to REPLACE on updates to the receipt field, but MERGE on updates to the payment field.

// Device A updates payment method
val orderA = existingOrder.copy(
    payment = existingOrder.payment.copy(method = "credit_card")
)

// Device B updates payment amount
val orderB = existingOrder.copy(
    payment = existingOrder.payment.copy(amount_cents = 5000)
)

// After merge: both payment fields preserved
// payment = { method: "credit_card", amount_cents: 5000, ... }

// But for receipt (REPLACE strategy):
// Device A generates new receipt
val orderA = existingOrder.copy(
    receipt = Receipt(pdf_data = newPdfA, generated_at = "10:00")
)

// Device B also generates receipt
val orderB = existingOrder.copy(
    receipt = Receipt(pdf_data = newPdfB, generated_at = "10:01")
)

// After merge: Device B's receipt wins entirely (higher timestamp)
// No partial merge of receipt fields—it's atomic

Use REPLACE for fields where partial merges don’t make sense: binary data, generated content, or tightly coupled field groups.

Results

Our approach is significantly more efficient and adds only minimal memory overhead to your existing data.

Why do we win? With a parallel version tree divorced from business data, we eliminate per-field CRDT container wrappers which lead to O(F x D) complexity during read and write operations. By only storing modified fields in our version tree, memory overhead plummets and we can achieve a massive reduction in DB operation latency.

But dropping the containers doesn’t come for free. You still need to support custom per-field merge strategies. We leaned into a schema-driven approach using protobuf options: MERGE for field-level resolution, REPLACE for atomic updates, COUNTER for commutative operations, and ID-based lists for element-level tracking.

Takeaways

Separating versions from data is less natural and requires additional information to be stored. Since the additional information was small in practice, our approach reduced CRDT memory usage by 4-5x when combined with various optimizations.

Our approach works for any application syncing structured protobuf data across distributed nodes. We use it in production and plan on scaling it to thousands of restaurants, allowing Otter POS customers to handle network partitions and concurrent updates seamlessly on low-end Android hardware.

If you know an even better approach, let us know.

Written by Steponas Dauginis and Sean Chen

We see more code being written by GenAI over time. Due to careful human inspection, we aren’t yet seeing a regression in the quality of code (link). But if trends continue, companies will want solid and lighterweight mechanisms in place to prevent bad code from shipping to customers. Robust canarying is one such mechanism.

In a canary deployment, new software is rolled out to a subset of traffic before wider distribution. If problems are detected, the software can be rolled back without impacting everyone.

At CloudKitchens, over 80% of all our service releases are conducted via canary, and over 95% of releases for services in the critical path (order fulfillment) leverage canary. In Q3 2025, canary blocked more than 1100 bad releases across the company, many of which could have resulted in user-facing regressions, if not downright outages.

If you search for blog articles on this topic, you will find writeups describing an idealized version of it. Many of these posts are penned by companies advertising their canary solution, often as part of a broader CI/CD offering.

Here we describe how canary adoption and usage evolved in practice at our company, including early missteps and later improvements.

We are not trying to sell you anything; instead, we will share insights into the essential properties of a canary system that you will want, regardless of whether you buy a solution or build your own (as we did!).

High-Stakes Deployments

At CloudKitchens, processing real-time food orders is at the heart of our business.

In the beginning, all order fulfillment ran in a single region on our cloud provider. When we deployed a new version of our service, the new software quickly handled 100% of customer traffic. If a bug was introduced, engineers would be paged to manually identify the faulty deployment and then roll it back – a process that could last over an hour. In fact, well over 75% of our outages were triggered by bad deployments. To make matters worse, not all problematic changes were even caught by our alerts in the first place.

To make ourselves resilient to datacenter outages, we began operating critical applications and supporting infrastructure across three separate regions. Conveniently, our deployment tools added a 60-minute delay between regional deployments. This delay allowed engineers to halt the deployment if issues were detected in the first region, effectively making it a deployment where the canary population was a whopping 33% of traffic (if problems were detected in time). Still, many bad deployments slipped through undetected until they reached region #3.

Canary Basics

In summary, we faced two closely related challenges:

• Knowing if something is broken.

• Rolling back a deployment once there is enough signal that it is causing the breakage.

Knowing Something Is Wrong

Even before we can make decisions at deployment time about whether a new software version is healthy to roll out fully, we need to answer: “how do we know something is broken?”.

When engineers are required to manually create their own dashboards and alerts, this not only requires an excessive amount of their time, but it also leads to inconsistent monitoring across the company.

Therefore, even before we invested in deployment automation, we built observability automation that:

1. Periodically scans our Kubernetes clusters to discover running services.

2. Associates each running service with Prometheus metrics that match labels derived from the service’s unique identifier.

3. Generates dashboard panels and alert specifications from the discovered metrics and pushes them to Grafana.

An over-simplified data flow

The importance of standardizing on middleware and client libraries that emit consistent metrics cannot be overstated. As a concrete example, a Java service that serves inbound GRPC traffic, reads from CockroachDB, and writes to Kafka would receive a tailored dashboard soon after its initial deployment.

Some of its critical alerts would cover:

• Container health: CPU request utilization, container memory limits, excessive container restarts. We leveraged kube-state-metrics.

• Application runtime health: JVM heap utilization, threadpool queue size, and GC pauses.

• RPCs: The availability and latency of each exposed GRPC endpoint. Users can also enable alerts on egress failures if appropriate.

• Database client performance: SQL statement success rate and latency.

• Kafka client performance: Enqueue success rate and latency.

• Excessive volume of error-severity logs.

• . . .

This automation meant that when an engineer deployed a service, they automatically received a standardized dashboard featuring highly relevant alerts – with no upfront payment on manual configuration. Updating a service – for example, adding a new RPC – would automatically result in new panels and alerts, requiring no continual maintenance either.

Rolling Forward Or Rolling Back

Having the required metrics in-place for each service, we can now stand up a rudimentary canary mechanism as follows.

For each service, we separate its instances into three groups: baseline, canary, and main. We then manipulate these groups during the following phases of our deployment:

• Pre-Deploy: Only the main partition has running instances – this is steady-state behavior.

• Create New Groups: We spin up additional canary instances running the new code and additional baseline instances running the previous code. The instances in these groups handle live traffic as soon as they are created. If there are 3 main instances at steady state, and we add 1 baseline instance and 1 canary instance during the deployment, then we are rolling out new software to 20% of traffic, assuming a round-robin load-balancing policy for our services.

• Analyze Metrics: We compare the behavior of the baseline and canary partitions by checking the metrics we already configured for this service, pitting the matching metric from each partition against one another. As its name suggests, we use the baseline partition as the basis for comparison (rather than the main partition) so that we can monitor startup issues – such as unusual memory utilization or caching patterns.

• Make The Call: The system decides whether to roll forward (promote all main partition instances to the new version) or roll back based on failing checks. Roll forward decisions wait for a longer analysis window. But we roll back more quickly at the onset of a failing check.

Roll forward to:

or roll back to:

We use the Mann-Whitney U test to determine whether metric fluctuations are significant or simply random noise. While we considered t-test and distance functions, this method produced the fewest false positives in our evaluation dataset.

💡 Takeaway: To bootstrap our canary deployment mechanism, we leveraged the same metrics that engineers already relied on to monitor their services in steady-state conditions.

Initial Results

This first pass was effective at catching the most glaring issues – the two most frequent classes of issue were degraded availability or latency for a modified RPC; and crashes / out-of-memory errors.

Though still primitive, adopting this approach for critical services across the company prevented several dozen production bugs each month. However, numerous gaps remained.

Towards Maturity

Below are the shortcomings of our original solution, and how we addressed each of them over a span of several months (and thousands of deployments). Comparing our Q3 stats against Q2, these changes enabled us to catch 82% more bad releases, while the total number of canary releases increased by only 28%.

Granularity

Consider the prior example, where a service is running on 18 instances at steady state. To achieve the smallest possible rollout, where we add 1 baseline instance and 1 canary instance during the deployment, our smallest rollout is 5%. But at our scale, this is still higher than desired, exposing more customers to new code. For example, if we process 1000 orders per minute, a bad deployment could affect up to 50 x 3 = 150 customers even if we could roll back within 3 minutes (though, we can lessen its impact via aggressive retries). Regardless, that’s far too many unhappy customers.

Even more problematic were critical services requiring very few instances. For a service running only 2 instances, the smallest rollout amounted to 25% (1 out of 4). That is not much better than the 33% from our makeshift canary that waits between regions.

To address this, we integrated our canary mechanism with our load-balancing solution. Using Istio as our service mesh, we adapted our process to create three sets of VirtualService resources (for main, baseline, and canary) to drive proportional traffic, adjusting weights gradually as we progress through the deployment.

This decouples our canary rollout granularity from the service’s horizontal scaling.

💡 Takeaway: Canary deployments must be fully integrated with the chosen load balancing solution in order to finely tune the rollout process, reducing the blast radius of potential failures.

Canarying for Indirect Failures

While guarding each service’s deployment with its own emitted metrics prevents many common issues, it doesn’t capture the whole picture.

Take for example, a bug where an important endpoint is deleted inadvertently. The canary mechanism would not have matching sets of metrics to compare endpoint performance between baseline and canary partitions; and furthermore, it cannot possibly know if the deletion was intentional or not (e.g. part of a planned migration). In practice, we have also seen outages where a backward-incompatible schema change is made, and canary instances actually exhibit lower error rates because only they can process the new schema definitions.

To address such end-to-end issues, we canary on traces across all our most important services. By tracking availability, latency, and throughput for these traces, we can monitor overall business health.

At deployment time, we compare regressions in end to end flows alongside service-specific metrics, helping us catch user-impacting outages.

💡 Takeaway: Monitoring end-to-end business operations provides a comprehensive view that safeguards against subtle yet critical issues that are not visible in service-level metrics alone.

Statistical analysis algorithm

Originally we used the Mann-Whitney U test as the single metric evaluation strategy. Later on, we did a collaboration with our data scientists to introduce the Proportional Check algorithm, which uses Fisher’s exact test under the hood. For our use-case it performs much better on small data sets and can provide meaningful insight quicker. Also, it is a more easily understood option for all checks which work on proportions (e.g. % of failure vs. total request count).

Rollback Criteria

The initial version of our canary mechanism would roll back a deployment if any metrics check failed, erring on the side of caution. However, flaky metrics led to many unnecessary rollbacks, much to our engineers’ frustration.

To better understand our canary deployments, we instrumented:

• True positive, false positive, and false negative rates — these correspond to deployments rolled back due to real bugs, deployments rolled back unnecessarily, and deployments that slipped through despite real bugs. Some randomly picked releases are labelled manually, and the rest is done by a LLM-backed classification to validate the performance and its change over time.

• Time spent in analysis in each of the above scenarios.

Our aim is to drive up true positive rates, drive down false positive and false negative rates, while minimizing analysis time (good deployments should not drag out, bad deployments should be rolled back ASAP).

Over time, we refined our default checks to optimize and above performance metrics. We also made significant investments towards our user interface.

This UI displays failing checks per release:

And it empowers users to make further adjustments on their own:

As we introduced new classes of metrics checks (e.g. error log volumes), and as users fine-tuned thresholds, it became necessary to differentiate between metrics checks that were mature from those still in early development.

Ultimately, we categorized them into:

• Critical: Immediate rollback upon failure to minimize impact.

• Warn: Pages the responsible engineer. They can then flag whether this is a real issue (prompting an immediate rollback). Well-performing checks are promoted into “Critical”.

• Debug: The check is logged (metrics collected), but no action is taken. We compare these after-the-fact to know how to modify them; or to promote them into “Warn”.

By granting users both visibility over failing checks and ultimate control if the default rollback parameters were, we drove user trust and wider adoption. At this point, nearly three-quarters of all service deployments across our company are guarded by canary.

Still to do: Implementing backtesting capabilities using historical data to experiment with new parameters without initiating real deployments.

💡 Takeaways:

• Invest in instrumentation upfront. To know where to adjust rollback criteria, it is vital to first track true positives, false positives, false negatives, and analysis time on a per metric-basis.

• Tiering enables iteration by distinguishing mature checks from those needing far more refinement before they can override a deployment.

• Canary requires a great UI. This enables engineers to gain visibility on why the system made any given decision, and is a springboard for them to make adjustments for their own services. This garners trust and adoption.

Conclusion

At this stage, we are far more confident in our ability to deploy new service code while minimizing adverse impact on our customers. There is still room for improvement. For example, we’re currently building more sophisticated rollout cohorts and tweaking AI techniques to further tune false positives and negatives.

Subscribe now

Written by Gaurav Nolkha and Tomek Sroka

When Otter first started to hyperscale, the cost of our logging stack grew out of control. Stackdriver (logging platform on GCP we used at the time) consumed 17% of our monthly cloud bill, growing 19% month-over-month. To mitigate without giving up observability we switched to self-managed OpenSearch, cutting costs by 80% but still paid way too much: $3.35 per 100GiB per month and spent significant time on cluster operations.

Even with a dedicated team optimizing OpenSearch, our clusters frequently degraded and our engineers started to lose trust in the observability platform. We were forced to eventually declare a code yellow and spent several months hardening the infrastructure, which was still expensive to run. We needed something radically different. Leveraging blob stores to provide reliable, scalable, cheap storage, we built LogProc: a logging engine optimized for low operational overhead and cost.

The results: LogProc now handles 750+ TiB of logs at 4.4x lower cost than self-hosted OpenSearch, with low operational overhead. We’re 50x cheaper than managed alternatives (like Elastic Cloud), and 70% of our engineers use it daily.

Scaling Pains: Our Journey and Hurdles with OpenSearch

OpenSearch offered powerful search capabilities, but at our scale, costs ballooned, making it one of our top 5 most expensive services.

The real killer was operational overhead. OpenSearch demanded specialized expertise just to stay running. We implemented a hot-cold architecture: SSDs for recent data, cheaper storage for archives, which reduced costs but added operational complexity and failure modes. Our team burned countless hours on cluster maintenance, index tuning, and firefighting. And we were losing.

We eventually realized that OpenSearch’s unit cost wouldn’t scale with our growth. We needed a cheaper, simpler, and more reliable alternative. Most of our challenges stemmed from managing our own storage, so we turned to blob storage, the de facto standard for large-scale data infrastructure. This became the foundation for our new logging engine.

LogProc - A New Logs Datastore

LogProc’s design centers on five core design decisions that directly address OpenSearch’s cost, reliability, and performance deficiencies.

Cost Efficiency with Scalable Storage: With 70% of our costs in storage, we moved to blob storage (Azure Blob) for sizable cost reduction. Unlike OpenSearch’s requirement for replicas, blob storage provides built-in durability, eliminating the overhead of managing redundant data copies (replicas). The trade-off: accessing blob storage is two orders of magnitude slower than fetching data from SSDs, so we must manage query performance through parallel processing and smart caching strategies. This works because most logs are written once but rarely read.

Simplified Reliability with Stateless Query Service: Logging must work when everything else is broken. We built a stateless query engine that requires zero coordination with sub-second startup times; unlike OpenSearch’s complex shard management, our nodes deploy independently. The trade-off: parallelizing and distributing queries requires thoughtful design, for example, using rendezvous hashing to maintain a high cache hit rate.

Horizontal Scalability: LogProc scales seamlessly for both ingestion and querying by simply adding more nodes, without requiring rebalancing. No manual tuning required. With OpenSearch, adding nodes triggered massive shard rebalancing with enormous east-west traffic (intra-cluster data transfer during rebalancing), while removing nodes was a manual, hours-long operation to safely drain shards before scaling down.

Effective Durability: For durability, we use RocksDB for high-performance local buffering before writing to blob storage, reducing the number of blob API calls (and creating fewer but larger blob objects) through batching. The trade-off: queries spanning the last ~15 minutes depend on the ingester; if an ingester node is down, results may be incomplete until it recovers.

Choosing Rust for Performance and Safety: Rust delivers the performance and safety critical for data-intensive workloads. Despite the learning curve, our cluster now consumes ~10x fewer resources (CPU and memory) than OpenSearch, which suffered from Java’s GC pauses and memory bloat.

The result: cheap object storage, minimal indexing, and Rust’s efficiency created the most cost-effective logging solution we could build. Today, 70% of our engineers use LogProc daily via a custom Grafana plugin.

Putting it All Together: LogProc Architecture

LogProc’s architecture separates concerns into two independent paths: a stateful ingestion pipeline that batches logs into blob storage, and a stateless query engine that retrieves them on demand.

Fig 1: LogProc architecture showing the ingestion pipeline (red), query path (green)

Before diving into how these components work, let’s establish the key concepts:

Logstream: A logical grouping of logs from the same source, identified by region, namespace, app, and container (e.g., region=centralus, namespace=orders, app=order-central, container=oc). Think of it as a labeled bucket for related logs.

Block: A unit of logs from a single logstream (up to 500MiB), uploaded to blob storage. Logs within a block are sorted by timestamp.

Chunk: A ~1MiB subdivision within a block that enables partial downloads during queries. Chunks cover non-overlapping time ranges.

Ingester: Stateful component that batches incoming logs using RocksDB, then uploads blocks to blob storage. Uses High Random Weight, a.k.a. Rendezvous hashing (HRW), a hashing variant that minimizes remaps, to distribute logstreams across instances.

Querier: Stateless component that processes queries by fetching blocks from blob storage (or cache) and merging results. Also queries Ingesters for the most recent logs.

Note on scope: This post focuses on LogProc’s core architecture: ingestion and querying. We’ve omitted two components: the Compactor (background process for merging blocks and enforcing retention) and our Grafana plugin (query interface) for brevity.

Writing Logs

The ingestion pipeline balances two competing goals: minimizing storage costs while enabling fast queries. How we partition, batch, and index logs during writes directly determines query performance. The implication is that there’s no separating storage decisions from query performance optimization.

Partitioning Strategy: Drastically Reducing Search Space

Logs are partitioned into logstreams based on source identifiers: region, namespace, app, and container. This is a fundamental improvement over our OpenSearch setup, which was partitioned only by time.

While OpenSearch’s inverted indexes made field-based searches efficient, time-only partitioning created severe hotspots. Recent shards received the vast majority of queries since users overwhelmingly search recent logs. This concentrated query load on a few indexes degraded performance for everyone, regardless of cluster capacity.

The solution: By partitioning on the app label, we can dramatically narrow the search space. When users specify app and namespace (which we encourage in the UI), we only scan logstreams matching those labels. Instead of searching through terabytes of logs from all services, we search only the relevant logstreams, reducing the data volume 10-100x (proportional to the number of applications).

Ingester Sharding: Sticky Logstreams for Efficiency

Getting logstreams to the right Ingesters is critical for efficiency. We use sticky logstreams, ensuring logs from the same logstream go to the same Ingester. Although if a chosen Ingester fails to accept entries, the receiving Ingester stores them locally as a fallback, ensuring no data loss.

When an Ingester receives a batch of log entries from any source, it groups them by logstream and uses Rendezvous hashing to determine which Ingester should handle each logstream. It then forwards entries to the appropriate Ingesters via internal gRPC calls. Each Ingester maintains a live list of all active Ingesters in the cluster through service discovery.

Why Rendezvous hashing? Unlike simple hash-mod approaches that remap nearly all logstreams when scaling, Rendezvous hashing minimizes disruption: only L/N logstreams need remapping when adding or removing an Ingester (where L = number of logstreams, N = number of Ingesters).

Ensuring logs from the same logstream always go to the same Ingester, enables:

1. Builds denser blocks for better compression and fewer blob storage PUT API calls.

2. Avoids rebalancing storms by adding an ingester that remaps ≈L/N streams.

3. Simplifies failure: if an ingester is down, senders temporarily buffer or retry; once it is back up, it resumes from the local state.

Batching and Block Structure: Enabling Partial Downloads

Each Ingester handles logs from multiple logstreams simultaneously. For each logstream, it accumulates logs in RocksDB (local persistent storage) until one of two conditions is met:

1. The size threshold is greater than 500MiB of logs accumulated for that logstream.

2. Time threshold is greater than 15 minutes elapsed since the first log in that logstream.

Why batching matters for cost: Blob storage charges per PUT API call. Writing individual log lines would generate millions of expensive API calls per day. By batching into 500MiB blocks, we reduce API calls by orders of magnitude: a single PUT operation replaces tens of thousands of individual writes.

Once triggered, the Ingester creates a block for that specific logstream containing logs organized into ~1MiB chunks. Each chunk covers a non-overlapping time range and logs are chronologically sorted within chunks.

Why chunks matter for queries: When a query targets a specific time range (or uses our full text search index), the query engine can download only relevant chunks instead of the entire block. This dramatically reduces data transfer and speeds up queries.

Each block and its indexes are uploaded to blob storage, and metadata (logstream identifier, time range, block id) is stored in PostgreSQL. This means an Ingester is continuously creating and uploading blocks for different logstreams as they hit their respective thresholds.

The durability trade-off: RocksDB on persistent volumes acts as a write-ahead log (WAL), ensuring logs are durable even before blob upload. However, the most recent logs (last 15 minutes) haven’t reached blob storage yet and only exist on Ingesters.

For queries covering recent time ranges, the query engine must fetch data from both blob storage and active Ingesters. Since we don’t run Ingester replicas, if an Ingester pod goes down, real-time queries will return incomplete results for that pod’s logstreams until it recovers. When the pod restarts, RocksDB recovers the buffered logs and resumes uploading to blob storage.

In practice, this hasn’t been a concern: Ingester nodes have been stable, and temporary query incompleteness during node restarts is acceptable for our use case. If needed, we could add HA capabilities, but the current trade-off keeps operational complexity minimal.

Index Creation: Filtering Before Scanning

During block creation, the Ingester builds several indexes that allow the query engine to skip irrelevant blocks entirely:

Field Path Index: XOR filter, a compact probabilistic membership filter (similar to Bloom) for “might contain” tests, stores all field paths present in the block (e.g., payload.fields.operation). If a query searches for a field that doesn’t exist in the block, we skip it without downloading.

ID Index: An XOR-based filter, similar to a Field Path Index, helps determine if a specific block might contain a searched value. This filter uses values extracted from log entries via predefined regular expressions. It significantly accelerates “needle in a haystack” queries, such as searching for UUIDs or trace IDs, where the desired value is expected in very few locations.

Free Text Index: FST (Finite State Transducer: automaton for fast token/prefix matching) stores tokenized log content. Enables fast free-text searches by identifying which chunks within a block contain the search terms. Tokens are grouped into 2-token windows (e.g., “order not found” becomes “order not” + “not found”) to improve multi-word search accuracy.

These indexes answer the question: “Does this block contain relevant data?” before we spend time and bandwidth downloading it.

Why This Beats OpenSearch

OpenSearch also used size/time-based batching, but has critical differences with LogProc:

Replica overhead eliminated: Blob storage provides built-in durability. OpenSearch requires storing replicas of both data AND maintaining replica compute nodes: doubling storage costs and compute overhead.

Decoupled storage and compute: Our Ingesters are purely computational; they buffer and batch, but don’t store data long-term. OpenSearch nodes were tightly coupled; losing a node meant both losing compute capacity and triggering complex shard rebalancing.

Simpler failure handling: When an Ingester fails, Rendezvous hashing routes logstreams to other instances with minimal remapping. No shard rebalancing, no replica promotion, no cluster coordination. OpenSearch’s shard allocation and replica management under load often led to degraded cluster states.

The result: we achieve efficient batching and fast queries without the operational complexity and cost overhead of maintaining a replicated cluster. In our largest cluster we have about 9000 logstreams, batching results in ~1 PUT API call every 15 minutes for each logstream and indexes help us filter out ~95% blocks on average.

Querying Logs

The query engine leverages the partitioning, chunking, and indexing decisions made during ingestion to minimize data transfer and maximize cache effectiveness.

Query Flow

When a user submits a query (e.g., “show me errors from app=orders in the last 24 hours”):

1. Metadata Lookup: The lead Querier queries PostgreSQL to identify relevant blocks based on:

• Logstream filters (app, namespace, region, container)

• Time range

• Returns block metadata (location, time range, block ID)

2. Query Recent Logs: The query fans out to all Ingester pods to fetch logs from the last 15 minutes that haven’t been uploaded to blob storage yet.

3. Work Distribution: Using Rendezvous hashing, the Querier that received the request assigns blocks to other Querier instances for parallel processing. Each block’s hash determines which Querier node will handle it; the same block always routes to the same Querier to maximize cache hit ratio.

4. Block Filtering with Indexes: Before downloading any block, each Querier checks the block’s indexes (stored separately in blob storage):

• Field Path Index: Does this block contain the queried fields?

• ID Index: Does this block contain the searched ids?

• Free Text Index: Which chunks contain the search terms?

Blocks that don’t match are skipped entirely. For matching blocks, only relevant chunks are downloaded.

5. Local Processing: Each Querier:

• Downloads assigned chunks from blob storage (or retrieves from local cache)

• Scans the chunk data for matching logs

• Streams results back to the lead Querier

6. Result Merging: The lead Querier merges time-sorted streams from all worker Queriers and Ingesters, returning the final response to the user.

Caching for Performance

Frequently accessed blocks are cached locally on Querier pods. Rendezvous hashing ensures the same blocks always route to the same Queriers, dramatically improving cache hit rates. This means:

• Popular logstreams (actively debugged services) stay hot in cache

• Repeated queries return results in milliseconds instead of seconds

• Reduced blob storage egress costs

Stateless Design Benefits

Unlike OpenSearch’s stateful query coordinators, our Queriers are completely stateless:

• No cluster coordination or leader election needed

• Queriers can be added or removed instantly; just scale the deployment

• Node failures don’t require recovery; queries simply route elsewhere

• No “yellow” state degradation under load

The trade-off: performance depends on cache hit rates. Because LogProc does not build a massive reverse index, it needs to get and consult indexes for each block, download blocks, and scan them for matches. In practice, we’ve seen that indexes filter out ~95% of blocks. In addition, parallelizing work and caching also significantly reduces query times even for searching across all logstreams. We keep index caches warm for the last 7 days, maintaining an almost 99% cache hit ratio for queries for that period.

The result: simple query engine that achieved better performance compared to our OpenSearch clusters. For our biggest cluster, P50 is ~200ms and P95 is ~6s.

Conclusion

Building LogProc required challenging fundamental assumptions about logging infrastructure. By separating compute from storage and designing around blob storage’s limitations, we achieved both dramatic cost reduction and operational simplicity.

The key architectural decisions that made this possible: app-based partitioning to reduce search space, sticky logstreams to minimize API calls, aggressive batching to optimize blob storage costs, strategic indexing to skip irrelevant data, and stateless query engines for trivial scaling.

Today, LogProc handles 750 TiB of logs at $0.75 per 100GiB; 4.4x cheaper than our self-managed OpenSearch and 50x cheaper than managed alternatives. More importantly, it requires virtually zero operational overhead. No shard rebalancing, no replica management, no cluster coordination, no degraded states.

The trade-off: We lost OpenSearch’s semantic search capabilities. For our use case of debugging production issues, this was acceptable. The cost savings and elimination of operational headaches far outweighed losing a feature we rarely used.

What’s Next: Open Source LogProc

We’re preparing to open source LogProc in the coming months. As we finalize licensing, packaging, and documentation, we’d love your input, particularly from teams running 100+ TiB of logs. Reach out to help shape the release.

Written by Vishruth Ashok (Dev Platform) and Brian Attwell. Input from Alex Filipchik, Michael Li, and many others

At Atoms, we’ve been proactively investing in ways to accelerate our productivity and improve our Developer Experience (DevEx) with internal and externally built GenAI tools (section: Comparison of GenAI Vendors), even though it is clear that many of the industry opinions on this tech are overblown hype (section: Marketing Hype).

Our internal studies have shown that coding assistants have had persistent categorical limitations over time. Yet subjectively, their usability and effectiveness is increasing. Engineers with high baseline productivity report a sustained median weekly savings of 3 hours and bursts of larger savings when using off-the-shelf GenAI tools (section: Productivity Impact by Use-Case). Deep dives into engineers’ workflows corroborated this median uplift.

We were initially concerned that these tools would lead to decreased reliability. We’re finding little evidence of this downside as usage continues amongst seasoned engineers (section: Adverse Quality Impact Not Found).

This blog post will cover how we are pragmatically vetting and evaluating these tools and driving widespread adoption to accelerate our developer base.

The space evolves rapidly, with new players and tools popping up seemingly every month. There is not a clear winner in the market for GenAI coding assistants, and there appears room for disruption of the current market leaders. When we build GenAI into the core of our platforms to improve developer experience, we won’t do it in a way that couples us to any particular companies.

Our Journey So Far

We’ve been early and consistent adopters of new GenAI tools and capabilities. This extended to Developer Experience, which we illustrate in the figure below.

For example, early on, we built our LLM Gateway. Investing in this platform wrapper allowed all our engineers to interact with a variety of LLMs without spending time on the burdensome scaffolding of integrating with vendors while allowing us to manage rate limits and have fine-grained usage and cost reporting.

Marketing Hype

As we steadily integrate GenAI tools into our daily workflows, we’ve been noticing some other early adopters expressing extreme positions. This started raising questions for us.

Massive amounts of useful code generated by “10x engineers” - Source

The age of superbuilders has begun - Source

Strong top-down AI-usage mandates - Source

With so much hype, we realized we needed to immerse ourselves and perform our own study to uncover the truth. We summarize some of our findings in further sections.

Comparison of GenAI Vendors @ Atoms

We continuously reassess our tools and strategies to stay on top of newer capabilities and opportunities. We hold off widely rolling out tools before they’re useful so engineers aren’t put off by negative first impressions. Below is our latest assessment as of August 2025.

The CK Grade™ is a relative score, with A being the maximum value. There is no A+ since “There’s always room for improvement” according to my high school English teacher.

Since the developer satisfaction has been high with these tools, driving adoption has been fairly organic without the requirement of any strong top-down mandates. We’ve run a couple lightweight AI enablement sessions, but developers see others using it effectively and being more productive, get curious and test it out themselves.

Productivity Impact by Use-Case

Our baseline engineering productivity is high. CK engineering candidates are mainly sourced from big tech or highly successful startups. And our interview pass rate is only 1-3% despite strong sourcing pedigree. Of these engineers, we only included engineers in our study group with 5+ years of coding experience.

We measure a basket of metrics and countermeasures to try and understand impact. No single measure is sufficient. For example, engineers who opted into using GenAI tools experienced a 10-15% uplift on lines of code shipped. But without the ability to control for participant enthusiasm, this single measure doesn’t clearly show causality. Nor is more LOC necessarily a good thing.

Therefore, we also kept tabs on SPACE metrics, DAU, MAU, # completions, self-reported time savings, and did a number of deep dives into individual engineering workflows to try and overcome self-reporting bias (see study).

Our conclusion from these composite signals is that our engineers are saving a median of 3h per week as a result of using GenAI DevEx tools like Cursor compared to a baseline of conventional development + ChatGPT usage. And the time saving varies widely based on activity. We corroborated this via individual deep-dives.

Notably, we have not been able to tie back any of these improvements to overall engineering project velocity. We believe that this is because active coding takes <25% of engineers time typically (see study). And even while actively writing code, a small subset of the time is typically spent on tasks where these tools shine.

Adverse Quality Impact Not Found

GenAI-produced code can contain subtle bugs or shocking naivety that humans would not produce themselves. And, engineers find GenAI especially useful for generating code in domains they are unfamiliar with. For example, backend engineers using GenAI to generate frontend code. As a result, it’s easy to imagine that issues in code being produced by GenAI might slip undetected by the engineers using GenAI into our production code base.

At this early stage, we haven’t seen evidence of quality issues impacting production (incidents, bugs, etc). To try and find other leading indicators of risk, we surveyed teammates of the heaviest GenAI users at the company.

We asked a number of related questions to try and uncover problems. The most common complaint: the frequency of seeing code with style mismatched from our codebase has increased. This seems minor and fixable.

We haven’t seen a serious trend of increased bugs being discovered during review or overall reviewer burden increasing.

Interesting GenAI Applications

There are some basic GenAI applications involving summarization and contextual chatbots that index relevant data from Confluence, Slack, Google Docs and GitHub.

We wanted to explore avenues beyond that. Here are some such use-cases that are already live.

On-call AI

Instead of restricting GenAI to only code-writing, we also built an agent that queries observability data, metrics, and logs, providing engineers with crucial insights to quickly pinpoint and mitigate incidents. Early experiments demonstrate the agent's remarkable effectiveness, identifying incident root causes in one or two shots. Future improvements in this space will be evolving beyond the text-based chat interface, improved context management and custom fine-tuned LLM models specifically geared towards observability data.

The reason we can pull something like this off meaningfully is due to the past standardization efforts of our CICD processes and overall service observability.

Monorepo Agent

A GenAI tool equipped with MCP servers to understand the entire software development process at Atoms, starting with branch creation, local iteration, PR creation and CI validation.

Our first usage of this tool was on toil-heavy maintenance tasks, such as cleaning up static analysis suppressions that had been lingering in our backlog.

AnalyticsGPT

This is our take on the next generation of data analysts. A tool that’s designed to help answer data queries via natural language while retaining the user’s access controls which shows its chain of thought so you can trust and verify the analysis.

Superset Agent

We find that impact and adoption is organically high when we add targeted features to where developers are already spending a lot of time. An example is our agentic superset assistant that comes baked in with the full knowledge of the dataset and the table schema.

Future DevEx Platform Explorations

From Single Agent to Distributed Execution

As GenAI tools evolve beyond simple code completion, we're seeing a shift toward distributed agent systems that pass state and partial task completion between specialized components. This approach mirrors how software teams coordinate work, but compressed into a single developer's workflow. It’s almost like we’re reinventing how teams collaborate and are all playing the role of an Engineering Manager.

Investing in ephemeral development environments could pay off here to avoid being constrained by local resources.

Investing More in Evals and Observability

Due to the inherent non-determinism that comes with LLMs today, we need to build in mechanisms to know how well an LLM use-case is performing and monitor its quality and adoption over time.

E2E Agentic Campaign Management

Right now, we have the ability to have an agent with an understanding of our end-to-end developer lifecycle, but as always, we are hungry for more.

We want to be able to make it easier to run larger campaigns that require several dependent changes with more granular checkpoint validation. This will allow us to make the leap from smaller one-off changes like fixing specific bugs to doing full-fledged framework migrations - from an individual productivity boost to collaborative workflow with org-wide impact.

One concrete test case we’d like to prove this for is our white-glove migration of ML and DS tech stack’s package manager from Poetry to uv, specifically to validate the claimed performance improvements.

Conclusion

We are very excited about this new technology that makes developers feel like we have superpowers. We’re optimistic about the future, but we’re not seeing the “fall of software engineering” or that “SaaS is dead” like tech social media would have you believe. We’re actually seeing even more reward for good engineering fundamentals.

We’re publishing this blog post at the risk of it being out-of-date within a few weeks (if we’re lucky). But then again, maybe we could just have AI generate the next iteration autonomously. Right?

Written by Frederik Mogensen, member of the storage team who led the development of the Bucket-Gateway infrastructure.

Two years ago, we started a major cloud migration project between two large cloud providers. We migrated our entire microservice stack with hundreds of services, Kubernetes clusters, databases, low level infra, and streaming/batch pipelines in less than a year. And, we did it without downtime or material involvement from product engineering teams.

In this post, we focus on our strategy for migrating hundreds of buckets and petabytes of data across cloud providers. Discover how a cloud-agnostic, portable, and secure blob storage solution made this possible while solving other major problems for us.

Initial design requirements

The main requirements for the cloud migration solution were to be able to move all data from one cloud provider to another, without noticeable disruptions for the clients.

To accomplish this we identified the need for the following three key properties.

• A uniform data protocol on top of each cloud provider.

• A solution to transparently move data between providers without disrupting data access.

• A global authentication and authorization system for all requests, no matter which provider a client tries to access.

The uniform data protocol

We wanted to ensure that our applications would run seamlessly on any cloud provider going forward. This meant that the blob storage protocols used in all our in-house and open source projects could stay the same, no matter the underlying cloud provider. There is currently no shared standard protocol that works on all providers. Each cloud provider has their own blob storage infrastructure with custom APIs, and client libraries for each language with caveats and bugs.

Transparent data portability

The solution must allow for seamless data movement behind the scenes, as seen from the clients point of view. Like a modern cloud-native storage engine, such as CockroachDB where we can define the desired data placement and data moves automatically. Not like running on old-school hard drives where data is stuck when it is written, unless manually copied around.

Security and Audit

Blob storage stores and manages sensitive data for all parts of our business. Therefore our new blob service would need to include a strong authorization and authentication strategy. Leveraging the Spiffe network identities all our applications already get from Istio we can implement a zero-trust blob storage system that works for all cloud providers. No more access key distribution and rotations and no more provider specific service accounts for each microservice.

Finally our new design should also include uniform observability and audit logging for all backing cloud providers. The new blob storage implementation should know when any data was accessed, by applications or developers, as well as be able to generate cost insights down to the single blob level.

Architecting for portability and scalability

To create a cloud-agnostic layer that will work for all applications we decided to implement (most of) the S3 protocol. This protocol is very well documented and supported by most modern applications.

Our new Blob Storage architecture consists of a set of management components, and a new horizontally scalable stateless gateway.

Request flow from Client Application to backing blob bucket

The Bucket-Gateway

The Bucket-Gateway is designed to be S3 compatible and capable of fronting multiple cloud providers, such as Azure Blob Storage, Google Cloud Storage, and Amazon S3. This new component allows us to serve logical buckets that can be routed to actual cloud buckets on different cloud providers. By using the Bucket-Gateway to enforce Authentication (AuthN) and Authorization (AuthZ), it also allows us a single place to control access to our data no matter which cloud provider the data is actually stored on.

Routing

The Bucket-Gateway keeps the complete list of logical buckets (e.g. my-frontend or postgresql-backups), and the mapping for which actual backing cloud buckets to route the requests to (e.g. azure-cloudkitchens-frontend-xyz or gcp-cloudkitchens-pg-backups-123).

This allows us to swap the backing bucket to another region/cloud at any time, without having to re-deploy, re-configure, or even inform our clients and stakeholders. It also removes strict requirements that any bucket name needs to be globally unique which most cloud providers have.

Authentication and Authorization

To allow for easy onboarding of all the client applications we decided to add two different authentication methods. The standard S3 authentication way with Access key ID and Secret Key (Blob Access Keys), as well as a new option of using Spiffe identities from Istio. The Blob Access Keys are used to authenticate any workload or person running outside of our service mesh, as well as for some open source applications that do not allow anonymous authentication in their S3 blob implementations. The Spiffe Identity authentication is used by all applications inside our mesh. Running in the mesh with mTLS enabled between all applications, we have a strong enough security and consistency guarantee from our transport layer that we can remove the signing and checksumming done by the standard Blob Access Keys authentication. This removes a lot of crypto and hashing CPU cycles, as well as allows us to make request handling in a much more streaming fashion.

The Spiffe network identity authentication model allows us to create a zero trust architecture between all client applications and the Bucket-Gateway. Using Spiffe network identities for AuthN removes the need to distribute and rotate keys, as this is already done by Istio. The Bucket-Gateway can use the Spiffe identity from any incoming http request and ask our in-house ABAC-style authorization service if the calling service is allowed to perform the desired S3 action, on the given bucket, for the specific path.

Authorization: The Bucket-Gateway will call the Authorization Service to get an authorization verdict for each incoming request.

Verdicts are cached for a short time, to keep latencies down and guard against DDOS attacks.

# The following is an example of a ABAC style access request

Principal = "postgresql-operator"
Attributes
    namespace = "BLOB"
    action = "GetObject"
    bucket = "postgresql-backups"
    object = "myPgCluster/wal/123.tar"

Observability

Because the Bucket-Gateway handles all access to blob storage on all our cloud providers, this is a great place to add the required observability. The Bucket-Gateway exposes Prometheus metrics, audit logging, tracing, and all info needed to attribute costs to any client application. Getting request/response logging with the resolution down to a single blob will either be impossible or extremely expensive depending on the cloud provider. With this we can get cost attribution all the way down to a single blob. We know exactly which client read/wrote a given blob, when they did, and how often.

Tradeoffs

Implementing a proxy service like the Bucket-Gateway will of course also introduce certain costs. First, some latency is inevitably added to each data request as it must pass through an extra network jump and a service before reaching the actual cloud provider. We spend a fair amount of time optimizing the Bucket-Gateway by keeping memory allocations low and ensuring that any metadata we need to handle a request is already cached in an up-to-date in-memory cache. Compared to the time spent moving large amounts of data into and out of external storage the extra latencies from the Bucket-Gateway is negligible.

The Bucket-Gateway also introduces an additional point of failure. Apart from rigorous testing, we mitigate this by autoscaling the Bucket-Gateway horizontally across multiple cloud regions, and by isolating heavy data warehouse clients from latency sensitive end-user traffic.

Migration flow

The migration flow consists of moving data and changing the source of truth. For the actual migration we need two extra components.

The Bucket-Migrator

The Bucket-Migrator is responsible for moving the actual data from one backing bucket on any cloud provider, to another backing bucket on any other cloud provider.

It handles initial backfill (Copy) of the new backing bucket by copying all blobs from the source bucket to the sink bucket, as well as synchronizing (Sync) the content of the two backing buckets on demand.

The Bucket-Operator

The Bucket-Operator looks at a desired specification for a bucket and tries to ensure that the real world matches the spec.

For a cloud migration this is done by provisioning a new backing bucket in the right cloud and region, backfilling the new bucket by starting a job in the Bucket-Migrator, and setting up the correct authorization policies for any client on a given bucket.

During the migration from one backing bucket to another the Bucket-Operator is responsible for changing permissions on the logical bucket (setting the bucket as read-only at critical times), triggering copy and sync jobs in the Bucket-Migrator, and updating routing information for the logical bucket. Thereby setting the new bucket as the source of truth.

Copy and Sync

To ensure write-unavailability is kept to an absolute minimum we implemented the following Copy-and-Sync design.

Migration flow: Bucket-Operator manages the permissions on the bucket, and instructs the Bucket-Migrator to copy and sync blobs.

The Copy Phase

The Copy part of the algorithm will make an initial bulk copy of all blobs in the source bucket to the sink. This is performed by listing the source bucket and starting a server-side upload for each blob, by signing a URL for the source bucket and instructing the sink bucket to fetch the blob from the signed-URL. This ensures that we do not have to pull all the data through our Kubernetes clusters and internal network.

The Copy phase does the bulk of the work and the duration of this phase depends on the number of files in the bucket. The phase is completely asynchronous with no impacts on the client workloads, even though it might run for multiple hours or days.

The Sync Phase

The Sync phase of the migration lists all files in the bucket on both source and destination to check whether they are identical. It performs the following series of checks and actions:

• Check whether any blobs are missing in the sink

  • If any blobs are missing in the sink, it uploads them from the source

• Checks whether any blobs exists in the sink that are no longer in the source bucket

  • If any such blobs exist, they are deleted.

• Check if the ETag on the sink and source blobs match

  • If not it re-uploads the new version of the file

• Check if the Metadata content matches the sink and source blobs

  • If not it updates the metadata

The duration of the Sync phase depends on the number of files in the bucket and the amount of changes performed since the one-time copy was started. During this phase the bucket must be read-only to ensure consistency.

The actual migration process consists of the following 6 simple steps:

1. Provision new backing bucket on destination cloud provider

2. Initial One-time Copy of all blobs from the Source bucket to the Sink bucket

   1. This part is completely asynchronous and has no impact on the availability of the bucket

3. Disable writes on the bucket in the Bucket-Gateways

   1. After this the bucket is only read-available.

4. Sync blobs that has changed or been deleted since the Copy phase

   1. The length of this step is proportional to the number of blobs changed and deleted since the copy phase.

   2. To ensure this step is as fast as possible we plan it for a time of day/week when the given bucket has the least amount of activity.

   3. The sync can also be run multiple times without problems. This means that for high through buckets, we can run the sync once, then disable writes, and then run the sync again. In this case the last sync only needs to handle the blobs updated while running the first sync.

5. Update the routing information for the logical bucket in the Bucket-Gateways to route all requests to the Sink bucket.

6. Re-enable writes in the Bucket-Gateway for the logical bucket.

   1. The bucket is now completely migrated and write-available again.

For some clients we were able to completely skip the Sync phase, by pausing client ingestion pipelines, such as Flink and Spark jobs or frontend build pipelines.

The End Result

We created a cloud-agnostic interface and a new Blob Storage architecture with horizontally scalable components, including the Bucket-Gateway, Bucket-Operator, and Bucket-Migrator. The Bucket-Gateway handles S3-compatible requests, routing, authentication, authorization, and observability. The Bucket-Migrator orchestrates data migration using a Copy-and-Sync design, ensuring minimal write unavailability and no read unavailability.

We successfully onboarded all our stakeholders to the new Bucket-Gateway instead of going directly to the cloud provider proprietary APIs. Using the tooling described above we migrated hundreds of buckets with petabytes of data, from multiple regions and environments, to our new home cloud. Most clients experienced only a few seconds or minutes of write unavailability, and no clients experienced read unavailability.

This new architecture also allows us a very interesting set of additional features. The later iterations of the Bucket-Gateway includes

• Sharding buckets across Storage Accounts in Azure to remove Azure throughput limits.

• Implementing shared in-cluster caches to skip reading the same file multiple times in different services

• Multi-region/multi-cloud buckets with replicated data for local reads and disaster recovery/business continuity.

Read more about how we implemented those in the next article Cloudless Blob post.

Cover photo by Carlos Horton.

Written by LV Lu (Data Science), Rob Harell (Product), and Brian Attwell.

An exceptional customer support experience is the cornerstone of a lasting customer relationship. This is why we built Otter Assistant, our in-house Gen-AI chatbot that currently handles ~50% of inbound customer requests quickly without human intervention. This post describes our journey over the past year building and scaling Otter Assistant.

About Otter

Otter is a delivery-native restaurant hardware and software suite used to manage restaurant operations and aggregate & derive insights from restaurant data. Each of its product lines has many features, with many integrations. And restaurants want a lot of customizability.

A broad feature set creates customer demand for support. These customers appreciate the speed and reliability that Otter Assistant can deliver over traditional call center agents, as long as we continue to offer the option to contact a 24/7 human agent if preferred.

Build vs Buy

In Q1 2024, we started by analyzing the distribution of customer issues and found that resolving these tickets required deep integration with our systems. For example, a support agent has tightly controlled permission to review a customer’s menu, update their account, or tweak remote printer configuration. At the time, there were no vendors that offered our needed level of integration flexibility without hard coded decision trees, like shown in the image below.

And when we began to experiment, we found that much of the historical value provided by these vendors (Zendesk has great features for configuring workflows through UI, NLP based intent matching, etc) was significantly reduced by LLMs. LLMs reduced the need for this vendor infra, allowing us to largely focus on domain specific problem solving when building our agent.

Bot architecture

The bot system and workflows span an online conversation flow and an offline management flow. The next section covers the primary components of each.

Conversation flow

When we began implementation in Q2 of ‘24, the term “agentic” had yet caught on. However, by setting out to emulate the diagnostic and resolution steps taken by our agents, we naturally followed agentic approach. Concretely, we designed the bot to use function calling and mimic how human support agents work:

1. Based on customer request, find out the corresponding predefined procedure

2. If there is one, follow the steps

3. If not, try to conduct research in knowledge base

4. When run into issues or can not find information, escalate

To accomplish this, we designed four main types of functions: GetRunbook, API & Widget Functions, Research, and EscalateToHuman, to accomplish each of the essential support tasks above.

GetRunbook function - injects relevant issue resolution steps

After evaluating support issues by volume and resolution complexity, we selected the top high-volume and low-to-medium complexity issues to translate into bot “runbooks”. These runbooks supply instructions on the steps the bot should take to diagnose the user’s issue statement and tie it to a known root cause, and then to either make the necessary API calls to resolve the issue, or surface the corresponding solution “widget” (both of which will be covered below). Conceptually, the runbooks function like a decision tree, but unlike the previous generation of bot tech, can be written in plain text, making them 1) significantly easier to implement and maintain 2) more modular and traversable during runtime diagnosis.

Mechanically, GetRunbook function takes the user issue description as an input and outputs a corresponding runbook if it can find one. Otherwise it returns “Not Found”. Under the hood, we use a LLM for intent matching and runbook retrieval. Based on the embedding of user issue description, retrieve relevant runbooks from our runbook repository (vector db) based on semantic similarity [Ref], then issue a separate LLM call to let LLM pick the correct one from the candidates and return not found if there is no good match.

If there’s a match, the LLM then begins to work through the listed runbook steps, gathering follow-up information from the user and/or executing API calls as needed until it reaches the end.

API call functions - retrieves customer data for diagnosis

As the bot works through a runbook, it has the ability to choose from a list of API wrapper function calls to gather information (e.g. fetch store status status) and/or modify a user’s account. Fortunately, we were able to largely reuse pre-existing API calls within the Otter ecosystem.

Critically, to avoid leaking user data to those that shouldn’t have access, for any internal APIs, we call backend APIs with the user token passed over as part of each Otter Assistant service request. This way, we maintain and reuse existing permission control models and auth infrastructure.

Widget functions - performs action with user confirmation

After the bot has identified the root cause, it then takes the appropriate action to address the issue. With exception of simple account modifications, we present most write operations within “widgets”, or embedded UI modules. For example, customers requesting to pause their store are presented with the following widget:

Widgets provide the following benefits:

1. Encapsulation/reuse

2. Distributed ownership

3. Information density

4. Easy for user to confirm & eliminates risk of hallucination

When the bot decides it’s appropriate to surface a widget, it calls the corresponding widget function (informing the LLM that a widget is being displayed) and simultaneously emits a notification to the external chat UI. The chat UI then renders the widget UI within the message.

For any critical write operation (e.g. Pause store, update price) we require user review and explicit click confirmation before kicking off the operation. We strictly follow this approach to mitigate risk from LLM hallucination (e.g. incorrectly assume that the user wants to pause a store). This approach also provides users a quick way to modify inputs for the write operation if the LLM got details wrong.

Research function - finds answers in knowledge base

The research function is designed to retrieve and summarize helpful answers to user questions that don’t match a runbook. We designed the research function to mimic how humans find answers online in our help articles: conduct a search with a question, then read through top search results to come up with final answers.

To implement this flow, we first convert the help articles in Otter’s knowledge base to embeddings using an LLM offline and store them in a vector db. Then, when we receive a request, we convert the user question to embeddings to retrieve the top relevant articles from vector db based on semantic similarity. Next, we issue a LLM request to each top article to find relevant answers to the user’s question. The process stops when either it has found n answers or gone through m results (both configurable parameters). We lastly issue a separate LLM call to combine the answers into a final answer to return as the function response.

EscalateToHuman function - hands off to human agent

This function provides LLM the capability to inform us the conversation should be escalated to a human agent. When the LLM detects a user’s intent to escalate, we can inform the chat message interface to pass conversation control to the assigned human agent, which in turn calls Zendesk to connect to a live agent.

Bot management

The aforementioned components cover Otter Assistant’s core conversational capabilities. However, like with any software, robust testing and management processes are needed to ensure the bot works at scale. Unlike with traditional software, however, the inherent randomness and unpredictability in the LLM-powered conversational flow called for a bespoke set of tools to serve this need:

1. Local development and playground

2. Bot validation testing

3. Bot conversation review & analytics

Local development and playground

Given the stochastic nature of LLMs and multi-modal nature of Otter Assistant conversations (which encompass both text and bot actions/widgets), developers require a chat simulator for effective debugging. To facilitate this, we developed a Streamlit-based library. This library allows developers to interact with the bot through a web UI, displaying input and output arguments for each function call to ensure the bot's end-to-end flow is correct.

Bot validation testing

After confirming new capabilities work in the development environment, we pass the bot through a round of validation testing. Given the randomness inherent in LLM systems, it often requires multiple iterations of conversation to expose & verify specific bot behaviors, which is time consuming if done manually. Moreover, changing prompt logic in one place could cause unanticipated behavior changes elsewhere that could be difficult to detect. These challenges surpass traditional software testing frameworks, which rely on 100% deterministic execution and structured output.

Therefore, we developed a new test and evaluation framework for Otter Assistant and any other chatbot, which involves:

1. Predefine a set of test scenarios, e.g. customer’s store is paused

2. For each test scenario we also define a list of expected behaviors, e.g. confirm which store, check status, then launch widget

3. Launch a chatbot using LLM to play as customer to chat with our bot

4. Leverage LLM as a judge to assert on expected behaviors based on conversation transcript between the bot and customer

With the above framework, we can evaluate chatbots through a mechanism that is similar to traditional unit tests, where we define a set of inputs and assert on expected output for each.

Bot conversation review & analytics

Once the bot has been validated, it can be deployed with a reasonable degree of certainty that it will behave as expected. But then comes the question: how is it performing? To answer this, we defined and instrumented a “resolution” metric. This metric informs us of the Bot’s overall performance and in turn the business impact it generates, and allows us to identify issues and improvement opportunities.

Bot issue analysis presents challenges compared to error identification and resolution in traditional software development. Concretely bots can err many ways at the software layer and the model layer, and it’s impossible to know which without manual inspection. To streamline this conversation review process, we built a conversation inspector tool in Streamlit that allows reviewers to load each past conversation and visualize the chat history and action logs similarly to the local testing app:

This tool is available to both Otter developers and non-developers, which has helped scale our efforts to investigate issues and suggest improvements.

Lessons from running a bot initiative

When we began implementing Otter Assistant last year, there were no established bot guidelines or frameworks. Though frameworks have begun to emerge (e.g. OpenAI’s Agents SDK), and as solution providers continue to enhance their offerings, we still feel building in house has proved to be the right decision. Other organizations should weigh their build-vs-buy according to their abilities and the degree of control and customization they foresee requiring for their use cases.

Beyond build-v-buy, the most important takeaway from the development and launch of Otter Assistant has been the importance of defensible, actionable success metrics to the overall project’s success. These metrics have proved instrumental in persuading ourselves of the Bot’s value to Otter and users and establishing a feedback loop to improve the bot over time.

Lastly, Otter Assistant (specifically, the high fidelity conversational feedback generated by Otter Assistant) has exposed multiple product and platform issues previously lurking undetected in Otter systems. We’ve thus incorporated bot-derived feedback into our product strategy alongside traditional sources such as user interviews and competitive analysis.

Next steps

After close to one year of development, Otter Assistant solves ~half of support requests autonomously without compromising customer satisfaction. In future blogs, we will share more about our lessons learned around prompt engineering, as well as best practices we found for how to design and structure functions.

While it is great to see existing LLMs frameworks already starting to deliver value to our customers and unlocking use cases that weren't possible before, in certain scenarios, we have started to hit limitations on how much we can improve without more fundamental improvements on the LLMs. Therefore we are exploring how to establish a more efficient feedback loop mechanism so the bot can self-sufficiently become smarter over time.

Looking ahead, we think this is just the beginning of a new era for product design and development. At CSS, we believe agentic chatbots can hugely elevate customer experience. Handling customer support requests is just a starting point!

Appendix: Q1 2024 Vendor Comparison

After categorizing support tickets, we identified the following key requirements for our chatbot.

1. LLM-native: no reliance on hard-coded decision trees to define Bot logic

2. Ability to choose the underlying model(s) and control prompt text

3. Ability to update user accounts (stores, menus, orders, printers, etc) via API function calls, while maintaining adherence to Otter’s access controls and permissions as a guardrail on the above

4. Ability to seamlessly escalate from bot to human within a single chat window

With these key requirements in mind, we conducted an evaluation of third party solutions while simultaneously developing an internal prototype Q&A bot that performed RAG on our existing support knowledge base. Below was our comparison.

Established vendors' products primarily featured hard-coded decision trees and were still working to determine their LLM product strategy. On the other end of the spectrum, we spoke with several startups building LLM-native chatbots for support, but didn’t encounter one we believed would be able to manage the complexity of resolution steps required for our top issues. We thus decided to build our own in-house Bot back end while leveraging Zendesk’s Sunco Web SDK front end (to minimize time to market; we have since replaced it with our own custom front end) as our MVP solution.

Written by Wenpin Cui, Shao Liu, and Alex Filipchik, members of the engineering teams that work on infrastructure.

For months, small spikes of unexplained 520/524 errors sat quietly on our dashboards, less than 0.03% max, easily ignored. Nobody complained, so the Infrastructure team put it in the we-will-get-it-done-eventually (read never) bucket. However, even minor issues can signal deeper problems. When one of these elusive errors unexpectedly disrupted a key customer experience, it quickly escalated from a rare anomaly to a critical infrastructure investigation, leading to very intriguing discoveries.

Chasing Ghosts in the Infrastructure

Once we knew that those disruptions were indeed a real problem, we jumped straight to our logs, expecting clear answers. At Cloudkitchens, we run most of our web applications on Azure AKS, fronted by Cloudflare (fairly solid stack) and a very simple proxy (Ingress Gateway), as illustrated by the above diagram, so our first instinct was to look for possible issues in either product stack (bugs) or our own infrastructure. Surprisingly, we found nothing suspicious (like missing health checks or improper shutdown sequences). The next suspect was our service mesh. We inspected all the logs and metrics we collected, but found nothing suspicious.

We realized we needed deeper instrumentation, so we started capturing Ray IDs, a special trace request header that Cloudflare can supply. The next time the errors hit, we’d have the fingerprint, which we can trace through all the layers.

When the next spike came, we caught it immediately, only to find something even stranger. The Ray IDs for these failed requests were still missing from every log inside our cluster. They weren’t just failing; they never even made it into Kubernetes.

The mystery deepened: our infrastructure seemed innocent. Was the problem actually upstream, somewhere in Azure’s managed load balancing?

Ray ID: How to

If you are wondering how to capture Ray IDs yourself (assuming you are using Istio as well), here is the configuration example:

Diving Deep Into Azure AKS Internals

At this point, we knew the packets were disappearing somewhere upstream, but where? We had to figure out exactly how Azure’s Load Balancer interacts with AKS services. Initially, we assumed that the Azure Load Balancer picked a healthy pod to route requests to, perhaps randomly or in round-robin fashion. It seemed logical enough, but as it turned out, it wasn't how it worked at all.

Confused? We were, too. So we dug deeper and realized Azure Load Balancer doesn't route directly to pods. In fact, Azure Load Balancer has no knowledge of Kubernetes pods, nor does it know which node hosts which pod or which pod belongs to which service.

In reality, each AKS cluster automatically receives two Azure Load Balancers: one for external and one for internal traffic. Every time we create a Kubernetes Service with type Load Balancer, it is mapped to an external Azure LB, which comprises several key components:

• Frontend IP: The public IP address for ingress traffic. Each frontend IP maps to a Kubernetes LoadBalancer service.

• Backend pool: The group of VMs or Virtual Machine Scale Set instances serving requests. AKS creates a backend pool containing all Kubernetes nodes. Crucially, Azure Load Balancer doesn't route directly to pods. Instead, it routes traffic to nodes. Not just nodes running the service, but all nodes in the AKS cluster.

• Load-balancing rules: Define how incoming traffic is distributed across backend pool instances. Each rule maps a frontend IP configuration and port to multiple backend IP addresses and ports.

Thus, every request makes two hops to reach our pods:

1. Azure LB → Node: Azure LB picks a random healthy node from the entire cluster.
   

2. Node → Pod: The node relies on IPtables rules configured by kube-proxy to forward traffic to one of the ready endpoints associated with the LoadBalancer service, which could be on a different node. IPtable internally also uses a conntrack module to maintain the mapping so at TCP level, the routing is consistent.

Here is a detailed example of the configuration in question:

Given a service that we want to expose externally:

A corresponding frontend IP configuration, matching the public IP found in .status.loadBalancer.ingress[0].ip of the Kubernetes service, can be located within the Azure Load Balancer.

Additionally, a load balancing rule is established to direct traffic from the frontend IP to the appropriate backend servers.

In the above example rule, you can see that Azure Load Balancer configures the frontend IP, backend pool, and a health probe. The Azure Load Balancer only directs traffic to healthy nodes. The health probe, configured in this rule, determines node health. In this example, by default, Azure checks for a successful TCP connection on port 31260 for each node in the cluster. This port is a NodePort, meaning it's open on all AKS cluster nodes, ensuring successful health probes for healthy nodes.

Why do requests vanish?

You might be asking: What actually happens when a Kubernetes node suddenly goes away?
The answer is simpler, and worse, than you might think: timeouts.

To understand why, let's look closely at Azure Load Balancer’s behavior. Azure determines node health through periodic health probes (TCP checks) every five seconds. If a node abruptly fails, such as during a spot node eviction, the Azure Load Balancer may still send traffic to it for several seconds, until the VM is eventually removed from the serving pool.

During that brief window, packets sent to these unhealthy nodes vanish silently, never reaching your pods. From our customers' perspective, this manifests as unexplained 520/524 errors from Cloudflare, depending on the exact TCP connection state when the node disappeared.

But why doesn't AKS handle this better? Ideally, nodes should gracefully terminate, actively closing connections and immediately signaling to Azure Load Balancer to stop routing traffic their way. Unfortunately, AKS doesn't currently handle node termination gracefully. We suspect flaws in this termination process prevent AKS from promptly deregistering nodes from the load balancer backend pool.

As a result, even healthy services running exclusively on stable nodes are vulnerable to intermittent errors caused by unrelated node failures, especially if Spot VMs are churning.

Mitigation: a workaround with trade-offs

With clarity on the root cause, the next logical step was to find a solution. We quickly realized that Azure’s Load Balancer doesn't allow us to explicitly exclude spot nodes, but Kubernetes itself provides an important configuration option: externalTrafficPolicy. This policy controls how incoming external traffic is distributed across Kubernetes nodes.

Default mode: Cluster

By default, Kubernetes sets the externalTrafficPolicy to Cluster. In this mode, the Azure Load Balancer forwards traffic evenly across all nodes in your AKS cluster, regardless of whether or not a given node hosts relevant pods.

This design aims to evenly spread load, but we've discovered a critical drawback: when unrelated nodes (such as spot instances) fail abruptly, Azure continues to route traffic to these failing nodes until their health checks fail, leading to intermittent packet drops and 520/524 errors.

Alternative: Local mode

The alternative is to set externalTrafficPolicy to Local. This option addresses the problem by instructing the Azure Load Balancer to forward traffic only to nodes that are actively running pods associated with the LoadBalancer service.

As explained clearly in Azure's official AKS documentation:

"With the Local traffic policy enabled, the load balancer health probes automatically detect which nodes are running the pod for a given service and only send traffic to those nodes."

This means spot nodes, or any other unrelated nodes, will never receive traffic meant for services not explicitly hosted on them.

Here's how we configured our Kubernetes service in Local mode:

Most Azure Load Balancer configurations remain unchanged, with one key difference: the health probe switches from TCP to HTTP. Kubernetes manages this transition, ensuring only nodes hosting Istio Ingress Gateway pods pass the health checks. Because the second hop (node to pod) occurs locally in Local mode, latency is also reduced compared to Cluster mode.

Switching our ingress gateway to Local mode immediately solved our 520/524 error problem, reducing errors to nearly zero. It also had a latency benefit, fewer hops mean less delay in the traffic path.

However, the Local mode comes with trade-offs, also documented by Azure:

• Uneven load balancing: Traffic only flows to nodes hosting pods, which might create load hotspots.

• Pod scheduling considerations: If your ingress pods relocate frequently, traffic may be temporarily disrupted as Azure’s Load Balancer re-adjusts.

Due to these trade-offs, Local isn't Kubernetes’ default choice. While our adoption of Local mode proved stable, we still consider it a mitigation rather than a complete solution. Ideally, Azure would provide a more robust fix, such as better node eviction handling or configurable backend pools.

Until then, Local mode keeps our traffic stable and helps us sleep better at night.

Summary

We operate several large AKS clusters that experience regular node churn due to the usage of spot instances and our autoscaling policies. During our investigation into persistent but elusive 520/524 errors, we uncovered a fundamental limitation with Azure Load Balancer's default Cluster traffic policy.

The key problems we found:

• Insufficient Documentation: Documentation doesn’t clearly warn users about the reliability implications of the default Cluster policy.
  

• Default Mode Pitfalls: AKS defaults to Cluster mode, suitable perhaps for smaller or stable clusters but problematic at scale, especially when spot nodes are present.
  

• Limited Flexibility: AKS currently offers no way to selectively exclude unstable nodes or node pools from the Load Balancer backend pool, limiting our ability to control availability.
  

• Suboptimal Node Termination: AKS does not gracefully deregister terminating nodes from Azure Load Balancer backends, leaving windows of vulnerability during node evictions.
  

Switching to the Local external traffic policy significantly reduced our ingress availability issues. But Local comes with trade-offs, such as uneven load distribution and increased scheduling complexity. To mitigate this, we use pod anti-affinity rules to ensure even distribution.

Written by Karthik Manamcheri and Alex Filipchik, who lead the data and the infrastructure teams.

Introduction

In a perfect world ML infrastructure would work like a well-oiled machine, balancing competing needs for flexibility, usability, maintainability, and cost effectiveness. Time from idea to production would be mere minutes. Let's be honest: many companies, including ours, often fall short. Users face a jigsaw puzzle of systems cobbled together with digital duct tape. "Synergy" isn’t exactly the word that came to mind.

Over the last year, we’ve begun fixating on productivity of Data Scientists foremost. We swapped out some conventional tools and patterns. Data Science teams estimate iteration speed has already increased several fold with our initial results.

Lessons from Our First Attempt

Our initial ML infrastructure was ambitious but flawed. We aimed to build an industry-leading and future-proof system which supports undefined use cases and avoids vendor lock-in. This led us to a tech stack that included

• Kubernetes and Istio for the compute and service mesh

• Trino and Apache Hudi as the data warehouse layer

• Argo and a modified kubeflow pipelines SDK for workload orchestration

• MLFlow, Jupyter Notebooks, and Seldon for rapid model iteration

• and Bazel as our build system

While this setup offered flexibility and extensibility, it was overly complicated. Data scientists faced steep learning curves with containers, Helm, and YAML. Simple ML tasks took over 10 minutes to start, and setup times for more complex tasks could exceed 45 minutes. Bazel, though powerful for languages like Java and Go, was cumbersome for Python due to compatibility issues.

Through this experience, we learned several key lessons:

• Complexity Hinders Productivity: Even powerful tools can become a liability if they're too complicated.

• User Experience is Crucial: Prioritizing usability is essential to ensure efficiency and ease of use.

• Flexibility vs. Functionality: It’s important to balance flexibility with the need for a simpler, more maintainable system.

Realizing these complexities were holding us back, we knew it was time for a change. In the next section, we’ll explore our new, user-focused ML infrastructure. This redesigned stack eliminates the pain points of the past while maintaining the scalability and ensuring we are prepared for future needs.

The New ML Infrastructure

Internally, we call our stack the ✨DREAM✨ stack!

Daft
Ray
poEtry
Argo
Metaflow

Our new ML Infrastructure can be viewed in roughly 3 layers: Infrastructure, Services, and Libraries. Here is a diagram of each layer and corresponding tech stack:

Kubernetes: As we mentioned, we are a big Kubernetes kitchen (look at our cool Kubernetes related blog posts here, here, and here) and we run all our applications and services on it. Our old and new ML Infrastructure run on Kubernetes, so there are no changes there.

Argo: Our trusted workflow orchestrator over the past five years. However, it does come with some challenges. First, it’s quite low-level, often diving into infrastructure details that users would rather not deal with. The primary interface is YAML, which can become complex quickly, even with language wrappers. Another challenge is that developing and testing locally is significantly different from running it in a cluster. Local development requires custom CRDs and ongoing maintenance, making the process a bit complicated. To address these issues, we introduced Metaflow, which tackles these specific challenges and simplifies the workflow by keeping Argo behind the scenes.

Metaflow: This is a user-centric framework that helps data scientists and engineers build, manage, and deploy data workflows at scale.. Metaflow lets you write and debug workflows locally, then scale them effortlessly to production with Argo (or others) when you're ready to go big. It delivers the 'infinite laptop' experience by integrating smoothly with Argo and Kubernetes, giving you the power of the cloud without the headache of YAML. And since it’s all in Python, our users can finally leave the YAML struggles behind and focus on what they do best—creating impactful ML projects.

Ray: Ray is a distributed compute engine for data and AI workloads. Ray serves up a high-performance cluster with a side of Python libraries, making it a breeze to interact with. It’s got a menu of ready-to-use libraries for ML tasks like Ray Data for preprocessing, Ray Train for training, and Ray Serve for serving.

Ray is positioned primarily as a compute engine for AI/ML tasks. Given its architecture, we believe it can also excel in data processing, potentially disrupting well-established frameworks like Spark and Flink. That's why we are doubling down on Ray as our main data compute framework. However, one issue with the Ray library for data processing, Ray Data, is that it doesn't cover the full range of DataFrame/ETL functions and its performance could be improved. That's where Daft is here to save the day–

Daft: Our latest and greatest kitchen gadget for ETL, analytics, and AI/ML at scale. It fills the gap of Ray Data by providing amazing DataFrame APIs that cover our needs. In our tests, it’s faster than Spark and uses fewer resources. Plus, its seamless integration with Ray has made it our go-to for whipping up scalable, high-performance data workloads. Our users are loving its APIs and experience. It makes working with big datasets a breeze. And since it’s written in Rust, it’s practically a tech Twitter darling by default 😛

Poetry: Given the challenges of mixing Bazel and Python—an oil-and-water scenario—we sought a more suitable alternative and landed on Poetry. This well-established tool proved to be both pluggable and extensible, perfectly aligning with our needs. Its ability to support multiple library versions has been a game-changer, accommodating Python’s diverse and ever-evolving ecosystem. The adoption of Poetry has significantly improved the efficiency of our Python developers, reducing dependency conflicts and simplifying library updates. While Bazel remains our tool of choice for other languages, Poetry has streamlined our Python workflows, making development smoother and more productive.

In the next section, we’ll showcase how we used the new ML stack using a case study

Case Study: Calculating high-demand zones

At CloudKitchens, we’re all about turning chaos into culinary gold— and that starts with data. Let us tackle a deceptively simple task: calculating high-demand zones for food orders. Why? Because it is very useful to know where people are hungry before building out a new kitchen facility.

The Challenge: Find Hungry Zones Before Dropping $$$ on Real Estate

This would have taken us a week (or more) to build on our previous stack. It would be amazing if we could do this in a single day. Below we’ll show how, with our new stack. As with every problem, data is the first step…

Step 0: Scaffolding the Dream (aka Hello, World! 👋)

Every epic begins with a humble start. Ours starts with a folder.

mkdir my_project1
cd my_project1
poetry init

🎉 Voilà! A Python project is born. If you’re new to Poetry, the pyproject.toml file is like the ingredient list for our project. Here’s a quick look:

[tool.poetry]
name = "my-project1"
version = "0.1.0"
description = ""
authors = ["my.name <my.name@cloudkitchens.com>"]
readme = "README.md"

[tool.poetry.dependencies]
python = "^3.12"
css-metaflow = "*"
h3 = "*"
css-dw = "*"
...

It’s basic for now, but we’re about to turn this into a Michelin-starred script.

Step 1: Defining the Flow (the recipe 👨‍🍳)

We need to break down the task into manageable bites. Enter Metaflow—our trusty sous-chef for data workflows. Here’s what our recipe looks like:

1. Start: Logs that something’s happening.

2. Read: Get the order data.

3. Calculate: Identify high-demand zones.

4. Write: Save results for future feasting.

5. End: Logs that we’re done.

Here’s the skeleton of our flow:

from metaflow import FlowSpec, step, poetry

@poetry
class MyFlow(FlowSpec):
    @step
    def start(self):
        print("Starting the flow!")
        self.next(self.read)

    @step
    def read(self):
        # Fetch data
        self.next(self.calculate)

    @step
    def calculate(self):
        # Calculate high-demand zones
        self.next(self.write)

    @step
    def write(self):
        # Save results
        self.next(self.end)

    @step
    def end(self):
        print("Flow complete!")

if __name__ == "__main__":
   MyFlow()

This file is a straightforward Python script with a few well-defined functions, but there are three key elements worth noting:

1. Inheriting FlowSpec: This step transforms the class into a Metaflow class, enabling it to function as part of a Metaflow workflow.

2. Decorating functions with @step: Each function marked with @step becomes a distinct step in the workflow, orchestrated by Metaflow.

3. Decorating the class with @poetry: This is our in-house Metaflow plugin decorator. It reads the pyproject.toml file and ensures that all specified dependencies are seamlessly integrated and available within the Metaflow environment.

Step 2: Implementing the Flow (the cooking 🍳)

Here’s where the magic happens:

• Read: Pulls data on fulfilled orders

• Calculate: Groups orders by geographical H3 hexes.

• Write: Saves the results to our data warehouse.

Here’s the same flow in previous step, but with implementation:

import logging

from metaflow import FlowSpec, poetry, step, project
import css_dw as dw
import h3

logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)

@poetry
class MyFlow(FlowSpec):
   """Calculates high-demand zones."""

   @step
   def start(self) -> None:
       """Start the flow."""
       logger.info("MyFlow run started!")
       self.next(self.read)

   @step
   def read(self) -> None:
       """Reads order data from the data warehouse."""
       self.read_table_result = dw.read_sql_as_table(
           sql="""
               SELECT order_id, latitude, longitude,
                      subtotal, order_time
               FROM fulfilled_orders
               WHERE city='San Francisco, CA' AND
                     order_time >= '2024-01-01'
           """,
       )
       self.next(self.calculate_high_demand_zones)

   @step
   def calculate_high_demand_zones(self) -> None:
       """Calculates high-demand zones."""
       df = dw.read_sql_as_df(
           schema=self.read_table_result.schema_name,
           table=self.read_table_result.table_name,
           df_type="daft",
       )
       h3_resolution = 8
       df = df.with_column(
           "h3_area",
           df["latitude"].zip(df["longitude"]).map(
               lambda ll: h3.geo_to_h3(ll[0], ll[1], h3_resolution)
           )
       )
       self.summary_df = df.group_by("H3 Area")
                           .agg({"subtotal": "sum"})
                           .rename({"subtotal_sum": "Total Orders"})
       self.next(self.write)

   @step
   def write(self) -> None:
       """Writes high-demand zones to the data warehouse."""
       self.result = dw.write_df_as_table(
           df=self.summary_df,
           mode="overwrite",
       )
       logger.info("Output written to DW.")
       self.next(self.end)

   @step
   def end(self) -> None:  # pylint: disable=no-self-use
       """End the flow."""
       logger.info("MyFlow is ending")

if __name__ == "__main__":
   MyFlow()

The result? A table of hexagonal zones with order counts. Think of it as a heatmap for hangry customers.

Step 3: Scaling Up (Because 200 Million Rows Is a Lot)

If you’re working with millions of orders, your laptop might throw in the towel. Enter Ray, our scalability hero. With just one decorator, we turn the “calculate” step into a parallelized powerhouse:

@step
@raystep
def calculate_high_demand_zones(self) -> None:
 ...

That’s it. Ray handles the heavy lifting, so you can focus on the fun stuff.

The @raystep decorator is an in-house Metaflow extension, much like @poetry. It seamlessly integrates Ray into your workflow by submitting the code package to a Ray cluster for execution. This allows your tasks to scale effortlessly across multiple nodes without requiring any changes to your underlying code.

Step 4: Automating the Flow (Because Daily Data Is Best Data)

Want this flow to run daily? No problem. Add a schedule decorator:

@schedule(daily=True)
@poetry
class MyFlow(FlowSpec):
...

Then deploy it with Argo Workflows:

poetry run python my_project/flow1.py argo-workflows create

Result: The DREAM Stack in Action

By combining the best tools, we created a scalable, maintainable, and automated system for identifying high-demand zones. This helps the business make decisions such as buying real-estate for future cloud kitchens a breeze! In addition to this, we are also on the cutting edge of data processing tools here. The tools we have picked are less than 5 years old, and are being actively developed,

Wrapping it all up

In our journey to revamp our ML infrastructure, we've learned a lot about balancing flexibility, usability, and performance. The result? A highly efficient and user-friendly environment for our data scientists and engineers.

Our new setup, the DREAM stack, includes:

• Daft: For data manipulation and calculations

• Ray: For scalability

• Poetry: For dependency management

• Argo: For workflow automation

• Metaflow: For a superior developer experience

Together, they make data engineering simple, effective and easy-to-use.

The impact has been immediate: our new data scientists can now spin up projects in a matter of hours, a process that previously took weeks. This increased productivity is a testament to the effectiveness of the DREAM stack.

Thank you for joining us on this journey. We hope our insights and experiences help you in building an effective ML infrastructure that doesn't suck. We’re going to keep pushing and making this even better. If you’re passionate about working on this, message me on LinkedIn or look at our positions. We are always looking for people to help us push the boundaries.

Written by Jordan Hurwitz and Henning Rohde, members of the engineering teams that work on infrastructure.

At Atoms, our software systems interact with humans in kitchens around the world in real time. From robotic conveyance to reliable messaging infrastructure, these systems need to “just work” to not impede food order fulfillment – yet, at scale, it is traditionally difficult to design and build them well enough to satisfy our latency, correctness and reliability requirements.

There are no simple cookie cutter options. On one hand, stateless designs that rely on databases tend to be too slow. On the other hand, stateful designs that coordinate using Etcd, Redis or internal consensus protocols do achieve low latency, but tend to be too complex and ultimately incorrect or unreliable. Older versions of our systems ran into these conundrums.

To overcome them, we built an opinionated sharding service, Splitter, aimed at exclusive in-memory ownership with dynamic explicit assignments, load balancing, and client-controlled routing logic. Today, our most critical, high-performance kitchen systems rely on Splitter under the hood.

The Stateless Model

The majority of distributed services are designed to be request-driven – i.e., actions are taken only in response to a request – and stateless – i.e., service instances hold no state beyond the request handling. Persistent state is stored externally, typically in a database, and written and fetched on demand. More requests to the service are directly reflected in more requests to the database.

There are a number of perceived advantages to this model:

• Scalability. Horizontal scalability can be nominally achieved simply by deploying more instances of the service. However, external dependencies must have their own scalability story and databases often become a bottleneck without careful capacity planning and expertly-crafted database queries and indices.

• Simplicity. Service implementation complexity starts lower, since the service limits itself to request handling. However, as requirements on the service increase, they become harder and more expensive to implement, paradoxically ending in an overall more complex system. Low latency and consistency become hard to achieve. How do you rate-limit specific resources? How do you invalidate caches correctly? How do you design performant push-based apis? 

Stateless services often end up depending on a battery of external services to overcome these challenges. Caches are used to reduce database round-trips. Timer services provide request triggers. Message queues are used to chain together multiple stateless services. But each dependency adds failure modes, latency, and inevitably introduces its own problems.

The Sharded Model

As an alternative, services can be designed with internal coordination and statefulness instead of necessarily relying on external services. It opens up both powerful opportunities and an abundance of pitfalls. At the heart of its difficulty is horizontal scalability.

The sharded model scales by breaking up ownership of the work domain (e.g. users, orders, stores) into shards and dividing it among service instances. Each shard owner holds exclusive control over certain actions, depending on the use case. For example, a service can use shard ownership to maintain an authoritative in-memory cache backed by the usual database. By making all read and write requests go through the shard owner, it can return cached values from memory with a global read-after-write consistency guarantee – in contrast to the stateless design above. 

The sharded model provides low-latency and correctness at scale when done well.

But what about reliability and complexity?

This is indeed where one easily runs into trouble. In addition to a partitionable work domain, sharded services implicitly also need:

1. Ownership. A mechanism for assigning and distributing shard ownership; and

2. Routing. A mechanism for routing requests to current shard owners.

Two approaches are common: either roll your own internal work distribution mechanism, using a gossip or consensus algorithm or a persistence-based sharding library; or rely on a generalized data store for coordination that implements consensus under the hood like Etcd or Zookeeper. Both approaches require significant commitment and tend to get complex fast. Advanced concerns such as re-sharding, multi-region domains, or graceful handover are often neglected.

Stateless services are widespread for good reasons. At Atoms, we took a different approach.

The Splitter Model

Splitter is a highly-resilient multi-region control-plane service for distributing work to connected clients. Splitter makes a number of opinionated tradeoffs aimed at reliability and ergonomics for sharding our real-time kitchen services. 

First of all, it targets reality in a modern cloud environment:

• Dynamic: service instances are ephemeral. They generally do not use local physical disks and may come and go as the service scales up or down. There should be no special logic needed for auto-scaling nor regional failure. Shards should be load-balanced across whatever instances are connected and shard re-assignments might be mildly disruptive at worst, but fundamentally benign.

• Multi-region: services are deployed across multiple regions for geo-resiliency. Shard assignments should align well with underlying databases, where we know or control the data placement for region-local latency. Shard placement decisions should not be the concern of the service and operational controls around data location should be available.

The key needs of the sharded model are handled as follows:

1. Ownership. Domain and shard management is configured and handled centrally in Splitter, which internally uses Raft for distributed storage and coordination. At runtime, service instances establish a connection to Splitter to start receiving leased shard assignments. If an instance crashes or loses connectivity for too long, its lease lapses and its shards are assigned to other connected instances.

2. Routing. Connected instances receive up-to-date shard routing information, i.e., which instance owns what, that can be used for internal forwarding and fanout. This “routing logic as a library” approach is essential: it works for both batch and streaming applications, supports non-1:1 routing and leaves the service in control of the data path.

Splitter deliberately supports only three kinds of work domains: UNIT (singleton), GLOBAL (UUIDs) and REGIONAL (UUIDs with region). A singleton domain is used for leader election in certain low-scale and advanced use cases. A non-singleton shard is a half-open UUID range for a domain. This seemingly peculiar choice provides a number of advantages over discrete or custom shard spaces (e.g. integers or strings). UUIDs force a uniform distribution that naturally avoids hotspots and shards can be evenly split or merged dynamically. Since services in practice never know their eventual scale upfront as the business evolves, no-downtime re-sharding is a crucial feature. Most entity identifiers are UUIDs as well.

Finally, Splitter is also notable for what it does not do. It does not offer storage, instead assuming services use a database for such needs. It does not offer distributed locks, favoring instead “ownership is locking". It does not offer dynamic shard weighting, due to its complex service interaction. Choice is great when it comes to food, but not always for coordination primitives.

Sharding in Practice

In this post, we’ve alluded to Robotic Conveyance Routing (RCR) and Keyed Event Queue (KEQ) as examples of real-time applications that rely on Splitter in different ways for internal distributed state and coordination.

For RCR, one of the main challenges is that controlling robots involves high-frequency location sensor telemetry that far exceeds what is practical to persist in a database. RCR uses facility id as a GLOBAL work domain, with each facility handler controlling all robots in it. The facility handler is responsible for both subscribing to telemetry feeds for its related robots and communicating tasks to the connected robots. It is critical that exactly one connection is made per robot to the vendor broker. Robot telemetry largely resides in memory for real-time control, unlike persisted task queues and robot metadata.

KEQ is a more advanced example. KEQ is a multi-region message broker that stores messages across millions of queues in a multi-region distributed database for resiliency. A key challenge is that such a database incurs high transaction latencies, which is not conducive for real-time message delivery.

For each topic, KEQ uses a Splitter domain to maintain consumer progress in-memory and a small authoritative write-through cache for enqueued messages. New messages are briefly cached and then evicted once all consumers have processed them, usually within seconds. In steady state, KEQ does not read messages from the database at all, even as consumers are slightly out of sync with each other. This setup is eminently effective for low-latency message delivery while respecting per-queue ordering requirements.

KEQ also uses a UNIT domain to elect a leader for assigning and load-balancing subscribers to specific ranges. The leader uses the Splitter routing information to match subscribers with appropriate topic ranges and is an example of non-trivial routing made possible by design.

Conclusion

The Work Distribution Service is an opinionated control-plane service for shard management. Services integrated with Splitter can take advantage of powerful built-in primitives for shard ownership and routing to satisfy strict latency, correctness and reliability requirements.

Splitter was built to enable real-time scalable kitchen systems that go beyond what the stateless model can achieve. It succeeded. Our most critical, high-performance services rely on Splitter as a secret ingredient and are simpler and more reliable for it.

Written by Alexey Pavlenko and Pasha Yakimovich, members of the engineering teams that work on storage infrastructure.

At Atoms, we reviewed all database-related outages over the last few years and found that roughly 80% were caused by schema management issues. Our analysis revealed several contributing factors:

1. Wrong assumptions about the current state of the target database. For example, a proposed database schema may not match the one subsequently generated by migration software (e.g. Flyway) via incremental updates. This could happen if the production schema was manually modified ad-hoc at some point (unknown to the same migration software).

2. Wrong assumptions about how the schema migration is executed. Various database technologies have their own gotcha moments (CockroachDB, for example).

3. Not knowing nuances of how a particular database implements a schema. As in the previous point, there are plenty of opportunities to make mistakes.

4. Forgetting the consuming application’s usage patterns. A classic example is deleting a column or index that is still in use.

Let’s walk through a few case studies. All of them were taken from real production outages (simplified for readability). They mostly apply to Postgres and CockroachDB, but are still relevant to other relational database technologies.

Example 1: Broken Uniqueness Constraint

Say we have a table bad_1 defined as follows. What could go wrong?

The problem is that the c1 and c2 columns can both be NULL, as they lack a NOT NULL declaration. Consequently, the UNIQUE constraint won’t be enforced on the (c1, c2)pair if either is NULL. It’s worth remembering that NULL in SQL is not a value, but rather the absence of a value. Therefore, comparisons with NULL always produce false, which allows the columns to keep identical values despite the constraint (multiple rows could contain c1=123, c2=NULL). Applications that rely on the constraint must also ensure that no null values are inserted into the table.

This behavior can be altered via NULLS NOT DISTINCT in Postgres, but one has to know about this limitation in advance.

Example 2: Silently Dropping An Index

Let’s examine a schema migration. Say we have a production database defined as follows.

Seeing that column c1 is no longer being used by the application, a developer submits a schema update.

The unexpected consequence of the ALTER statement above is that the c2_c1_index will be dropped as well. After all, two columns are required to populate index entry – without c1, it’s no longer possible. Both CockroachDB and Postgres drop the c2_c1_index silently, which may negatively impact the queries that rely on this index (to quickly filter by c2).

The situation wouldn't be as bad if c2 were dropped instead of c1, because an index is normally associated with the first column in its definition. Therefore, we shouldn’t prevent indices from being dropped if its first column (c2 in this case) is dropped. Well, unless the latter column is in use by the application.

Example 3: Keeping Two Tables In Sync

Here’s an example where the schema definition could result in suboptimal behavior that may only reveal itself under sizable load.

ON UPDATE CASCADE instructs the DBMS (CockroachDB, Postgres) to keep the value of fk in sync with the pk value of the specified table. An update of other_table.pk will automatically change bad_3.fk. We can imagine that the following statement is executed in the same transaction that changes other_table.pk.

This may work fine when our data volume is low or when such requests are infrequent. However, upon a spike of changes in other_table.pk, the DBMS will have to change bad_3 values respectively. And, due to the absence of an index for fk, it will have to run a full scan for every one of them. Which will lead to rapid performance degradation – possibly an outage if the table contains a large number of rows.

Example 4: Potential Runtime Failure

And finally, a simple one.

Is it safe to execute a drop of a table, column, or index? It’s not possible to answer without knowing the context. There can still be references in the application code, or even some 3rd-party system that generates monthly reports. We should use the history of queries to some_table to know with a certain degree of confidence.

Nuance: A Better Schema Analysis System

We should detect the issues discussed above, and more, as part of our everyday development flow. This would maintain developer velocity, ensure that past outages are not repeated, and eliminate the need for manual intervention by infrastructure maintainers. We built Nuance: a schema analysis system to accomplish exactly this!

The idea to analyze and lint schemas is not new. There are a few open-source products that attack this problem. However, they focus primarily on syntax, while we aim to address issues related to the runtime usage of the database. Usability depends on the signal-to-noise ratio – ideally, we want to confidently highlight actual problems rather than focus on indentation and naming conventions. This requires a broader perspective, as the database schema alone is not sufficient to identify problems.

A high-level diagram of the analysis module is presented below.

At this point, it’s worth noting multiple data sources that are used to produce facts about both the database schema and the execution environment.

Production Database Schema & Migration Library Metadata

Since any schema upgrade will eventually be applied to a running database, it doesn’t make sense to consider any schema other than the one that is actually deployed to production. It is always possible that the schema committed to the company’s version control system diverges from the actual one, whether due to early experimentation or ad-hoc mitigations. This should be taken into account when assessing whether the upgrade is safe.

For example, Flyway, a commonly used migration tool, keeps the previously applied versions in a special table (flyway_schema_history) assumed to be in sync with the updates committed to a version control system. Nuance can validate this invariant and warn when discovering drift. CockroachDB and Postgres, like most database technologies, easily allow schema dumping, which is then parsed, transformed to AST, and used to derive necessary facts.

Schema Upgrade Statement

These are the actual SQL commands executed against a production database to migrate it to the target schema. Since we store schemas as code, these can be extracted from the developer’s pull request.

Depending on the chosen upgrade path, this can either be a single file with statements, or a list of newly introduced versioned files with incremental updates. Our implementation differentiates these approaches in order to support catching issues where migration tooling depends on whether a change is performed via a single file or multiple.

Database Runtime Information & Query Log

Modern storage technologies expose a lot of information via metrics and system tables. A few examples of the latter are cumulative statistics in Postgres and crdb_internal in CockroachDB.

Additionally, a full log of queries executed against the target database is also extremely powerful. This enables us to assess the ongoing usage of certain entities, whether they be tables or columns. This can also be applied to other use cases, such as access audits and performance monitoring.

We use a system built on top of ClickHouse to store, process, and aggregate both runtime information and the query log of the entire storage fleet. This provides us with a summary of database usage for a given time period. (We also use the same system to monitor and debug performance issues, detect suboptimal transactions, and submit recommendations on resource tuning.)

Other Data Sources

We plan to eventually add more data sources. For example, consider the impact of a schema upgrade on derived systems such as database changefeeds, Kafka topics (plus other message queue technologies), various OLAP consumers, and more. Theoretically, we could fetch this information and use it in the analysis to determine whether there could be a failure further down the pipeline.

It’s clear at this point that a well-designed system allows for the easy integration of new data sources with little to no impact on existing functionality. Such a design requires a certain level of abstraction over the utilized data sources. More on that below.

Datalog and Fact Store

Instead of delivering data in an aggregated state specific to the source (e.g., a full database schema snapshot as SQL or AST), we can split it into individual facts (e.g., table X has column Y, column Y is of type INT, column Y is NULL-able, index IDX was used N days ago, etc.). This becomes a "database" of facts that can be queried uniformly.

With a database of facts, the next step is to implement rules that extract interesting properties from it (problems or recommendations) – optionally unwinding multi-level relationships between individual facts. 

This approach is a natural fit for logical languages, particularly Prolog and Datalog. We use the latter and rely on the Datalog implementation by Google Mangle project. 

During validation, information extracted from the data sources above is converted into facts (atoms) that are stored in an in-memory fact store. Individual rules query this to check for the existence of interesting properties. Successful rule evaluations, depending on the purpose of the rule, indicate either a potential issue or an optimization opportunity. All atoms in the store are unique; therefore, by using atom terms (arguments), it’s possible to point to the specific change and improve developer awareness.

Putting It All Together

Let’s see how it’s done in practice. Below is a Datalog source code of a rule that checks that no nullable column is a part of the UNIQUE constraint (Example 1). In our codebase it’s associated with a Code that uniquely identifies the rule itself, warn-level Severity (not good, but not necessarily outage-inducing), Unit to define the top-level rule and accompanying sub-rules (presented below), and the Predicate to indicate the name of the top-level rule (uniq0001).

The analysis engine will evaluate the top-level rule, and if it produces results, then there’s indeed an issue. We can generate its human-readable description by using the accompanying template that receives terms (arguments) of result atoms that uniq0001 produces. This description will point to the exact entity that has a problem (Table and Column).

Let’s try to solve Example 2. The rule below checks for situations when an ALTER TABLE DROP COLUMN statement tries to drop a column that is secondary to some index. “Secondary” here means that the column is either not the first one in the declaration list or is a column that’s only stored in an index for faster retrieval (e.g. a covering index for index-only scans).

To make things more complicated, imagine a chain of tables that point to one another via foreign keys (official reference). A CASCADE drop of the first column in a chain may trigger respective drops in all linked tables, which at some point may lead to exactly the same issue (unwanted index drop). A more sophisticated rule (only a part of it is presented here for brevity) may check for that as well. (This is also applicable to Example 3.)

Finally, having table / column / index usage information from recent queries (transformed into respective facts) enables us to trivially solve Example 4. All we need to do is check whether the table has been accessed within the last N days (where N is sufficiently large).

As we can see, Datalog is pretty expressive. It has become evident during the implementation that any generic approach based on more common programming languages (e.g., Golang) would need to implement some portion of the Datalog engine anyway – so we’re happy with our choice!

Developer Experience

Nuance appears to developers as just another lint check triggered by a new or updated pull request. If there are no issues, the pull request can be merged as usual. Otherwise, we block it and present resolution opportunities. 

Not all detected issues require immediate action. Sometimes, users are confident in their changes, or the detected issues are merely informative. To accommodate these scenarios, we left an escape hatch enabling users to acknowledge and skip the detected issues. 

Command Line Tooling

We’ve augmented Nuance with two important CLI commands.

The validate command enables developers to submit their SQL scripts for analysis and issue detection. Behind the scenes, this command spins up the same validation process that runs for pull requests. This includes fetching the production or staging schema and retrieving real-time query data, ensuring a comprehensive analysis.

The inspect command inspects the live database schema for any existing issues. 

Both CLI commands make it easy to experiment, analyze, and detect issues before developers even submit pull requests.

Documentation

When presenting an issue, along with a brief description, we always include a link to more detailed documentation. The issue page typically contains a longer description, examples, extracts from official documentation, code snippets, and links to past outages and their associated postmortems. This is critical for the smooth adoption of Nuance. By educating developers and improving their awareness, we teach them to trust the tool, preventing it from being viewed as a noisy distraction.

Results

At CloudKitchens, Nuance has been running in production for almost a year. In its current state, it has accommodated past outage history and checks for common anti-patterns and flaws described in Postgres (e.g., here) and CockroachDB documentation.

Current statistics show that 56% of our pull requests with schema changes initially contain issues, 23% of which are critical (guaranteed to cause outages, either immediately or later on). When developers become aware of the issues through tooling, we no longer see outages repeat themselves. As our infrastructure evolves, we will continuously improve and augment our ruleset.

Introduction

In the modern age, orders are placed from multiple channels – delivery apps, online ordering storefronts, in-store, and more – so having a best-in-class digital menu is equally as critical as a restaurant’s physical storefront. We build technology that allows restaurants to easily create and manage their menus in both the physical and digital space, enabling them to succeed across all channels.

Menus are easy

The concept of a menu is simple. It is a catalog of food the restaurant can make and its associated prices.

…But menus aren’t so easy

There’s more to a menu than meets the eye. Let’s look at a basic digital menu designed to mirror a restaurant’s physical menu.

Much like their physical counterparts, menus are expected to have some structure.

Generally, this means menus have different categories of items they can sell. “Entrees”, “Sides”, and “Drinks”, with items organized under those categories. Some restaurants may offer add-ons on specific items, like adding a side of fries to a burger. Some of these add-ons may even have their own add-ons, such as adding melted cheese to the side of fries. Some restaurants may even have different menus they sell depending on the time of day – a lunch menu and a dinner menu, for example.

How CloudKitchens helps restaurants create user-friendly menus across channels

When we learned that menus aren’t easy, we began building solutions to make setting up and managing menus across multiple channels a better experience for the restaurateur. Since a good menu requires categories, add-ons, add-ons to the add-ons, and perhaps even a different menu depending on the time of day, we developed a primary set of four entity types that we store in a tree-like structure to model these menus: Menus, categories, items, and modifier groups.

Separating these entities allows us to store specific information for them. For example, items will often have photos and prices. Menus will have the hours they are sold. Modifier groups may restrict the number of underlying modifier items that may be selected. 

In our model, we allow entities to be re-used without creating duplicates. This lets us reduce the workload for restaurateurs managing a menu. Information about an entity must only be updated once and will appear everywhere the entity is referenced.

The CloudKitchens digital menu solution in action 

Let’s look at an example. Say we’re making a menu for a fast-food burger chain: Burger Prince. Many items can add sauces (burgers, fries, onion rings, etc), but they are all the same. Rather than specifying every possible sauce for every item, we can create a single modifier group titled “Add a sauce” with all the sauces underneath and reference the same modifier group several times. If we would like to introduce a new sauce, we can just add it once to the “Add a sauce” modifier group, and it will be available on every item with the modifier group attached.

While this helps in most cases, it also adds some complexity if entities should have different properties depending on where they appear on a menu. This is most common when it comes to pricing.

Let’s look at another example from our existing burger restaurant, Burger Prince. “French Fries” can be ordered as a stand-alone item on the menu or as an added-on for a burger combo meal. When ordered as a stand-alone item, French Fries should cost $2, but when ordered as part of a combo, they go down to $1.

To support this, we built menu-path-based pricing for the “French Fries” item, so we can set different prices depending on whether it’s ordered as part of the modifier group for a burger combo or from the category of the menu.

Menus are hard 

Now that we’ve examined how we model menus internally let’s add another layer of complexity.

Adapting menus across different channels

While we offer an online, direct-to-consumer storefront, the menus created by our technology will likely also appear on external delivery channels (e.g., Uber Eats, Doordash, Grubhub) so the restaurant can take orders through those channels.  

Because every delivery channel models their menus differently and has different information they store per entity, we must be able to adapt our internal menu to theirs and vice versa. This requires our menu model to be the superset of all information available on these different channels while still not overcomplicating things for ​​restaurateurs.

Each delivery channel may support different features compared to our internal menu. For example, some channels may not support nested modifiers (add-ons to add-ons), meaning we’ll have to copy and automatically unnest modifiers when we convert our menu data to the external service's format. Some channels may not support the same item appearing in a category and as an add-on to another item. For those cases, we have to split the item into two copies on the external menu, which link back to the same item on our internal menu.

We have a complex integration suite for adapting internal menus to external menus, which we’ll discuss in detail in a future post.

Managing menus across different channels

Since delivery channels often charge restaurants a commission on every order, a restaurateur may decide to alter their menu pricing depending on the ordering channel. This means we must now model differences in prices across delivery channels and prices based on where an item appears on a menu (e.g., a standalone order of fries or fries added onto a combo). 

One of the core features we offer customers is inventory management – the ability to mark items as out of stock when they are sold out or mark items as back in stock when they do have inventory. This allows restaurants to quickly toggle which items are being sold without logging in to each channel and editing every menu during day-to-day operations. 

For this feature to be effective, once a user marks an item as available or unavailable on our internal menu, we must quickly and accurately propagate this information to all connected delivery channels. Hence, they mark the equivalent items on their channels. We do this by storing links between our internal entities and every external entity at a published channel so our system can propagate the availability update for the correct item.

We have a complex integration suite to adapt internal menus to external menus, which is a deep rabbit hole, so we’ll save those details for a future post.

Menus at scale are really hard

But let’s take it a step even further. Many of our customers aren’t just operating on multiple channels but also at multiple locations. Menus are usually the same across these locations, so we do not need to manage a copy of every menu.

Let’s say Burger Prince had 50 locations. If we wanted to start selling a new seasonal item at each location, we’d have to go to every single location’s menu and make the same change 50 times. That would be a lot of work!

Meet template menus

To alleviate the issue above, we created template menus. Template menus are meant to model a digital menu that can be used at multiple locations, so restaurants managing a menu sold at multiple locations can make all their changes once on a single template menu.

While locations sharing a template menu are more likely than not to share information, many aspects of the menu can be customized per location. This means that sources of complexity, like per-channel pricing, are potentially multiplied by the number of locations using a template menu.

Menu differences across multiple locations 

By default, a single price may be set for an item on a template menu, and that same price will propagate to all locations. However, menu items are often priced differently for different locations, so we have to manage price per location, price per delivery channel, and price based on where the item appears.

A restaurateur managing a template menu may want to mark an item as unavailable across all locations. Also, it’s common for a restaurant to run out of stock of an item for the day at just a few locations, so the item should only be marked unavailable at a subset of locations.

This means we must store a link from each item on the template menu to every item across all delivery channels for each location so we can manage its availability at any number of locations with a single button click. 

Menus can also be generally different per location. Some items or categories may be sold at some locations and not others. For example, global fast-food franchises usually sell a few specialty items based on the region in which the restaurant is operating.

Menus are still just one part of the larger picture

We’ve explored some of the complexities menus may present, but we must remember that menus are only one part of the restaurant ecosystem.

In addition to creating an easy-to-use menu management experience, our menu system must integrate with many other systems, such as online order management, POS (point of sale), KDS (kitchen display systems), printers, business intelligence reporting, analytics, promotions, and inventory/supply chain.

Here’s an example of a simplified order interaction involving all these pieces:

At a high level, when an order comes in from a third-party channel like DoorDash, Uber Eats, or Grubhub, our system must be able to properly match the items in the order to items on the internal menu. These orders must then be converted to a kitchen-readable format and sent to the KDS (kitchen display systems) or printers for chefs to prepare. At the same time, if the storefront has a physical point-of-sale device for taking orders, we must also inject any online orders into their point-of-sale device so they can manage their entire restaurant from one channel. On top of that, other integrated apps like business intelligence or inventory management must also be able to understand what items are being ordered and consumed so we can properly decrease inventory count for our users.

Each of these interactions is very complex, so stay tuned for the next few posts, where we’ll explore the details of some of these challenges.

Written by Grzegorz Głąb and Nibir Bora, members of the engineering teams that work on core infrastructure.

Many organizations opt for managed Kubernetes distributions like Azure Kubernetes Service (AKS) to get up and running quickly without needing a large engineering team to operate Kubernetes clusters. This is a core design principle of the Core Infrastructure team at City Storage Systems. However, over the years, we’ve learned that the true operating cost of managed Kubernetes distributions is not in fact zero.

Even public cloud experiences occasional failures. Hardware faults, kernel misconfigurations, network bottlenecks, problematic rollouts, resource scarcity, security vulnerability, etc. leads to complications lasting for minutes, or in some cases, weeks. In this blog we share our experience illustrating how minor glitches, if left unattended, could quickly escalate and impact business continuity.

Rather than engaging in constant firefighting we designed a self-healing framework, often implementing automations with a turnaround time of as little as 1 day. These automations were sometimes temporary fixes until resolved by the cloud provider, and at other times, they became permanent enhancements to our platform’s reliability. While our journey began with a focus on AKS, this framework is a general-purpose pattern to improve resilience of any Kubernetes platform.

The Self-Healing Framework

The first self-healing use case was implemented as a monolithic program. But, as we added new use cases, we identified several reusable libraries that nudged us to organize it into a framework. The framework today consists of Automations that each address a specific failure mode. Automations are implemented as an independent Detector and a Fixer, which are either a controller or a Go program.

Detectors are responsible for collecting signals and flagging failure conditions. There are two types of detectors - Cluster level (Deployment) and Node level (DaemonSet). Cluster level detectors monitor for cluster-wide failure events and have permissions to watch or create API server resources. Node level detectors monitor for node level failures (e.g. misconfigured OS flags, image pull issues, missing systemd services, etc.) and have privileged host access.

Fixers complement detectors by executing remediation steps to rectify or cleanup failure states. Similar to Detectors, there are two types of fixers - Cluster level (Deployment) and Node level (DaemonSet). Cluster level fixers execute remediation actions that operate on cluster-level resources and have permissions to watch API server resources. Node level fixers execute remediation action at node level, or operations that require privileged host access.

Generalizing as such allows us to keep the framework simple and appropriately isolate permissions. This was key to swiftly adding new automations when needed. Whenever we identify a new degradation, we implement and deploy the corresponding detector and fixer across all our clusters. The following automations are a few examples that shielded our internal developers and applications from potential impact, and also significantly reduced our team’s support toil - cutting it in half from 30% of engineering time.

Building these automations over the last year and a half has taught us some key lessons:

1. Kubernetes is not the end product. It is a framework for building platforms. Managed Kubernetes still greatly benefits from business specific customizations that create leverage for developers.

2. Abstractions don’t erase underlying layers. Kubernetes, while powerful, still requires us to dive into the host VM or kernel layer for deep debugging - a skill set we've honed through experience.

3. Cost optimization has a reliability tax. Optimizing cost needs to be counterbalanced with increased vigilance on reliability. For example, running stateful workloads on Spot nodes required us to invest further in automation.

4. Cloud bugs cannot be predicted. Instead of anticipating imaginary failure scenarios, it’s better to optimize for speed of diagnosing unforeseen issues and implementing automation for them. For example, we consolidated all node malfunction signals on a single “node inspector” dashboard empowering our developers to respond swiftly when paged.

In the following sections we describe a few of the automations in detail, covering how each failure mode was identified and how we automated its self-healing.

Handling Abrupt Spot Node Preemptions

We use Spot nodes extensively on our Kubernetes platform to optimize resource costs, running both stateless and less critical stateful workloads. However, Spot nodes on AKS lack any SLA, which can lead to potential abrupt preemptions. We experienced an incident where a large number of Spot node preemptions caused multiple stateful workloads to fail, causing cascading application failures resulting in downtime.

When Spot nodes on AKS are preempted, a scheduled preemption event is emitted 30 seconds before the underlying VM is abruptly removed. The node isn’t cordoned, workloads aren’t gracefully shut down, and the Node isn’t deregistered from the Kubernetes API server. The Node object remains without a physical VM (see issue #3528) until cleaned up after 5 minutes due to failed heartbeats. When this happens, stateless workload pods (controlled by Deployment and ReplicaSet) are automatically rescheduled, but not StatefulSet pods. StatefulSet pods leave behind “phantom” pod objects (with .status.phase: Unknown) in the API server, which is not an acceptable behavior for our stateful workloads.

To address this, we implemented a self-healing automation that intercepts Spot node preemption signals and gracefully evicts all pods on the affected node. A detector watches for VMEventScheduled node conditions (example below) and creates a SpotNodePreemption Custom Resource (CR) with details for the fixer. The fixer then evicts the pods with a 10-second grace period.

.status.conditions: [
    {
        status: "True",
        type: "VMEventScheduled",
        reason: "VMEventScheduled",
        message: "Preempt Scheduled : Tue, 14 May 2024 12:57:00 GMT",
        lastHeartbeatTime: "2024-05-14T12:56:43Z",
        lastTransitionTime: "2024-05-14T12:56:42Z"
    }
]

Once this automation was operationalized, we noticed that some Spot nodes were still terminated without a scheduled preemption event. This was because when Node Problem Detector (NPD) queries Azure Metadata Service for the VMEventSchedule event, the request occasionally fails resulting in a NoVMEventScheduled node condition (example below). To handle this, we added another self-healing automation to clean up after terminated Spot nodes when the preemption event wasn’t intercepted. The detector creates a SpotNodeDeletion CR when a Spot Node object is deleted from the API server, and the fixer force deletes all pod objects on that node assuming they are no longer reachable.

.status.conditions: [
    {
        ...
        type: "Unknown",
        reason: "NoVMEventScheduled",
        message: “Timeout when running plugin \"/etc/node-problem-detector.d/plugin/check_scheduledevent_consolidated.sh\": state - signal: killed. output - \"\"”
    }
]

Handling StatefulSet pods on Unreachable Nodes

AKS node pools are built on Azure Virtual Machine Scale Sets (VMSS) infrastructure. We observed that VM failures in the VMSS layer often make AKS Nodes unreachable. When this happens, the node controller adds a NoExecute taint, and all pods on the node are evicted after 5 minutes. While stateless pods are rescheduled automatically, StatefulSet pods are not (see issue #54368, and design proposal). This can lead to data loss caused by under-replication in stateful workloads like CockroachDB or OpenSearch.

To address this, we implemented a self-healing automation that watches the Kubernetes API server for Node objects with a node.kubernetes.io/unreachable taint. The detector filters nodes tainted for more than 5 minutes, and the fixer force deletes all pods (assuming they are unrecoverable) on these nodes, allowing new pods to be scheduled.

Cleaning Up Succeeded and Evicted Pods

While investigating a cluster health degradation due to increased etcd disk size, we identified the accumulation of Succeeded pods as a significant factor. These were created by short lived cron jobs, pods without a controller (e.g. Flink jobs), and evicted pods. Since kube-controller-manager doesn't automatically clean up succeeded pods, this is a problem on our large multi-tenant clusters. This default behavior can be modified by configuring the --terminated-pod-gc-threshold flag. However, since we use managed Kubernetes the control plane is managed by the cloud provider and not user-configurable.

To address this, we implemented a self-healing automation that monitors the Kubernetes API server for pods with, either status.phase = Succeeded, or status.phase = Failed with pod.Status.Reason = Evicted. The detector flags pods that have remained in these phases for at least 15 minutes. This threshold is configurable per namespace. The corresponding fixer deletes these flagged pods from the API server.

Handling Network Packet Drops Due to Unbalanced IRQ

We noticed increased packet drop rates in network IO-intensive workloads, initially thought to be application errors. However, we saw that the nodes with affected workloads had VMFreezeEvents (see AKS docs). Investigation showed hardware interrupts from the node’s network interface were unevenly handled by only 2 of 8 CPU cores, causing 100% utilization on those cores (see detailed investigation in blog). Restarting the irqbalance service, which should distribute interrupts evenly, resolved the issue.

To address this, we implemented a self-healing automation that flags nodes where fewer than half of the CPU cores are configured to handle interrupts from the network interface. This is done by checking /proc/irq/IRQ#/smp_affinity, which denotes CPU core affinity to the interrupt request queue (IRQ). The corresponding fixer restarts the irqbalance systemd service on the host VM. We also expose the number of cores used for IRQ per node as a metric for continued observability. The upstream issue was later fixed in later versions of ubuntu (see bug #2038573).

Despite this fix, some packet drops persisted. This was traced to a backlog in the network interface's receive queue. We found if the receive queue size was set to less than 10000 it caused packet drops. To address this, we implemented another automation that flags nodes where net.core.netdev_max_backlog is less than 10000. The corresponding fixer, resets it to 10000 on the host VM.

Addressing Failing nftables During OS Image Migration

While migrating our nodes from Ubuntu to Azure Linux OS, we noticed nftables wasn’t running on the migrated nodes. Kubernetes relies on nftables on the host VM for inter-pod routing rules on the node and egress traffic. This prevented Network Policies from being applied correctly, leading to irregular network failure on nodes. After investigation, we identified this was due to a missing newline in the nftables.conf file (see issues #4144 and #7301, and pull request #8310).

To address this, we implemented a self-healing automation that flags nodes where the host VM isn’t running nftables. The corresponding fixer corrects the nftables.conf file by appending a newline to the end and restarts the nftables systemd service.

Addressing node-problem-detector Missing on Nodes

AKS runs node-problem-detector (NPD) to monitor node health and flag for removal during malfunction. It runs 10 checks every 30 seconds and injects the output into node conditions. We integrated these conditions into our observability stack. During a workload failure investigation, we noticed a node had only 4 status conditions instead of the usual 14 (10 from NPD and 4 from kubelet). This led us to discover NPD wasn’t running on the node. The workload failed because Container Runtime Interface (CRI) malfunctioned on the node preventing kubelet from verifying workload status.

We implemented a self-healing detector that flags nodes where NPD isn’t running. Further analysis revealed 25% of our nodes had this issue. Automatically terminating these nodes was deemed too risky. Instead, we rolled back the node OS on all of our nodes to a previously working version and escalated the issue to the cloud provider (see issue #3988). This was later attributed to an upstream CVE that was fixed. We also set up automated alerts for nodes without NPD to prevent future issues.

Mitigating ImagePullBackOff Errors for Large Container Images

We faced a surge in ImagePullBackOff errors for workloads with large container images (7-10GB). The kubelet error messages (example below) were unhelpful, and workloads failed to start for hours. Manual eviction sometimes helped after multiple retries. An unrelated experiment benchmarking write speeds on Azure Managed OS disk and Ephemeral OS disk, led us to identify that the issues occurred exclusively on nodes with Managed OS disks.

Oct 31 11:57:43 aks-nodepool0392-17898922-vmss0000LX kubelet[2874]: E1031 11:57:43.120279    2874 remote_image.go:242] "PullImage from image service failed" err="rpc error: code = Canceled desc = failed to pull and unpack image \"cssacrprod.azurecr.io/chronorepo-companion-cron:efdc4a316aebcc878c38483b09bb939524dbd94a\": failed to commit snapshot extract-332345855-2JgL sha256:d2bd2b7dd52900b17c2e8d2f50d94273892a45d96a760f078aeb58bc54fbc160: context canceled" image="cssacrprod.azurecr.io/chronorepo-companion-cron:efdc4a316aebcc878c38483b09bb939524dbd94a"

We implemented a self-healing detector that flags nodes with ImagePullBackOff errors by parsing kubelet logs. Currently, we lack an automatic fixer. Instead, we emit a custom warning event for each affected pod. Affected workloads can either retry, or if the issue persists, set a node affinity for label ephemeral-storage = true. All nodes in our platform with Ephemeral OS disks have this label.

Conclusion

Building out a self-healing solution for Kubernetes has allowed us to enhance the reliability of our Kubernetes platform without burdening ourselves with operational and support toil. Automation proved to be the right principle for us to scale to 100s of clusters.

So, what’s next? We are constantly adding new detectors and fixers to our self-healing framework. Low level networking, noisy neighbor problems, CPU core use optimizations are few examples of areas we are actively investigating how to automatically detect and rectify problems. Furthermore, we plan on extending the framework beyond platform deficiencies to application deficiencies. We are confident the same mechanics of self-healing are widely applicable. Self-healing is the only answer to having a platform’s maintenance costs scale sublinearly with business growth. So we’re serious about investing in it further.

Written by Henning Rohde and Jordan Hurwitz, members of the engineering teams that work on infrastructure.

At Atoms, real-time food order fulfillment is at the heart of our business. When a customer places an order, an elaborate orchestration workflow is run behind the scenes to ensure the meal is cooked, assembled, and delivered efficiently and seamlessly.

What needs to happen – and when – depends on the specifics of the order and is time-sensitive, typically involving 3rd party integrations with varying characteristics. If an order contains multiple items, each has to be prepared at the right time. If an order is prepared in a facility with conveyance robots, it requires additional coordination. Our internal systems listen to a central order event message bus to understand and react to the state of orders in real time. Lost updates or long delays result in a poor experience for both kitchens and customers.

The nature of order processing presents some unique challenges. To this end, we designed the Keyed Event Queue (KEQ) service as a central message bus for order processing. Today, all orders and order related events flow through the system alongside other internal traffic at Atoms.

Event-driven order processing

Order processing is inherently reactive. Our software systems observe and interact with the physical world, where progress is driven by real-world events, for instance, “customer places an order”, “order ready to pick up”, “driver arrives”, “item removed from locker”, etc. Event-driven systems are commonly structured around an asynchronous, durable message bus, managed by a message broker. 

The Atoms order processing system follows this approach:

Orders transition through a dozen or more events as part of fulfillment, some of which are generated internally by our systems or devices. Each consumer service is responsible for a distinct aspect of fulfillment, such as ticket printing or robot conveyance, and reacts to order updates for that purpose. Order progress is driven by events and actions in concert with each other and the physical world. Many thousands of orders are typically in progress simultaneously.

In our kitchens, timeliness and event ordering are important. We originally used Apache Kafka, a widely-used, open-source message broker that offers total event ordering with high performance. Kafka is a standard solution.

However, as order volume grew, our systems started to run afoul of how Kafka works and what it is designed to do well. First, the distinctly heterogeneous order processing frequently involves 3rd party integrations with at times transient failures or delays. Second, our software stack is stretched across multiple regions to overcome regional cloud provider failures. Neither works well with Kafka or similar message brokers.

Head-of-Line (HOL) blocking and failure handling

With Kafka, the order processing message bus is represented as a topic. A topic acts as a queue where emitted events are delivered to each consumer in the same order. Kafka scales its parallel processing by dividing topics into a fixed number of distributed queues called partitions. Each message provides a key that determines its partition. Event ordering is preserved within partitions. With N partitions, up to N consumer instances can process events concurrently.

Although events are processed concurrently across partitions, each partition still contains the events of thousands of different interleaved orders. And because a partition is processed sequentially (with batching), failure to process any event blocks further progress for the whole partition. This situation is known as Head-of-Line (HOL) blocking.

The big question is: if a consumer fails or is slow to process an event, what do we do?

For time-sensitive processing, there are no good simple options. We can wait for event processing to succeed, but that may take a while, during which all later events in the partition are stalled. Or we can ignore the message and move on, which avoids the delay but may fail to act on something important. Either option leads to poor experiences and order cancellations.

To mitigate HOL blocking, consumers must defer failing or slow events one way or another. A common option is to move problematic events to another topic, a so-called Dead-Letter-Queue (DLQ), for later processing. A DLQ ensures progress without losing events.

However, a DLQ introduces other problems: for example, it breaks event ordering when postponed events are processed later than intended. And for transient failures or delays, when exactly should we give up? If we give up too quickly, then even small disruptions create outsized ordering inversions. If too slowly, then processing is still delayed. Such logic is hard to get right when seconds matter. And how do we handle slow or failing DLQ events?

A DLQ works best when it is immediately apparent that an event is “dead” and has no hope of being processed, such as corrupt or invalid data records. This is indeed the case for a large number of Kafka uses. But for order processing it is not quite so simple.

The reality with HOL blocking is that time-sensitive processing involves complicated failure handling. While using a DLQ is a standard solution, it is a solution for another problem. However, since HOL blocking is a consequence of partition mechanics, what if we designed a system to avoid it?

KEQ: A different message broker

Keyed Event Queue (KEQ) is a new message broker based on a deceptively simple idea to avoid HOL blocking: instead of using N permanent partitions, use a separate, temporary partition for every message key. It is the exact ordering we ultimately want for orders: the events of each order are in sequence, but separate orders can progress independently. HOL blocking is then a non-issue, because the scope of each partition is a single order.

There are additional benefits to this design. Standard techniques for handling transient errors apply, such as retries with exponential backoff. Potentially slow actions can be performed synchronously. Consumer logic – especially failure handling – becomes far simpler. The problems brought by time-sensitive processing and fixed partitions are largely gone.

In addition, there is no N to pick and adjust for the number of partitions as order volume grows. The number of partitions is dynamically determined by the data.

The primary tradeoff with this design is that bookkeeping becomes expensive. Where progress is tracked by a small, fixed number of partition cursors in Kafka for each consumer, KEQ requires millions of such cursors to keep track of the individual progress of each order. Consumers are thereby comparatively heavyweight.

For order processing, neither tradeoff is an issue. Moreover, consumers are almost always caught up and it is desirable that a new event for an order can be sent immediately and processed individually.

Multi-region, low-latency message delivery

KEQ is a multi-region, high-performance and scalable message broker for managing a large number of independent strictly-ordered message queues. It provides an at-least-once ordered delivery guarantee as well as a processing exclusivity guarantee with explicit leases. For consumers, this means that events are delivered in order with no other consumer instances trying to process the same event. This is important when consumer actions involve slow non-idempotent side effects.

KEQ uses an active-active, multi-region distributed SQL database to store messages, cursors, and metadata. That choice makes a tradeoff: it ensures KEQ can honor its guarantees even during a regional cloud provider outage, but I/O latency is necessarily higher than a single-region database. 

For performance, KEQ maintains authoritative, sharded in-memory state for each topic using our coordinator for stateful services Splitter. It allows KEQ to track progress and deliver new messages without reading from the database; order and progress updates can be blind writes. New messages are briefly cached and then evicted once all consumers have processed them, usually within seconds.

KEQ also optimizes its internal communication, a necessity for low-latency multi-region services. Internally, each topic is divided into key ranges with an explicit region. These key ranges are dynamically assigned by the Splitter to KEQ instances in their region, so that database operations may benefit from region-locality.

When a consumer instance connects to KEQ, it is internally assigned to specific ranges and a streaming connection is made to each range owner. New and pending messages are thereafter streamed to the consumer, typically from memory and otherwise from the database if it has fallen too far behind.

In addition to topic ranges, KEQ runs an exclusive global leader responsible for allocating and load-balancing ranges between connected consumers. For instance, if two consumer instances are connected, each is assigned half of the ranges while taking into account which ranges are closest to each consumer. This work allocation is dynamic as instances come and go and factors in both region-affinity and node-affinity to minimize delivery latency.

The centralized dynamic allocation simplifies the operational model. Consumer instances are ephemeral and need no individual configuration or coordination; they simply connect to make themselves available. Work is assigned to whatever consumers are present. Wide consumer failures and fast up-scaling are handled equally naturally.

As a result, KEQ works well with auto-scaled multi-region consumers.

Failure-handling revisited

KEQ is built for reliable real-time order processing in adverse conditions. Its active-active multi-region design offers elastic scalability without manual regional failover, explicit re-partitioning or temporary ordering violations. Order event consumers can freely scale up or down to match the rhythm of the restaurant business.

KEQ’s main value is in simplifying how consumers handle failure, transforming failures into delays. With no HOL blocking, consumers can retry transient failures indefinitely without impacting other orders. A DLQ is not needed. Processing failures – and how they are overcome – is handled inline. Even consumer code bugs can be fixed in a reasonable time, which lets consumers sidestep complex failure-handling and detection in favor of a trivial retry. 

For widespread system failures, HOL blocking does not matter because no processing will succeed. But consumer code, in practice, never knows which kind of failure is happening, yet it must make a real-time decision. KEQ simplifies that decision. And as Dijkstra put it, "Simplicity is prerequisite for reliability".

Written by Xiangyu Sun and Ye Tian, members of the engineering teams that work on Otter Marketing.

Running promotions and ads on Online Ordering Platforms has become an effective marketing tool to increase exposure and drive sales for restaurants. However, making informed decisions about which promotions to deploy, customizing them for different times and platforms, and assessing their performance remains challenging. Relying solely on manual processes for campaign management can be not only labor-intensive but also suboptimal.

We've developed an automated marketing system inside Otter, the all-in-one restaurant operating system. It has helped Otter customers boost their online marketing with an average revenue uplift of 12%. In this blog post, we'll explore the key factors and design decisions that contributed to this success.

Prediction and optimization

At first glance, manually creating a static timetable for marketing campaigns may seem straightforward, but this approach becomes impractical due to escalating complexity. With numerous configurations targeting different audiences, discount levels, and so on, the combinations can easily exceed hundreds. When factoring in variables such as days, variations across different Online Ordering Platforms, and multiple store locations under management, the number of choices multiplies to tens of thousands.

We address this complexity by forecasting the potential outcomes for each option, allowing us to compare alternatives and identify which performs better. Then, we employ linear programming to search for the optimal combination of options, adhering to global constraints like the marketing budget.

Prediction

Historical data offers insights into how business outcomes fluctuate with the timing of marketing campaigns, as well as the impact of geolocation and restaurant type. Using this data, we constructed a suite of time-series regression models to predict key business indicators, such as revenue, order volume, profit, and marketing spend. We experimented with various modeling techniques, ranging from linear models to gradient-boosted trees and neural networks. Ultimately, we chose gradient-boosted trees implemented through LightGBM for its superior performance (see Table 1)  and simplicity.

More importantly than the type of model, we discovered that feature engineering warranted greater effort to improve training metrics. As we aimed to capture the nuances between different stores and promotion types, one challenge was encoding this information within the feature set. Our initial method was to use one-hot encoding, assigning unique IDs to each restaurant and promotion type. However, this approach created high-dimensional sparse feature vectors that not only inflated the model size but also failed to capture semantic meanings. For instance, it did not readily reflect when two stores were in close proximity and served the same cuisine, or that promotions like “Spend $20, save $5 for all” only differed from “Spend $20, save $5 for new customers” in their target audience.

A common approach to address this high dimensional feature space is to project them into low-dimensional dense embedding spaces, allowing the system to reflect relative semantic closeness. However, this technique is generally more compatible with neural networks and demands a considerably larger amount of data and fine-tuning. Therefore, we opted to derive the low-dimensional space directly from the semantic meanings. Rather than employing restaurant IDs, we represented each restaurant by the demographic information of its location and operational characteristics, such as ratings and cuisine type. Likewise, instead of using promotion type IDs, we utilized marketing metadata including the intended audience, minimum spending threshold, maximum discount amount, and ad bidding values. This method yielded a much shorter and denser feature vector with significantly improved training metrics as shown in Table 1.

Optimization

In the absence of constraints, making marketing decisions can be straightforward—simply choose the option that maximizes daily revenue, for example. However, restaurateurs often face more granular requirements, such as running promotions only three days a week or keeping marketing spend below a certain threshold within a given period. When multiple Online Ordering Platforms or stores under management are subject to a shared pool of constraints, the combinatorial possibilities skyrocket, rendering the greedy approach ineffective.

Under these constraints, the challenge transforms into a global optimization problem: determining where, when, and how much of the marketing budget to allocate in order to maximize the aggregate goal across platforms and stores while adhering to the constraints. We formulated this as a mixed-integer programming problem and employed OR-Tools as the solver to identify the optimal combinations.

Putting together

We deployed cron workflows orchestrated by Argo to execute training jobs concurrently, followed by inference and the linear programming solver. All pipelines are updated daily to accommodate new observations that come in every day and to reflect updated constraints.

To enable experimentation, we implemented a shadow system, allowing us to modify models, post-process results, update linear programming solvers, and log and compare outcomes with the production system. Any changes to the models are subjected to offline evaluation to ensure they do not compromise training metrics. Before releasing model updates and alterations to the rest of the pipeline, they must undergo online A/B testing to confirm that the changes yield beneficial and statistically significant results. In the following section, we will detail the setups for online testing.

Measurement and experimentation

The shadow system acts as a testing ground where we can explore new features, refine models, and evolve our optimization algorithms. Once changes are primed for trial, we allocate a subset of stores to operate with the shadow system and compare their performance with a similar group of stores running on the production system.

Timeline A/B

The Timeline A/B experiment is conducted using two comparably profiled groups of stores over two consecutive, equal-length time frames: the control period and the treatment period. During the control period, both the control and treatment groups operate under identical configurations. Then, at the beginning of the treatment period, we transition the treatment group to the shadow system.

To eliminate any pre-existing biases in the groups, the control and treatment groups must not exhibit any statistically significant differences across any of the test dimensions such as daily average revenue. Our initial step uses high-level filtering criteria to select candidates for the experiment. Then, we take data from the period immediately preceding the start of the experiment, bucketize it based on a primary dimension like revenue, attempt a random split within each bucket, and combine the subdivided buckets into control and treatment groups. Next, we conduct t-tests across all testing dimensions. A split is deemed successful if it shows no significance in any of the dimensions. If such a split proves challenging to achieve, we would then revisit and adjust the selection criteria for candidates.

The Timeline A/B experiment design enables store-level comparisons and difference-in-differences analyses. Using these methods, we can determine if changes implemented in the shadow system result in statistically significant improvements by controlling for temporal variations.

A/B testing results should also be congruent with human judgment for interpretability. For instance, we conducted a test to compare system responses under different objectives: maximizing order volumes versus maximizing profit, in scenarios without budget constraints. The outcomes were insightful. When the objective was to increase order volumes, the system suggested highly aggressive promotions, which entailed steep discounts with minimal qualifications. In contrast, when the goal was to maximize profits, the system recommended more conservative strategies, like offering free delivery exclusively to new customers or modest discounts with higher spending thresholds.

Switchback

In situations where the population is insufficient for timeline A/B splitting, or when the test is better conducted within the same cohort (like budget optimization only applies to the same organization), we utilize switchback testing. This method alternates between control and treatment settings at fixed intervals on the same cohort. Companies like DoorDash have employed this technique to run experiments in scenarios where network effects might influence the outcome, where control and treatment groups are not independent and can impact each other.  Because the experiment involves a single cohort, we can pair the control and treatment intervals and employ paired testing methods to enhance statistical power.

Switchback testing avoids the issue of uneven cohort splitting, but it does not completely eliminate the potential impact of time differences between control and treatment periods. To mitigate the risk of results being affected by coincidental timing, one strategy is to extend the duration of the experiments. Another approach is to run concurrent experiments on two cohorts with reversed control/treatment schedules and then pool the data for analysis.

We carried out an experiment to evaluate the effectiveness of global optimization across a large group of stores with a shared budget constraint. The control group utilized a greedy approach, optimizing decisions for each individual store, whereas the treatment group applied the global optimization method discussed earlier. Under this holistic optimization strategy, not every store receives the decision that would maximize its individual outcome, but it prevents marketing pauses due to overspending. This approach is particularly effective when budget constraints are tight. In our experiment, where the total discount could not exceed 30% of revenue, we observed a statistically significant increase of 7% in total order volume for the treatment group compared to the control group.

Continuous monitoring

Beyond online testing, which is designed for controlled experiments to validate system improvements, we also establish continuous monitoring of the production system to gauge the efficacy of marketing decisions. A key metric we monitor is the "incremental gain attributable to marketing," which we calculate by comparing scenarios with and without marketing at an aggregate level over a rolling window. Within a rolling window, we calculate the ratio of revenue between days with promotions and days without promotions. We expect that revenue on promotional days will be higher than on non-promotional days, hence the ratio should remain above 1.

If the benefits of marketing-led scenarios begin to diminish relative to non-marketed ones, we initiate a human-led investigation to identify potential system bugs or sudden shifts in market conditions. Fortunately, our experience thus far has demonstrated a consistent uplift from marketing efforts between 10% to 20%.

Conclusion

In this post, we discussed the development of Otter Marketing, a tool designed to help restaurant owners navigate online marketing across various platforms to maximize their budget's efficacy. Utilizing predictive models and optimization solvers, the system determines the most effective action from tens of thousands of possibilities each day. With our online testing and monitoring frameworks, we ensure that the benefits of marketing are tangible and significant, guaranteeing that our customers' promotional dollars are effectively utilized.

Written by Jakob Schultz-Falk, member of the storage team at City Storage Systems who led the development of the PvcAutoscaler.

Since the introduction of the StatefulSet more and more stateful workloads have been added to the Kubernetes ecosystem. Unfortunately there are still plenty of caveats to running stateful workloads in Kubernetes, some of which are caused by fundamental limitations of the StatefulSet controller and the PVCs it generates.

While stateless workloads enjoy almost boundless elasticity in Kubernetes, stateful workloads, once deployed, are bound by the volumes they mount. These volumes are static and practically immutable, making it difficult to scale to match the ever changing needs of the workloads using them.

This post will present the solution we’ve developed at City Storage Systems aimed at reclaiming stateful elasticity in order to improve cost efficiency and reduce toil.

The Problem

While there is some support in Kubernetes for expanding storage volumes, there is no such mechanism for reducing storage capacity, nor for changing the underlying storage type. In fact the original Kubernetes Enhancement Proposal (KEP) for supporting volume expansion explicitly lists volume reduction as a non-goal due to the complexities of shrinking volumes.

Furthermore, the volumeClaimTemplates section of the StatefulSet spec is currently immutable, preventing operators from expanding volumes via a high level API, though a future enhancement has been proposed.

This leads to a situation where the total capacity provisioned only grows, and any storage related change requires a high-effort and risky migration, inflating the cost of running stateful workloads.

To summarize, the two primary culprits restraining stateful elasticity are:

• Only volume expansion is supported, and only by a subset of volume provisioners

• Immutable volume claim templates embedded in the StatefulSet spec

Our Objectives

Now that we have a clear understanding of the problem, let’s list the objectives we want to achieve with our new solution.

• Volume expansion for growing storage needs

• Volume shrinking to reclaim cost overhead from unused storage

• Volume modification, e.g. be able to swap out an HDD with an SSD or vice-versa

All the objectives above should be solved by an on-demand, declarative, and toil-free solution which can be used by any software engineer accustomed to Kubernetes and StatefulSets.

Kubernetes Storage Concepts

Before diving into the inner workings of the PvcAutoscaler, let’s briefly introduce some of the core Kubernetes components we will rely on to achieve our objectives. These concepts are described in more detail in the official Kubernetes storage documentation.

Persistent Volumes and Persistent Volume Claims

Persistent Volumes (PV) are resources in Kubernetes which contain information regarding the underlying storage they represent. These are typically (but not necessarily) cloud provider managed disks.

Persistent Volume Claims (PVC) are resources which bind to PVs, thereby reserving usage of the PV across the cluster. Pods in the same namespace as the PVC can then reference the PVC to use the underlying storage.

The StorageClass

In order to provision different types of storage, Kubernetes offers the concept of a StorageClass. These can be used to describe different types of storage, e.g. one StorageClass could be configured to provision PVs backed by SSDs while another provides access to HDDs.

In general the process of provisioning a new PV/PVC combination to be used by a pod, is done by creating a PVC referencing a StorageClass. As the PVC is not bound to any PV at creation the volume provisioner of the StorageClass will attempt to provision a PV and bind it to the PVC.

The StorageClass contains a reference to a volume provisioner along with parameters needed to create a new PV backed by some underlying storage. E.g. a volume provisioner developed by a cloud provider would take the input parameters and provision a network attached disk with the desired configuration.

The StatefulSet

This resource type was added to Kubernetes in order to get stable naming mapped to persistent storage across pod restarts. So while the Deployment object would create pods with unique names on each creation, the StatefulSet will create pods with serial ordinals mapping one-to-one to a PVC of the same ordinal.

The PVCs can either be manually provisioned upfront, or it can be left up to the StatefulSet controller to create the PVCs based on the volume claim templates defined in its spec. As mentioned above the StatefulSet offers no support for changes to the PVCs or the underlying storage once provisioned.

The PvcAutoscaler

Now that we’ve introduced the core storage concepts we will be referencing, we can begin using them to build up our PvcAutoscaler solution. We start out by addressing the shortcomings of the StatefulSet.

Taking Control

Since volume claim templates in StatefulSets are immutable, our first task is to detach them so we can modify them as needed. We accomplish this by creating a new Custom Resource Definition (CRD) called PvcAutoscaler. One PvcAutoscaler resource can contain a single volume claim template and a reference to the StatefulSet whose PVCs it manages.

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: nginx
  namespace: default
spec:
  serviceName: "nginx"
  selector:
    matchLabels:
      app: nginx
  replicas: 1
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: registry.Kubernetes.io/nginx:latest
        ports:
        - containerPort: 80
          name: web
        volumeMounts:
        - name: www
          mountPath: /usr/share/nginx/html
  volumeClaimTemplates:
  - apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      name: www
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 4Gi
      storageClassName: hdd-sc
      volumeMode: Filesystem

Figure 1: Splitting out the volume claim templates from the StatefulSet

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: nginx
  namespace: default
spec:
  serviceName: "nginx"
  selector:
    matchLabels:
      app: nginx
  replicas: 1
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: registry.Kubernetes.io/nginx:latest
        ports:
        - containerPort: 80
          name: web
        volumeMounts:
        - name: www
          mountPath: /usr/share/nginx/html
---
apiVersion: storage.css.com/v1
kind: PvcAutoscaler
metadata:
  name: nginx
  namespace: default
spec:
  statefulSetName: nginx
  volumeClaimTemplate:
    apiVersion: v1
    kind: PersistentVolumeClaim
    metadata:
      name: www
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 4Gi
      storageClassName: hdd-sc
      volumeMode: Filesystem

Figure 2: Moving the volume claim templates into PvcAutoscaler resources

Now that the volume claim templates are isolated into PvcAutoscaler resources we can create a Kubernetes operator for the CRD which will generate PVCs for each pod in the referenced StatefulSet based off of the embedded volume claim template.

Figure 3: PvcAutoscaler becoming responsible for the creation of PVCs for a StatefulSet

We now have PVCs for the pods of a StatefulSet based off of a volume claim template embedded in a PvcAutoscaler custom resource. Unfortunately the StatefulSet controller is completely unaware of these PVCs and as such cannot include them in the Pod specs to make them mountable volumes. This brings us to our next task.

Attaching Volumes

Luckily Kubernetes gives us a tool we can use to ensure PVCs are available as volumes in the pod spec. By adding a  mutating webhook targeting the pods of StatefulSets referenced by a PvcAutoscaler, we can look up and attach the PVCs to a pods’ volumes spec prior to validation and creation.

Figure 4: Mutating pod webhook attaching PvcAutoscaler PVCs to StatefulSet pods

After attaching the PVCs, each pod now has a valid spec with volumeMounts all referencing correctly declared volumes.

Expansion Unlocked

After all that work we have returned to a functional setup similar to the plain StatefulSet. We can define a StatefulSet and a number of PvcAutoscaler resources which will manage PVCs and attach them to the pods at creation.

However we have unlocked one great benefit from this exercise. We have gained complete control over the volume claim template embedded in the PvcAutoscaler and can decide what limitations we wish to impose, i.e. we can make properties mutable.

The first target for mutability is the .spec.resources.storage.requests property since our first objective is to allow seamless volume expansion. Allowing volume expansion when the underlying volume provisioner natively supports it is trivial - we just propagate the change in requests to all PVCs and wait for the provisioner to finish the expansion.

In the cases where volume expansion is not natively supported, we would need to follow a more cumbersome process detailed in the next sections.

The Volume Populator

While volume expansion is supported by some volume provisioners, neither shrinking volumes nor changing the underlying storage device are in any way supported. To enable these features we introduce a new custom component, the volume-populator.

PVCs have for some time supported the DataSource property, originally intended to bootstrap a PVC from a volume snapshot. However by enabling the AnyVolumeDataSource gate (which is enabled by default since 1.24) it is possible to use the DataSource property to reference any custom resource (CR).

To leverage this we create a new CRD called the PvcSourcePopulator, which is owned by the volume-populator and whose only purpose is to reference the old PVC, and can itself be referenced in a new PVC through the DataSource property.

apiVersion: storage.css.com/v1
kind: PvcSourcePopulator
metadata:
  name: populator-pvc-0-ba51f
  namespace: default
spec:
  sourcePvcRef:
    name: pvc-0-8770e
    namespace: default
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pvc-0-ba51f
  namespace: default
spec:
  accessModes:
  - ReadWriteOnce
  dataSourceRef:
    apiGroup: storage.css.com
    kind: PvcSourcePopulator
    name: populator-pvc-0-ba51f
  resources:
    requests:
      storage: 2Gi
  storageClassName: ssd-sc
  volumeMode: Filesystem

Figure 5: A PvcSourcePopulator CR being referenced by a new PVC with a different storage class and smaller requested capacity

The volume-populator is responsible for transferring the content of the old PVC referenced in the PvcSourcePopulator to the new PVC. The process it follows can be summarized in the following bullets:

1. PVC_new is created with a PvcSourcePopulator data source referencing PVC_old. PVC_new enters a Pending state since the volume provisioner does not recognize the datasource type - this allows us to take over the task of eventually binding a PV to PVC_new

2. The volume-populator monitors pods in the namespace waiting for the PVC_new to be referenced to ensure the data transfer is executed immediately prior to pod initialization

3. Once PVC_new is referenced, the volume-populator creates an empty PVC_tmp with identical content to PVC_new, but without a datasource. Without the datasource, the volume provisioner is able to create a PV and bind it to PVC_tmp

4. The volume-populator spins up a pod which mounts both the PVC_old and PVC_tmp, and transfers content from PVC_old to PVC_tmp

5. Once transfer completes, the volume-populator unbinds the PV from PVC_tmp and instead binds it to PVC_new

6. The volume-populator discards PVC_tmp which no longer has a PV bound to it

7. PVC_new becomes ready as it now has a PV bound and the pod can initialize normally

Figure 6: The volume-populator bootstrapping a new PVC for pod-0

Just Change

By using the volume-populator in the PvcAutoscaler operator we can efficiently and safely swap out the PVC to match the desired state of the volume claim template. This allows us to support changes to almost all aspects of the volume claim template, including the storage class to change the underlying storage device.

When the PvcAutoscaler operator detects drift between the volume claim template and the current PVCs of a StatefulSet it will try to determine whether the change can be done via online expansion. If it cannot, it initiates the process of swapping out the PVCs using the volume-populator. This process can be summarized by the following:

1. Create new PVCs for each pod containing a PvcSourcePopulator DataSource referencing the old PVC

2. Initiate a rolling restart of the StatefulSet

3. Upon pod creation the mutating webhook will inject the new PVC

4. The volume-populator will detect a pod being created with a reference to a PVC with a PvcSourcePopulator DataSource

5. The volume-populator will transfer the data and once it completes the pod will start up normally

6. This process is repeated for each pod in the StatefulSet

Figure 7: Swapping out the PVCs by using the volume-populator to transfer all data from the previous PVC

Once all the pods have been restarted and had their PVC bootstrapped, the StatefulSet re-enters a steady state. After a pre-configured retention period the PvcAutoscaler will automatically delete the old PVCs to reclaim any cost associated. 

The capabilities of the PvcAutoscaler allows us to ignore future requirements for a stateful workload's storage characteristics, and focus on the current need. We can start with a small HDD, confident that we can boost it to SSD if the need arises. We can add on extra capacity, knowing that we can scale-in again to save on cost later. All on-demand, declarative, and toil-free.

Next Steps

The current solution is quite robust and battle tested, and has drastically improved the cost efficiency of our stateful workloads, but there is always room for improvement. In the future we’re likely to explore the following areas.

• Actual auto-scaling to PvcAutoscaler allowing automated execution of PVC modifications based off of declared sizing strategies and disk metrics

• Support for stateful workloads not using StatefulSets but are instead running using a custom pod controller

• Hot copying of data to reduce the time it takes to scale down disks and switch storage types

Conclusion

The key takeaway we’ve taken to heart from developing the PvcAutoscaler is that Kubernetes is immensely extensible. With enough insight into the ecosystem it is possible to extend even core components with new features and remove otherwise hard limitations. This ability to extend kubernetes is especially relevant for stateful workloads which tend to have more specific requirements and limitations compared to stateless workloads.

Overall the PvcAutoscaler has provided great value since it was rolled out. While the original focus was on reducing cost by eliminating wasteful over-provisioning of storage capacity, the PvcAutoscaler has also enabled us to easily move workloads onto more performant disks when usage reached IOPS and throughput limits.

Without the PvcAutoscaler we would have been forced to expend a lot more time and resources to right-size the storage of our stateful workloads as consumer workload usage patterns change over time. In many instances the effort required to scale in storage would not be worth the cost reduction, leading to an ever increasing overhead. But the PvcAutoscaler makes even small cuts in storage worthwhile.

This post was written by Adam Share, an engineer that helped build Otter POS's offline mode. Matt Park and Tim Pinkawa contributed notes on backend design.

A little over one year ago, the Otter team embarked on a mission to build a point-of-sale (POS) product that meets the needs of many of our restaurant partners. Otter's Order Manager product has already proven an invaluable tool for managing a restaurant's "online" orders from food delivery providers, but what about all those in-person "offline" orders that make up 80% of a restaurant's business?

This post describes how we built a single solution to manage both your online and offline orders using a restaurant's existing hardware, and how that product maintains business continuity in the face of unexpected outages and connectivity issues. Our goal is to deliver the most reliable POS experience in the industry: no orders lost, ever.

OtterPOS: Going online

OtterPOS combined all the things we're good at in the online world with new hardware and software to support the offline world. Payments are through a stripe terminal and a large screen for easily customizing new orders.

This version worked well, and we delighted our initial customers! They loved our simple user interface with a single home for all their orders. But, as we discussed our product with larger restaurant groups, we learned there was an important feature they cared about that we didn't yet support: 

What happens when the internet goes down? What if our servers go down? If you have a customer physically handing you money, it's unacceptable that your POS stops working for any reason. 

Traditional POS systems solve this by running a large server onsite in the restaurant. The POS and other devices connect to the server and the server connects to the internet. When the internet goes down, the server insulates the devices by continuing to process orders. This has some drawbacks though - restaurants are required to buy and install the server, the server is a single point of failure, and onboarding to the system is a lot more complicated.

Can we do better?

OtterPOS: Off the grid

So we built "offline mode".

Two new components allow our devices to continue to operate in the event of connectivity loss:

1. A write-ahead-log in the OtterPOS device to capture all of the events that happened while the device was offline

2. An asynchronous workflow engine in our servers, built on Temporal, to process the log when connectivity is restored

The write-ahead-log captures every event that happens on the device. Every order created, order update, payment made, refund given, and clock-in/clock-out are saved in persistent storage. This means that if the device is turned off for the night and turned back on for the morning shift, the events are still there and will attempt to be flushed. The events are kept in a local database until the device is sure that the server has received them and processed them, in which case they are cleared from the DB.

The device will attempt to flush the log after every new event, and periodically if it detects that connectivity has been restored. When the server receives the log, it immediately starts up a durable workflow to process the events. As soon as the server is sure the workflow is started, it returns the workflow ID so that the device knows the events have been received and will be processed.

Events are processed serially within the scope of a single order to ensure they are processed in the correct order. However, we process events from different orders in parallel with separate Temporal activities to ensure a failure from one order does not block the processing of another order.

In the event of a Temporal activity failure, the workflow will retry indefinitely to ensure the event is not lost. These events already happened in the real world, so we cannot fail to process and record them in our system of record. The durable nature of the workflow and our retry logic provide us with this guarantee.

For transient failures, the activity will eventually succeed on its own. For failures due to backend bugs, an engineer will fix the bug and redeploy, at which point the activity will automatically retry and succeed, moving on to the next event to process. In the event of a client bug where we receive unexpected bad input, we even have the ability to manually rerun a workflow with the correct input.

Upon success, we send an event status to our push framework which pushes the status back to the device. In the event of an unrecoverable error, we send a failure status back to the push framework. In either case, the device now knows the event was processed and is safe to clear from the DB.

Offline mode has already saved our customers from downtime. In March 2023, our cloud provider, Azure, experienced an outage, degrading some of our services. Our customers using OtterPOS did not notice because orders and payments were still processed locally. When the outage was over, our workflows processed all the pending events and everything worked as expected.

OtterPOS: Offline evolution

Offline mode works great when you have a single POS device, but what about restaurants with multiple POS devices? What about restaurants with KDS devices (kitchen display systems)? We still rely on our backend to sync state between the devices, so that an order started on one POS can be finished on the second POS or viewed in a KDS.

The next version of our POS product will solve this by creating a mesh of interconnected devices all sharing state with each other and syncing events to our servers. This will allow all devices in the restaurant to share state even when the network goes down.

Our local mesh network of devices will leverage an existing open source peer-to-peer networking API Nearby Connections. Nearby Connections abstracts away Bluetooth, BLE, Wi-Fi, and LAN connections and will automatically connect using the best available method. Each device can connect to one or more devices over any of the available protocols to form the mesh. 

When two devices establish a connection they will move communication to the highest possible bandwidth pathway. So if two devices find each other over Bluetooth first but join the same WLAN, they will move from a Bluetooth connection to a higher bandwidth WLAN connection.

With devices able to discover and connect to each other to form a mesh network, all app data can now be distributed over the local network. We’ll follow a “publish–subscribe” pattern to generalize how data is sent and received to enable existing features and future development to leverage the offline-first approach. Events that are published will also be cached so that if devices become disconnected or new devices connect, the relevant data can be exchanged to bring all devices back into sync.

In this model, the devices act as both clients and servers and are able to sync data between themselves without the backend orchestrating it. The backend is still essential for durable storage of events, but its responsibility is reduced from being an authoritative coordinator to becoming a peer node in a mesh of devices.

With our new local mesh functionality, all devices in the restaurant can seamlessly continue to operate with no downtime when connectivity is lost, fulfilling our promise of business continuity.

Conclusion

Customers have been delighted with our single solution to managing both your online and offline orders. But a great solution won't get you very far unless it is reliable and allows you to run your business even when things go wrong. This is why offline mode has been a key focus area for our team and continues to be a competitive advantage as we scale offline mode to more scenarios. We've already seen offline mode save our customers hundreds of orders during various unforeseen connectivity issues.

We will continue working to make our entire fleet of devices work while offline, including front-of-house Order Manager, back-of-house Kitchen Display Systems, and customer facing kiosks. They will all work seamlessly together even when the network drops.

Additionally, we plan to make more features available while offline. We focused our efforts first on ensuring business continuity by supporting order placement and payment. But we plan to bring offline mode to menu management, analytics, and more in the near future.

Written by Zain Malik and Nibir Bora, members of the engineering teams that work on core infrastructure.

At City Storage Systems, our Core Infrastructure team navigates the complexities of managing over 100 multi-tenant Kubernetes clusters, each hosting up to tens of thousands of daily active pods. Our entire software stack runs on Kubernetes from mission critical microservices to stateful databases and observability solutions.

This blog delves into our journey of achieving complete automation in cluster provisioning, lifecycle management, and upgrades. With the new toolset, we slashed the time required to provision and prepare a workload-ready cluster from 1.5 weeks to under 1 day, all while maintaining a lean team of engineers. This transformation was catalyzed by our strategic decision to migrate to Microsoft Azure within a deadline of a few months. During the transition the number of clusters we operate more than doubled.

We present a set of Kubernetes custom resources and operators that models our infrastructure and associated operations. The flexibility of the Kubernetes operator pattern makes this approach extremely powerful and we are confident it can be used to manage clusters on any public cloud provider.

Adopting Cluster API

We initially created clusters using Terraform and thereon managed node pools using a custom homegrown Kubernetes operator. Changes, including Kubernetes version upgrades, were handled via GitOps. However, the cognitive overhead and manual intervention required made this approach unsustainable, specially to migrate 80+ clusters from one cloud provider to another.

This led us to explore Cluster API, which offers declarative APIs for simplifying provisioning, upgrading, and managing multiple Kubernetes clusters. Two key factors made Cluster API particularly attractive:

1. Extensibility: Cluster API's custom resources are extended by provider custom resources, which can then be extended by higher level custom resources that capture organizational needs. Since our clusters are multi-tenant, this allowed us to abstract away any details about node pools from workload developers.

2. Operator Pattern: Any extension of Cluster API and its providers are Kubernetes Operators, aligning with our Kubernetes-centric approach to infrastructure management. Leveraging our team's experience with building operators, the learning curve was minimal.

To create a cluster using Cluster API and the Cluster API provider for Azure (CAPZ) we simply need to create objects of the following custom resources:

1. Cluster (from Cluster API)

2. AzureManagedCluster and AzureManagedControlPlane (from CAPZ)

Similarly, to create a node pool we need to create objects of the following custom resources:

1. MachinePool (from Cluster API)

2. AzureManagedMachinePool (from CAPZ)

We were able to seamlessly integrate these processes into our existing CI pipelines, relying fully on GitOps for cluster management. We do not use the clusterctl CLI provided by Cluster API.

However, there were three major roadblocks to adopting Cluster API immediately:

1. Limited support for managed Kubernetes distributions in Cluster API. Although this was part of Cluster API’s roadmap, most of the work done was focused on self-managed Kubernetes.

2. CAPZ’s support for Azure Kubernetes Service (AKS), the managed Kubernetes distribution, was still in an experimental phase, lacking essential features required for our use case.

3. No major engineering organization was using Cluster API for AKS (not that we knew of at the time).

We leaned on our partnership with Microsoft Azure to find a path forward. They suggested we collaborate on the CAPZ project in open source to achieve feature completeness. Several engineers from the Microsoft AKS team along with engineers from our Core Infrastructure team contributed to the CAPZ project prioritizing features aligned with our production use cases. This collaboration was a huge success. It enabled us to launch our first Kubernetes cluster using Cluster API and CAPZ within three months.

Automating Workload-Ready Clusters

While Cluster API and CAPZ simplified cluster creation, these clusters weren’t ready for workloads yet.

1. The new cluster does not have permission to access container images from Azure Container Registry (ACR). It is a reasonable design choice to leave such dependencies out of Cluster API to keep the interface generic.

2. An AKS cluster comes configured with a default cluster autoscaler profile. Configuring anything other than default can be done by manually running a Azure CLI command. We tune Cluster Autoscaler to achieve resource optimization and bin-packing on all of our production clusters.

Using Terraform and running Azure CLI commands to configure these for every cluster wasn’t aligned with our principle of minimizing human intervention. So, we decided to write a companion Kubernetes operator instead. This introduces a AzureClusterAdditionalConfig, an extensible custom resource intended for any additional Azure managed service configurations necessary for a cluster. For ACR permissions this resolves to a AzureRoleAssignment object, and a AzureClusterAutoscaler object for custom cluster autoscaler configuration.

Introducing the companion operator enabled us to fully automate cluster creation and prepare them to be workload-ready by installing a single kubernetes resource using GitOps. This streamlined approach facilitated the migration of over 80 production clusters between different cloud providers.

Automating Node Pools

After running production workloads on a new cloud provider for a few months, we identified two major operational pain points.

1. We didn’t nail the node types (instance family, disk type, etc. settings) at the first attempt. Some of these fields like machineType, diskSize, diskType, maxPod, type (spot vs regular) are immutable fields on AKS. This meant we had to replace node pools running production workloads a handful of times. Each replacement involved creating a new node pool, draining the old one, then deleting it. This process required human coordination and multiple steps in our GitOps workflow.

2. While updating Kubernetes version we learned that AKS’s in-place node pool upgrade tends to enter an endless retry loop when it encounters an application with 0 disruptions allowed (PodDisruptionBudget setting). Since AKS only permits one concurrent node pool update operation per cluster, this block operations on other node pools including manual scale up. Consequently, we had to fall back to the multi-step node pool replacement process for upgrades as well.

We implemented a node pool Kubernetes operator, which introduces a single Nodepool resource encapsulating the multi-step process for replacing a node pool. Under the hood, the operator creates a new node pool, drains the old one, then deletes it in a process completely opaque from the users. From the user’s perspective all node pool manipulations are done in-place with a single GitOps change. This end-to-end automation is especially powerful during Kubernetes version upgrades.

The only limitation was that we couldn’t link the new Nodepool resource to the existing hierarchy of Cluster API’s MachinePool or CAPZ’s AzureManagedMachinePool resource using ownerReference. Instead, these resources exist side by side and are linked using objectReference. This drawback becomes apparent when deleting a node pool entirely, as the resource hierarchy is removed using finalizers.

Conclusion

Evolving our cluster management toolchain empowered us to manage twice as many clusters and applications, while sustaining the same handful of engineers on our Core Infrastructure team. This achievement marks a significant boost in operational efficiency, made possible by our steadfast commitment to GitOps principles and the elimination of human-in-the-loop processes. Along the same lines, self-healing, drift detection, etc. are first class ideas on our platform. This mindset allows us to always prioritize organizational needs leveraging the extensible nature of Kubernetes, while balancing efficiency, reliability and agility.

Reflecting on our journey, we've had some key learnings and missteps:

1. Big Bold Bets: When we explored the industry state-of-the-art for cluster management we couldn’t find anything that meets the level of automation we were striving for. At such times, we leaned on our company’s core value - Big Bold Bets. The bet on Cluster API was fruitful in the long run.

2. Hidden cost of Open Source: Strategic partnership and driving appropriate community engagement was vital to making our open source collaboration a success. We did learn that a long term commitment is necessary to be able to effectively steer an open source project to continue to meet our needs. We remain committed to contributing to Cluster API for the foreseeable future.

3. First Adopter Risks: A couple years ago we experienced a Sev1 incident lasting several hours where 60% of our nodes on production clusters were wiped out. We traced it to a bug in CAPZ where just the sequence number suffix was used to identify nodes instead of the full spec.providerID. This caused Cluster API to reference nodes from different node pools within a cluster using the same ID and in turn deleting them.

4. Automation Enhances Reliability: Despite not being the primary focus, our automation efforts significantly improved systems reliability. We’ve successfully completed 4 incident-free Kubernetes version upgrades. Previously, there used to be at least one internal incident per version upgrade.

5. Kubernetes Operator Pattern: We leverage Kubernetes operators extensively in our infrastructure organization. Adhering to standard design patterns like goal state reconciliation enables us to sidestep design debates, mitigate concerns about corner cases, and minimize the learning curve. This along with using managed Kubernetes distributions exclusively was the right tradeoff for us. We remain committed to this approach even today.

So, what lies ahead for us? We've already kicked off a strategic plan to get us to the next level of scale. This involves automatically partitioning workload clusters, considering factors such as API Server pressure and node size. We've also started provisioning more single-tenant clusters. Furthermore, efforts are underway to streamline the time it takes to prepare workload-ready clusters, including steps like IP address allocation and installing cluster addons. All these initiatives are guided by our commitment to minimizing human intervention and achieving full automation. With this roadmap, we aim to adeptly manage over 500 Kubernetes clusters efficiently.

We would like to acknowledge Cécile Robert-Michon, David Tesar, and Jack Francis at Microsoft. Each contributed during various phases of the project.

This post was written by Zhuyi Xue and Suming J. Chen, who led the development of prep time prediction.

Accurate food prep time prediction has a variety of use cases for food delivery orders. For instance, delivery service providers (DSPs) like UberEats and Doordash need prep time prediction in order to provide estimates for when food will be delivered to their customers, and to better determine when to dispatch couriers to pick up the food. While DSPs have been routinely predicting food prep time, CloudKitchens has an advantage in such prediction because we have more comprehensive insight into kitchen operations. We have deployed our own prep time prediction model that has led to more streamlined logistics (i.e. customers get fresher food and couriers wait less) and fewer customer complaints. 

In this post, we show how we predict prep time for kitchens at CloudKitchens facilities and describe the engineering challenges and product decisions made throughout the process.

Food prep time prediction

Problem Description

For each order, kitchens are required to send prep estimates to DSPs. These estimates are not only used by DSPs, but also by internal facility staff to coordinate logistics. Initially, this value was manually set and usually fixed (e.g. 0/5/10 minutes) – regardless of how busy the kitchen is or what items the order consists of, leading to logistics inefficiencies and customer complaints.

Intuitively, we know how long an order takes to complete depends on a variety of factors (kitchen busyness, order size), but it’s not feasible for kitchen staff to accurately and consistently provide order-specific prep time estimates.

At CloudKitchens, we have valuable kitchen data – real-time monitoring of every kitchen’s current orders-in-progress as well as when each order is complete. Leveraging this data allows us to develop a machine learning (ML) solution to predict the prep time that significantly outperforms heuristic approaches.

Methodology

Following the best practices for ML engineering, we did not adopt a ML solution right away at the beginning, and instead just used the median prep time of historical data as the prediction for each kitchen to ensure the infrastructure (e.g. training pipeline, gRPC service, metrics, monitoring) was set up correctly (Figure 1).

After an end-to-end heuristic model was deployed, we then switched to a linear model with a set of simple features that can be derived from the order itself, e.g. number of items in the order, subtotal of the order, time-of-day of the order. Following, we moved onto a gradient boosting model using LightGBM. We chose LightGBM for its specific way to handle categorical features (e.g. kitchen ID), which tends to outperform one-hot-encoding-based approaches. We also tried a neural network model, but it did not outperform gradient boosting in our experiment, so we decided to shift focus to engineering more advanced features. 

A couple of such features that improve model performance significantly are

1. Average prep time of trailing orders (avg_trailing_prep_time) for a kitchen. The trailing orders are the most recently completed orders before the order we’re predicting prep time for.

2. Total number of orders being currently prepared from all DSPs in the kitchen (num_queued_orders), a proxy for kitchen busyness. 

Both features are available to CloudKitchens kitchens but hidden from DSPs. What distinguishes them from the simple features is that they’re highly dynamic as they depend on the states of other orders from the same kitchen. The avg_trailing_prep_time feature needs to be updated whenever an order is finished cooking and the num_queued_orders feature needs to be updated whenever a new order comes in or an existing order is finished cooking. Such dynamic nature poses challenges to ensure feature consistency between model training and serving times, which will be further discussed in the Engineering Challenges section.

As we progress towards more complex models and features (Figure 2), model performance consistently improves. Compared to the manual predictions, our ML-based solution improves the mean absolute error (MAE) by 42%. We choose MAE over mean squared error (MSE) as the main metric for its robustness against outliers.

Engineering challenges

Feature consistency

The calculation of features like avg_trailing_prep_time and num_queued_orders requires tracking of all recent order states in a kitchen. Initially, at training time, we calculated such features in Python, while at serving time, the model client was written in Java. As a result, the same logic was implemented twice in different languages, which almost always led to some feature skew between training and serving time, and was unscalable for adding additional features.

To deal with the feature skew issue, we decided to use a feature store. We evaluated multiple open-source solutions for feature store implementations, but found that there were drawbacks to each approach, e.g. they could not integrate with the multi-regional technologies CloudKitchens uses, or their lack of support in SDK for our stack.

In addition, we recognized the need for a system that could handle the complex interdependencies between various data features due to their highly dynamic nature. Specifically, we required a setup where an update to one feature could automatically trigger updates to related features across different entities. For example, when a kitchen completes an order faster than anticipated, this event could alter our prediction of preparation times for pending orders in that kitchen, which might also affect the kitchen's direct-to-consumer ranking, which is also considered a feature, compared to others. To address this, we built our own feature store using CockroachDB (Figure 3), which was chosen for its resiliency and scalability. This feature store was designed to support the cascading updates of feature values that our data environment demanded. 

Automatic model refresh

As CloudKitchens expands rapidly, new kitchens continue to be added, so the prep time model needs to be retrained periodically with the most recent data to account for them. In addition, the evolution of prep time patterns in existing kitchens driven by events like cooking appliance upgrades/additions, menu updates, and staff churn also requires periodic model refreshes.

To make model refresh scalable to additional logistics models, we implemented an automatic model refresh strategy (Figure 4). Suppose the incumbent model (M_i) is trained on data between t₀ and t₃, and some new data is collected between t₃ and t₅. To train and evaluate a new model, we follow these steps:

1. Split the data between t₁ and t₅ into three parts: fit set (t₁-t₂), validation set (t₂-t₄), test set (t₄-t₅). Note, we keep t₅ - t₃ = t₁ - t₀ to avoid including very old data for model training because our research shows that, contrary to the common impression that the more data the better, they hurt model performance when evaluated on the most recent orders, which indicates that prep time patterns evolve quickly.

2. Fit a model on the fit set, validate it on the validation set, conduct hyperparameter tuning, and pick the best hyperparameters (HP).

3. Train a candidate model for comparison (M_c) on the fit + validation set with HP. 

4. Calculate the metrics for both M_i and M_c on the test set, and compare them.

5. If M_c outperforms M_i, train a new production-ready model M_p on fit + validation + test set with HP.

Note, the new data collected between t₃ and t₅ must be split into two parts because 1) if all of it is used for training, then we have no data for testing; or 2) if all of it is used for testing, then the new model can only be trained on the same or a subset of data used for training the incumbent model, and will unlikely show any improvement.

Independent of model training, the model service attempts to load the latest production-ready model in a cron-like fashion so that it can make use of M_p shortly after it becomes available. We also have monitoring in place to alert if the incumbent model gets too old for unexpected reasons.

Product Decisions

Quantile prediction

Some of our products require quantile predictions besides a point estimate of the mean/median prep time. For instance, our decision-making algorithm for deciding when to send a scheduled order to the kitchen for preparation is based on p25 and p75 quantiles of the prep time estimates. To accommodate the additional quantiles, we have opted for quantile regression for model training, which uses a so-called pinball loss:

where t is the ground truth label, x is the feature vector, y is the model, I is the indicator function, and τ ∊ (0, 1) is the alpha corresponding to the quantile interested. When τ = 0.5, pinball loss is equivalent to MAE loss. We evaluate our quantile regression models by checking their average calibrations, e.g. a p25 model should produce predictions that are lower than the ground truth labels about 25% of time. While exploring alternatives, we also considered NGBoost, which generates a probabilistic forecast based on a pre-assumed parametric distribution (e.g. lognormal). However, NGBoost took significantly longer to train a model on our dataset, rendering it less practical. Besides being more scalable, quantile regression offers the advantage of not imposing any assumptions about the distribution form. A drawback of this approach is the need to train separate models for each quantile, so p25, p50 and p75 correspond to three distinct LightGBM models. Given our focus on only a few quantiles, quantile regression remains a suitable choice. We view quantile regression as a compromise between point estimation and fully probabilistic forecasting. 

Remaining prep time prediction

When an order is first received, we always make an initial prep time prediction to return to the DSP. However, our belief in this prep time is fluid, meaning that over time there are events that may change our initial estimate. For instance, we may have initially predicted that an order would take 15 minutes, but 10 minutes into food prep the kitchen may have become much busier, so we would be able to take into account the updated state and make a more accurate prediction for the remaining prep time. Our motivation for this is to ensure that at any point in time, the most accurate estimate is available – important for use cases like planning robot routing for order conveyance as well as keeping couriers updated on the pick up time.

We solve this problem by:

1. Adding an additional feature that measures how much time has passed since an order is received (time_since_order_received). As illustrated in Figure 5,  at t₀, the time_since_order_received is 0, and at t₁, it is t₁ - t₀, and so on.

2. Sampling additional training examples whenever there is a change in kitchen state. An example is sampled whenever a new order arrives (t₂, t₃, t₅, t₆) or an existing order is being completed (t₁, t₄), so seven examples are obtained from the order instead of one in Figure 5. Using the aforementioned feature store, it is straightforward to sample the training data at different timestamps within an order’s prep time. 

   

As shown in Figure 6, at any point in time, we could simply estimate the remaining prep time by subtracting the initial predicted prep time (~20min) by elapsed time (orange line). However, we can see that 20 minutes in, the only option is to continually show 0 minutes remaining or show negative numbers. Instead, if the remaining prep time is used, we can continue to finesse the prediction all the way until the order is complete (red line). The remaining prep time model is shown to be 10% more accurate than the baseline approach.

Conclusion

We describe how we leverage the unique data we have in order to predict food prep time 42% more accurately than the naive baseline.  We discuss our approach to engineering challenges like train/serve feature consistency, automatic model refresh, and how we enabled this feature to be even more powerful by adding on quantile predictions and the ability to predict the remaining prep time. In the future, with the addition of new technologies and increased capacity to capture the state of the kitchen, we expect to continue to improve model quality.