Understanding Event Sourcing: Modeling with Landmark Events

Welcome, everyone! Quick show of hands: how many of you are currently using event sourcing? Great! For those who aren't, no worries—today's talk is all about event sourcing, so you're in the right place.

Now, I have a confession to make: teaching event sourcing is no easy feat. There, I said it. But let's make things easier by starting from the beginning.

The Origin Story

Picture this: It's me, completely clueless about event sourcing, just opening my first bank account. Fast forward 19 years (yes, it's been that long), and let's crunch some numbers. Assuming I conducted three transactions per day, totaling 20,805 transactions over the years (let's ignore leap years for simplicity). Now, if you're a bit more active, maybe 10 transactions per day, you're looking at nearly 70,000 events!

For a company with 100 transactions a day, that number skyrockets to almost 700,000 transactions. But why does this matter, you might ask?

The Relevance of Event Stores

Event stores care about these numbers, and here's why: it's intrinsic to the nature of event sourcing. When explaining event sourcing, we often start with the example of a bank account—it's relatable. You get events like "withdrawn some cash from ATM" or "incoming trust transfer recorded."

So, why should you care about these numbers? Because understanding the sheer volume of events is crucial to grasping the essence of event sourcing.

The Pitfalls of Snapshots in Event Sourcing

In the realm of event sourcing, the idea of snapshots often surfaces as a potential solution. However, I'm here to make a case against it. Let's explore why snapshots might not be the silver bullet you think they are.

Snapshotting Woes

So, what's the typical approach to snapshotting? Many advocate for taking a snapshot after each event. Sounds reasonable, right? But what if we rethink this strategy? Instead of creating a snapshot after each event, consider a more judicious approach, like taking a snapshot once per "something."

The Hope-Driven Design

This leads us to a crucial question: how often should we take snapshots? The truth is, it's often a shot in the dark. Whether it's every 10 minutes, every day, or after a specific number of events—there's no one-size-fits-all answer. It becomes a hope-driven design, where we hope we've predicted the optimal frequency. Black Friday might disrupt the steady flow we anticipated, and that's when issues arise.

Storage Dilemmas

Now, let's discuss where to store these snapshots. Storing them in the same event stream might seem like a logical choice—it's streamlined. However, in reality, it complicates matters. Instead of making the stream shorter and more manageable, it extends it. The atomicity of transactions becomes a concern when appending events to a single stream.

Storing snapshots in a separate stream is an alternative, but it introduces challenges related to atomicity. Many event stores offer atomicity when appending events to a single stream, but this advantage diminishes when dealing with multiple streams.

Another option is to store snapshots in a relational database or a cache like Redis. However, regardless of the choice, the atomicity issue persists. It's a balancing act between data integrity and system performance.

The Seduction of Snapshots

Why are snapshots so popular if they come with these challenges? Much of the advocacy for snapshots revolves around the central role of a messaging system. For instance, if you're using a messaging tool like Kafka, the lack of optimistic concurrency necessitates the caching of states. Snapshots, in this context, become a coping mechanism.

But here's the catch: snapshots are not the source of truth. In a messaging system, eventual consistency reigns, and relying solely on snapshots introduces uncertainty into your read models. They act as a cache, not a definitive record.

The Ticking Bomb

Snapshots, like many things in programming, are great until they're not. They bring an initial boost, like a loan or a mortgage, providing a temporary relief. However, the real challenge surfaces when the snapshots become outdated. Changing your aggregate structure or adapting to new requirements can quickly turn snapshots into a ticking bomb.

In essence, snapshots are a trade-off. They offer a performance boost initially, but the long-term maintenance and adaptability challenges may outweigh the benefits. Stay tuned as we delve into alternatives and best practices for effective event sourcing without relying heavily on snapshots.

Leveraging Business Events: A Practical Guide

In the quest to optimize event sourcing, let's explore a powerful alternative to snapshots: leveraging business events. This approach not only solves the challenges posed by snapshots but also aligns more closely with the natural flow of business workflows.

Rethinking the Stream

If we scrutinize the account stream model I previously presented, we might discern a crucial omission. Two small yet pivotal events could significantly enhance our modeling: "Accounting Month Opened" and "Accounting Month Closed." These events mirror the rhythm of accounting operations, where maintaining a record of every transaction isn't imperative for current month calculations.

The Power of Temporal Slicing

These events become foundational in the context of our stream's lifetime, effectively slicing the lifecycle into manageable segments. Instead of dealing with the entirety of transactions, we can focus on specific streams aligned with the temporal aspects of our business operations. This modeling approach extends beyond event sourcing—it's a best practice in general to align your models with temporal aspects, aiding in creating more concise and business-aligned representations.

Business Wisdom and Lifecycle Events

Why aren't these lifecycle events widely known or discussed? Business flows often take them for granted, considering them inherent and self-evident. It's our responsibility as developers to ask enough "whys" and dig deeper to uncover these implicit life cycles. In many business flows, these life cycles exist, and identifying them leads to more refined and manageable models.

Each life cycle commences with a specific event, signaling the initiation of the next period. While the event type may vary, there's always an event initiating the stream's initial state. Additionally, events closing the period are recorded for all streams, effectively encapsulating the aggregate's lifecycle.

This shift transforms our aggregate from a generic "account" to a more nuanced "account month," aligning with the business reality. The stream name reflects this change, incorporating a temporal aspect, like "account/month/2023/01."

Practical Implementation

Now, how do we practically implement this shift? Closing the books is highly dependent on your business use case, and understanding the business process is paramount. Here are some potential approaches:

Separate Operation: Treat the closing and opening of accounting months as distinct operations. This manual intervention may be suitable for cases where there's a deliberate process for closing one month and opening the next.
Batching: Depending on your tech stack, batching could be an option. Close the accounting month, and then have a subscription listening for the "Accounting Month Closed" event to automatically create a new stream.
Dynamic Approval Process: Engage with your business stakeholders. It might unfold that your use case involves a more intricate process than initially thought. Maybe there's an approval step before closing a month or additional reporting requirements.
Truncate or Archive: Most event stores support data truncation or archiving. Instead of creating new streams or snapshots, mark events before a specific position as "to be deleted." This operation effectively truncates the data, giving you a streamlined, manageable stream.

The Difference from Snapshots

You might wonder, "Isn't this just another form of snapshots?" The key distinction lies in their nature. While both represent the state at a point in time, lifecycle events like "Accounting Month Closed" and "Accounting Month Opened" are tightly linked to business processes. They reflect specific business operations and facts. Snapshots, on the other hand, are purely technical—a snapshot of the current state without the nuanced business context.

In essence, embracing business events and understanding the lifecycle of your aggregates provides a more intuitive, business-aligned, and scalable solution compared to the conventional use of snapshots. Stay tuned as we delve deeper into refining event sourcing practices for greater effectiveness.

Conclusion: Embracing Business Realities in Event Sourcing

In the realm of event sourcing, the allure of snapshots might seem tempting, promising a shortcut to managing states. However, before you embark on the snapshot journey, I implore you to pause and reconnect with your product team and business stakeholders. You may discover that your domain harbors implicit life cycles—critical temporal aspects that can revolutionize how you manage streams.

Unearthing Life Cycles

Life cycles are not an elusive concept; they exist in most domains. Engage in conversations with your business team, ask the right questions, and peel back the layers to reveal these hidden rhythms. Once identified, these life cycles empower you to slice your streams effectively, making your stream management more efficient and your development journey smoother.

Diverse Strategies for Stream Closure

Closing the books on streams doesn't have a one-size-fits-all solution. Experiment with different tactics—whether it's using multiple streams, a single stream, or dynamic approval processes. Tailor your strategy to each use case, recognizing that nuances in your domain may demand diverse approaches.

Search for Life Cycles in Your Model

Look closely at your model, whether it's a cashier registry, hotel management, or patient appointments. You'll likely discover distinct life cycles that prompt you to rethink your stream management. Slicing your domain based on these life cycles brings you closer to the business process, liberating you from unnecessary data and state obsessions.

Archiving for Efficiency

Consider whether archiving is a necessity in your domain. While not every scenario demands archiving, if your data volume requires it, planning in advance is prudent. Deliberate on what data is crucial for current operations and what data is reserved for historical read models, ensuring your business logic aligns seamlessly with your operational needs.

A Temporal Perspective on Data

Thinking about data from a temporal perspective offers valuable insights. Not all data within your aggregates may be essential for immediate business decisions. Distinguishing between operational necessities and historical requirements allows you to streamline your stream and optimize your event sourcing strategy.

Keep Your Streams Short

In closing, I leave you with a simple yet profound message: keep your streams short. By embracing the temporal aspects of your business, identifying life cycles, and tailoring your strategies accordingly, you pave the way for more manageable, efficient, and business-aligned event sourcing.

Happy stream slicing!