Enhancing event-driven architecture with Temporal and Fauna
This blog post demonstrates how to use Temporal and Fauna to simplify your event-driven architecture. Temporal is an open-source microservices orchestration platform that provides durable execution for distributed applications. You can get up and running with Temporal easily with its cloud offering. Companies like Stripe, Netflix, Datadog, Coinbase, and many more use it. Stripe and Coinbase use it for payment processing and transaction workflows, whereas Netflix uses it for media processing pipelines. Fauna is a fully serverless, distributed database that combines the flexibility of documents with the consistency and power of a relational database. Fauna is a general-purpose operational database, that's particularly well suited for event-driven, microservice applications because of its lightweight HTTPS API, native multi-tenancy and serverless. When used together, Temporal and Fauna can seamlessly integrate event processing and durable state management.
This blog includes a sample expense tracker app that uses Temporal for workflow orchestration and Fauna for state management. You can jump straight into the code, with a readme file explaining how to run it. The app shows how Temporal and Fauna handle failures, network issues, flaky APIs, and long-running processes.
Temporal provides a mechanism for handling failures and allowing workflows to resume from where they left off. Temporal has automatic retries and it persists the state of workflows, including all activity results, local variables, and execution progress into a Fauna database. This allows workflows to resume from their last known state after failures, worker crashes, or deployments.
Even though Temporal has a built-in datastore for managing workflow state, using Fauna as an external database offers more flexibility and advanced functionality. Fauna allows for complex querying, better reporting, and long-term storage of workflow data. It ensures data portability, durability, and integrates easily with other services. By using Fauna, you can handle auditing, historical records, and apply custom business logic, which adds versatility to your applications. Later in the article, we will briefly discuss the benefit of using Fauna with Temporal.
Event-Driven Architecture
Before diving further, let’s start by defining Event Driven Architecture.
An Event-Driven Architecture (EDA) is a software design paradigm in which system components communicate and react to events. An event is something that occurs within the system, such as a user action, a system condition, or an external input that corresponds with a change in the application state. An EDA's components are decoupled and communicated through events, enabling them to respond to changes asynchronously.
For example, consider a financial application. An event is produced when a client (i.e. web or mobile app) processes a payment. Let’s assume the payload looks as follows.
{
"event": "PaymentProcessing"
"data": {
"paymentId": "12345",
"processor": "Stripe"
},
"timestamp": "2024-09-10T10:01:00Z"
}The event is sent to a message broker. State-of-the-art EDA implementations are now distributed. In such applications, services like Kafka or AWS Eventbridge are commonly used as message brokers. Other services listen to the incoming messages.
A payment processing service will pick up the first event and will run some business logic. For this scenario, let’s imagine it processes the payment with Stripe or Paypal. After processing it created it own event and sends it back to the message broker. Let’s assume the produced event looks as follows.
{
"event": "PaymentProcessing",
"data": {
"paymentId": "12345",
"processor": "Stripe"
},
"timestamp": "2024-09-10T10:01:00Z"
}Now another service like an email service can consume this event and process sending emails.
Notice that in a distributed Event-Driven Architecture (EDA), a service is a self-contained unit that performs a specific business function and interacts with other services or components by producing and consuming events. Modern state-of-the-art EDAs are usually distributed.
The following diagram shows the flow of events within our imaginary application.
The main benefits of an Event-Driven Architecture (EDA) include: improved scalability, real-time responsiveness, loose coupling between system components, flexibility to add new services easily.
EDA is fault-tolerant and allows you to handle high volumes of events efficiently by distributing processing across multiple systems. This allows for better agility and adaptability to changing requirements. However, EDA comes with challenges.
Challenges of EDA
There is no panacea and an EDA also comes with its own set of trade-offs. For example, as your application grows, debugging becomes more complex as multiple services react asynchronously, making it difficult to trace issues. Ensuring event ordering and consistency can be tricky, especially in fast-paced systems where events may arrive out of sequence. Additionally, there's a risk of data duplication as multiple services may process the same event, leading to inconsistencies. These challenges unique to EDA can be resolved by adopting Temporal and Fauna, as we’ll see in the later section.
Take an e-commerce system, for example. An OrderPlaced event might be consumed by both the Inventory Service and the Billing Service. The Inventory Service reduces stock based on the order, while the Billing Service charges the customer. Inconsistencies can arise if these services don't handle the event properly, such as processing it multiple times due to retries or network issues. In a production environment, inconsistencies can lead to problems. Take Uber for an example, it is important to notify and match a driver with a rider within a reasonable timeframe. If the system matches multiple drivers to the same user due to inconsistencies in data, it will lead to confusion for the users.
The Inventory Service might accidentally reduce stock twice, causing it to show fewer items available than are actually in the warehouse. Similarly, the Billing Service might charge the customer twice for the same order. This leads to conflicting views of the system's data, with each service holding its own version of the truth, creating problems like inaccurate stock counts or overcharging customers. Developers usually implement strategies like Saga patterns to deal with these issues.
How Temporal and Fauna simplify EDA
Temporal and Fauna simplify event-driven architecture (EDA) by combining Temporal’s ability to orchestrate and manage stateful workflows with Fauna’s global, strongly consistent, serverless database. Temporal ensures reliable execution of long-running workflows and retries in case of failures, while Fauna provides immediate, consistent access to event data across distributed systems. This combination enables seamless event processing with strong data integrity, removing the complexity of managing distributed state and scaling concerns, allowing developers to focus on business logic.
At the application level, Temporal resolves this issue with workflows. Temporal automatically handles retries and ensures idempotency. Workflows are designed to resume exactly where they left off after a failure, meaning they don't restart from the beginning or reprocess events unnecessarily.
At the database level, Fauna ensures strong consistency. Fauna also provides Computed fields, Check Constraints and User-Defined Functions giving you the ability to run complex data compute logic directly in your database and apply additional constraints for consistency and data integrity. Think of User-Defined Functions as serverless functions that live inside your database, can perform complex compute operations with your data and have zero cold start.
Why Fauna?
Fauna is a compelling choice to use with Temporal for the following reasons:
- Strong consistency: Fauna provides strong consistency across distributed environments. This aligns well with Temporal's workflow execution model, which requires consistent state management. Event data must be reliably written and queried. Fauna guarantees that events are fully persisted and consistently available.
- Document Flexibility: Fauna allows you to store events as documents, enabling you to capture both structured and unstructured data. This flexibility makes it easy to store various types of events with flexible schemas, which is often needed in event driven architectures where events can evolve over time. The Fauna Query Language (FQL) allows you to define RDBMS-like relationships between JSON documents and resolve relationships at query time in a performant way, making it easier to store and access complex data.
- Scalability: Both Fauna and Temporal scale horizontally, allowing applications to handle increasing loads without sacrificing performance. You get horizontal scalability without needing to either manually scale up/down or build other automated scaling mechanisms, which are complex to implement and maintain.
- Built-in temporality: Fauna's native support for temporal queries aligns well with Temporal's focus on long-running, stateful processes, making it easier to implement features like auditing, rollbacks, or historical analysis of workflow execution.
- Event streaming: Fauna's event streaming capabilities can be leveraged to build event-driven architectures in conjunction with Temporal workflows, enabling real-time updates and reactions to data changes.
By combining Fauna's distributed, consistent database capabilities with Temporal's workflow orchestration, developers can build robust, scalable applications that efficiently manage complex, long-running processes while maintaining data integrity and accessibility across distributed environments.
Overview of the sample application:
This application tracks spending by category using Temporal. It has two main features:
1. Transaction Workflow: It calculates total spending based on a list of transactions. If the total exceeds 100, it sends a message and doesn't save the transactions.
2. Category Spending Workflow: It calculates the total spending for each category over the last 7 days.
Both workflows are managed by Temporal, with activities handling transaction processing, saving, and calculating totals.
You can find the complete code for this sample application here.
When you are building production applications for scale and flexibility, you require advanced querying capabilities from your database. Fauna is ideal in this case because Fauna gives you the flexibility to store unstructured data with the advanced query capabilities of a relational database.
While Temporal’s built-in store manages workflow state only, using Fauna ensures that the transactional data is available for broader use cases, such as tracking historical spending trends and handling custom business logic.
Unlike traditional databases, Fauna seamlessly handles distributed data with global consistency, making it perfect for applications that require complex transactions across different categories, clusters and time periods. Fauna’s scalability and ability to integrate with Temporal workflows ensure reliable performance without the need for complex infrastructure.
Furthermore, Fauna's document-relational model allows for easy adaptation to changing data requirements, which is essential for evolving applications.
Reference material:
If you enjoyed our blog, and want to work on systems and challenges related to globally distributed systems, and serverless databases, Fauna is hiring
Subscribe to Fauna's newsletter
Get latest blog posts, development tips & tricks, and latest learning material delivered right to your inbox.