Implementing Multi-Session Sequential Convoy Pattern with Azure Service Bus and Go

In many event-driven applications, preserving the sequence of events is essential. For example, an event-driven eCommerce application might have the following states, transitions, and events.

  1. A user adds N items to the basket. This action generates the item added event.
  2. The user checks out the basket. This action generates the basket checked out event.
  3. The user pays for the items. This action generates the payment made event.
  4. Inventory decrements the count of available items by N. This action generates the inventory updated event.

For such an application to function correctly, it is critical to maintain the sequence of events. For instance, processing the payment made event before the basket checked out event might lead to errors in billing or inventory systems. In a horizontally scalable system, we can not guarantee sequential processing of messages without creating groups of messages so that the service bus always delivers the messages within a group in sequence. The message consumer must finish processing a message and update the status of the message to the service bus before the next message in the sequence is delivered to it. Azure Service Bus supports the concept of the grouping of related messages through the message sessions feature.

Processing a set of related messages in the First In First Out (FIFO) order is documented as a pattern named the Sequential Convoy pattern in the Azure architecture pattern guide. I encourage you to familiarize yourself with the pattern’s details, including the use cases, to ensure that the architecture pattern is a good fit for your scenario.

Multi-Session Sequential Convoy

Using Azure Service Bus sessions, you can group related messages so that the message consumer locks a session and processes messages in the session one at a time. Processing messages in a session requires the message consumer to know when to close the session. When the consumer receives the last message of the session from the Service Bus, it closes the session and moves on to process messages of another session. There are a few approaches that you can consider to define the last message of the session so that the client can instantly identify it when it arrives.

  1. Producer and consumer agree on the name of the last event/message that the producer will specify either as a message property or in the message body.
  2. The producer uses a custom message property to inform the consumer that the message is the last one in the session.

As you can imagine, establishing such contracts makes the producer and consumer somewhat inflexible. Any change that extends the end of the session will require changes in the contracts and the consumer. Let’s solve this problem.

In most cases, our only requirement is that the messages in a session must be processed in sequence irrespective of the consumer instance that processes it. If we do not attach a consumer instance to a session, we can fix the session termination problem by adding a processing timeout period to the consumer. Following is how this approach will work.

  1. The consumer locks any available session on the Service Bus.
  2. The consumer records the system time when it receives a message from the session in a local timestamp variable and starts processing the message.
  3. An asynchronous routine in the consumer periodically reads the latest timestamp value and closes the session if it determines that the consumer has been idle for some time.
  4. The consumer again attempts to lock any available session on the Service Bus and reiterates the workflow.

What if the producer adds more messages to the session after the session is closed by a consumer instance? In that case, another consumer instance will receive and process them when it is available.

Azure Service Bus has a Golang SDK, which I will use to implement the pattern. Feel free to use the concept to implement the pattern using any supported Azure SDK programming languages. Note that Azure Functions already implement this pattern, so use the built-in Service Bus bindings for processing session messages in Functions.

Create Azure Resources

We need a Service Bus namespace and a queue in the namespace for this demo. You can either use the Azure management portal or use the following Azure CLI commands to create the resources. Note that only the standard and premium tiers of Service Bus support sessions. If you plan to use the Azure management portal for provisioning the resources, remember to set the Enable Sessions flag in the list of options to create the queue.

We will create a resource group namedmyresourcegroup, a Service Bus namespace named rahulr, and a queue named SessionQwith the following commands:

Shell

Record the connection string of the namespace that you created. Let’s now develop the application that reads messages from the session aware queue: SessionQ.

Source Code

Please download the source code of the sample application from my GitHub repository.

To keep the discussion focussed, I will explain the critical portions of the application, which will help you understand the code and customize it as per your needs.

Application Logic

Let’s begin developing the application’s logic. We previously discussed the approach that we will take at a high level. Usually, like writing the pseudocode of a complex implementation before writing the actual code. Please read the pseudocode of the application logic that we will implement next.

For processing messages in a session, all Service Bus SDKs support an asynchronous handler-based model. When a message on a session that is locked by the consumer is received, the Handle method is invoked to process the message. In our case, the message handler records the time when it received the message so that the asynchronous routine can determine whether the handler is actively processing messages.

Implementation

Let’s begin with installing the packages used by the sample application, which is a Go module. The application uses the following dependencies that you can install with the go get command:

  1. Azure Service Bus SDK: github.com/Azure/azure-service-bus-go
  2. GoDotEnv to load environment variables from a .env file: github.com/joho/godotenv

Alternatively, you can download the source code and execute the command go mod download to install the packages used by the module.

I used the Message Session example in the official documentation of the Azure Service Bus Go package as the base of the application. I removed several features from the sample, such as the message sender and the queue creation operation, to keep the implementation concise. In the main method, you will find the following implementation of the pseudocode that we discussed previously.

#azure

What is GEEK

Buddha Community

Implementing Multi-Session Sequential Convoy Pattern with Azure Service Bus and Go
Evelyn  Lucy

Evelyn Lucy

1619096139

Multi Service App Development | On-Demand Multi-Services App Solution

Having a multi-services app would help you to excel in your on-demand services business. An on-demand app solution empowers entrepreneurs to offer multiple on-demand services in a single app. In short, this is an efficient app to run a business successfully. If you are an entrepreneur who plans to start a business with the multi-services app, go forward with the multi-service app development.

What are the multiple services offered in the on-demand multi-services app?

Services are categorized as follows.
Ride services – Taxi ride, Moto ride, Car rental, and Moto rental.
Delivery services – Food delivery, Courier delivery, Logistics delivery, Grocery delivery, Medicine delivery, Flower delivery, Fuel delivery, and Plant delivery.
Other services – Plumber, Electrician, Car wash, Fitness, Handyman, Car repair, and beauty services.

Apart from these, you can consider integrating several other services while developing your app.

3 Significant reasons to invest in the on-demand multi-services app

The first and foremost reason why customers use this app is the on-demand multi-service on one platform. Usually, people do not like to install so many apps for availing various services. Instead, they can have a single app for that. This is the reason why the demand for such apps is high.

Next, the incurred cost is less in this app when compared to the single service app. With the seamless navigation feature, customers can easily avail of any services with just a few taps.

Thirdly, they feel more convenient in availing themselves various services in one platform.

Future scope of the multi-service industry

There are 7.6 million users for the multi-service apps in 2019. Recently, the demand for such apps is high considerably due to the covid-19 pandemic. It is expected to flourish more in the future. By 2023, this industry will hit 161.74 billion. This is so inspiring and so many entrepreneurs plan to propel into this industry in 2021.

Consider the following aspects for multi-service app development

Integrate the Multilingual and Multiple currencies features

Never let language be a barrier to your business. Therefore, incorporate the multilingual feature so that customers can use the app in their languages.

The global launch will help you to have a more extensive user base for your app. Just like language, do not let the currency restrict you from launching your app across many countries.

User-friendly design

The UI/UX of the app has to be simple and appealing. This plays a vital role in gaining more customers. If the design is not user-friendly and unimpressive, they won’t prefer your app for the next time. Instead, they prefer using your competitors’ app for availing multiple services. To gain new customers and to retain the existing customers, focus on the app design.

App platform

To cover all the audiences, consider launching the app on both the Android and iOS platforms. Decide on which platform you will launch the app based on your target audience.

White-label solution

It is a known fact that making an app from scratch needs more time and requires a considerable amount of money. On the counter side, creating the app using the white-label solution is budget-friendly and time-conserving. Because, it is a readily available solution. Upon making modifications, you can launch the app instantly. Being the customizable solution, any new features can be incorporated based on the requirements.

Wrap up,

The decision of starting a business with the on-demand multi-services app is good as the market will flourish further in the upcoming days. Take away the points from this blog to withstand in the highly competitive market. Reach out to Uberlikeapp for multi-services app development. We provide a customizable app solution that is scalable based on your needs.

#on demand multi services app #multi services app development #multi service booking app #on-demand service app clone #on-demand multi-services app solution

Adaline  Kulas

Adaline Kulas

1594162500

Multi-cloud Spending: 8 Tips To Lower Cost

A multi-cloud approach is nothing but leveraging two or more cloud platforms for meeting the various business requirements of an enterprise. The multi-cloud IT environment incorporates different clouds from multiple vendors and negates the dependence on a single public cloud service provider. Thus enterprises can choose specific services from multiple public clouds and reap the benefits of each.

Given its affordability and agility, most enterprises opt for a multi-cloud approach in cloud computing now. A 2018 survey on the public cloud services market points out that 81% of the respondents use services from two or more providers. Subsequently, the cloud computing services market has reported incredible growth in recent times. The worldwide public cloud services market is all set to reach $500 billion in the next four years, according to IDC.

By choosing multi-cloud solutions strategically, enterprises can optimize the benefits of cloud computing and aim for some key competitive advantages. They can avoid the lengthy and cumbersome processes involved in buying, installing and testing high-priced systems. The IaaS and PaaS solutions have become a windfall for the enterprise’s budget as it does not incur huge up-front capital expenditure.

However, cost optimization is still a challenge while facilitating a multi-cloud environment and a large number of enterprises end up overpaying with or without realizing it. The below-mentioned tips would help you ensure the money is spent wisely on cloud computing services.

  • Deactivate underused or unattached resources

Most organizations tend to get wrong with simple things which turn out to be the root cause for needless spending and resource wastage. The first step to cost optimization in your cloud strategy is to identify underutilized resources that you have been paying for.

Enterprises often continue to pay for resources that have been purchased earlier but are no longer useful. Identifying such unused and unattached resources and deactivating it on a regular basis brings you one step closer to cost optimization. If needed, you can deploy automated cloud management tools that are largely helpful in providing the analytics needed to optimize the cloud spending and cut costs on an ongoing basis.

  • Figure out idle instances

Another key cost optimization strategy is to identify the idle computing instances and consolidate them into fewer instances. An idle computing instance may require a CPU utilization level of 1-5%, but you may be billed by the service provider for 100% for the same instance.

Every enterprise will have such non-production instances that constitute unnecessary storage space and lead to overpaying. Re-evaluating your resource allocations regularly and removing unnecessary storage may help you save money significantly. Resource allocation is not only a matter of CPU and memory but also it is linked to the storage, network, and various other factors.

  • Deploy monitoring mechanisms

The key to efficient cost reduction in cloud computing technology lies in proactive monitoring. A comprehensive view of the cloud usage helps enterprises to monitor and minimize unnecessary spending. You can make use of various mechanisms for monitoring computing demand.

For instance, you can use a heatmap to understand the highs and lows in computing visually. This heat map indicates the start and stop times which in turn lead to reduced costs. You can also deploy automated tools that help organizations to schedule instances to start and stop. By following a heatmap, you can understand whether it is safe to shut down servers on holidays or weekends.

#cloud computing services #all #hybrid cloud #cloud #multi-cloud strategy #cloud spend #multi-cloud spending #multi cloud adoption #why multi cloud #multi cloud trends #multi cloud companies #multi cloud research #multi cloud market

Implementing Multi-Session Sequential Convoy Pattern with Azure Service Bus and Go

In many event-driven applications, preserving the sequence of events is essential. For example, an event-driven eCommerce application might have the following states, transitions, and events.

  1. A user adds N items to the basket. This action generates the item added event.
  2. The user checks out the basket. This action generates the basket checked out event.
  3. The user pays for the items. This action generates the payment made event.
  4. Inventory decrements the count of available items by N. This action generates the inventory updated event.

For such an application to function correctly, it is critical to maintain the sequence of events. For instance, processing the payment made event before the basket checked out event might lead to errors in billing or inventory systems. In a horizontally scalable system, we can not guarantee sequential processing of messages without creating groups of messages so that the service bus always delivers the messages within a group in sequence. The message consumer must finish processing a message and update the status of the message to the service bus before the next message in the sequence is delivered to it. Azure Service Bus supports the concept of the grouping of related messages through the message sessions feature.

Processing a set of related messages in the First In First Out (FIFO) order is documented as a pattern named the Sequential Convoy pattern in the Azure architecture pattern guide. I encourage you to familiarize yourself with the pattern’s details, including the use cases, to ensure that the architecture pattern is a good fit for your scenario.

Multi-Session Sequential Convoy

Using Azure Service Bus sessions, you can group related messages so that the message consumer locks a session and processes messages in the session one at a time. Processing messages in a session requires the message consumer to know when to close the session. When the consumer receives the last message of the session from the Service Bus, it closes the session and moves on to process messages of another session. There are a few approaches that you can consider to define the last message of the session so that the client can instantly identify it when it arrives.

  1. Producer and consumer agree on the name of the last event/message that the producer will specify either as a message property or in the message body.
  2. The producer uses a custom message property to inform the consumer that the message is the last one in the session.

As you can imagine, establishing such contracts makes the producer and consumer somewhat inflexible. Any change that extends the end of the session will require changes in the contracts and the consumer. Let’s solve this problem.

In most cases, our only requirement is that the messages in a session must be processed in sequence irrespective of the consumer instance that processes it. If we do not attach a consumer instance to a session, we can fix the session termination problem by adding a processing timeout period to the consumer. Following is how this approach will work.

  1. The consumer locks any available session on the Service Bus.
  2. The consumer records the system time when it receives a message from the session in a local timestamp variable and starts processing the message.
  3. An asynchronous routine in the consumer periodically reads the latest timestamp value and closes the session if it determines that the consumer has been idle for some time.
  4. The consumer again attempts to lock any available session on the Service Bus and reiterates the workflow.

What if the producer adds more messages to the session after the session is closed by a consumer instance? In that case, another consumer instance will receive and process them when it is available.

Azure Service Bus has a Golang SDK, which I will use to implement the pattern. Feel free to use the concept to implement the pattern using any supported Azure SDK programming languages. Note that Azure Functions already implement this pattern, so use the built-in Service Bus bindings for processing session messages in Functions.

Create Azure Resources

We need a Service Bus namespace and a queue in the namespace for this demo. You can either use the Azure management portal or use the following Azure CLI commands to create the resources. Note that only the standard and premium tiers of Service Bus support sessions. If you plan to use the Azure management portal for provisioning the resources, remember to set the Enable Sessions flag in the list of options to create the queue.

We will create a resource group namedmyresourcegroup, a Service Bus namespace named rahulr, and a queue named SessionQwith the following commands:

Shell

Record the connection string of the namespace that you created. Let’s now develop the application that reads messages from the session aware queue: SessionQ.

Source Code

Please download the source code of the sample application from my GitHub repository.

To keep the discussion focussed, I will explain the critical portions of the application, which will help you understand the code and customize it as per your needs.

Application Logic

Let’s begin developing the application’s logic. We previously discussed the approach that we will take at a high level. Usually, like writing the pseudocode of a complex implementation before writing the actual code. Please read the pseudocode of the application logic that we will implement next.

For processing messages in a session, all Service Bus SDKs support an asynchronous handler-based model. When a message on a session that is locked by the consumer is received, the Handle method is invoked to process the message. In our case, the message handler records the time when it received the message so that the asynchronous routine can determine whether the handler is actively processing messages.

Implementation

Let’s begin with installing the packages used by the sample application, which is a Go module. The application uses the following dependencies that you can install with the go get command:

  1. Azure Service Bus SDK: github.com/Azure/azure-service-bus-go
  2. GoDotEnv to load environment variables from a .env file: github.com/joho/godotenv

Alternatively, you can download the source code and execute the command go mod download to install the packages used by the module.

I used the Message Session example in the official documentation of the Azure Service Bus Go package as the base of the application. I removed several features from the sample, such as the message sender and the queue creation operation, to keep the implementation concise. In the main method, you will find the following implementation of the pseudocode that we discussed previously.

#azure

Azure Series #2: Single Server Deployment (Input)

In the previous article, we discussed the Gateway to your single server deployment (example: webserver). In this section, we shall continue with Input and Core Infrastructure.

Input for single-server deployment

When you talk about Data for your organization, it covers all three things, “People, Process, and Technology”. More details for the “Streaming and Sourcing Layer” can be found in a separate section (will update the link soon).

**_People: The Who. _**Producers and Consumers of data.

**_Process: The How. _**How the data is curated and put to use.

**_Technology: The What: _**What technologies are used to fetch, process, pass on and store.

Data: While People, Process and Technology is the golden triangle, if you think about it, the very reason the entire state-of-the-art ecosystem exists is merely to get the raw data to a usable form.

1. Data catalog

Any great state-of-art ecosystem is a waste if the data in need for consumers cannot be discovered and from the Producers side, if data cannot be documented/tagged properly that makes it useable for the consumers or end-users. Azure Data Catalog helps to bridge this gap of making the data correctly discoverable by fixing the traditional problems for both consumers and producers and also helps organizations to get the best value out of their existing information assets.

2. Streaming

While we will discuss more as part of the sourcing section, we shall cover the basics of streaming.

1/ Queue Storage

2/ Service Bus

3/ Event Hubs

4/ Event Grid

#azure-interview #azure-event-grid #azure-event-hub #azure #azure-service-bus

Jolie  Reichert

Jolie Reichert

1596948420

How to Use Azure Go SDK to Manage Azure Data Explorer Clusters

Getting started with Azure Data Explorer using the Go SDK covered how to use the Azure Data Explorer Go SDK to ingest and query data from azure data explorer to ingest and query data. In this blog you will the Azure Go SDK to manage Azure Data Explorer clusters and databases.

Azure Data Explorer (also known as Kusto) is a fast and scalable data exploration service for analyzing large volumes of diverse data from any data source, such as websites, applications, IoT devices, and more. This data can then be used for diagnostics, monitoring, reporting, machine learning, and additional analytics capabilities.

In case you’re wondering, we are talking about two different SDKs here. The one covered in this blog is for resource administration (also known as the control plane SDK) and the the one I used in the other post is data plane SDK for interacting with the Azure Data Explorer service itself (ingestion, query etc.)

What’s Covered?

A simple CLI application is used as an example to demonstrate how to use the Go SDK. We’ll try out the application first and go through how to:

  • Create and list Azure Data Explorer clusters
  • Create and list databases in that cluster
  • Delete the database and cluster

Once that’s done, we’ll walk through the sample code to understand what’s going on

The code is available on GitHub https://github.com/abhirockzz/azure-go-sdk-for-dataexplorer

Please note that this CLI based example is just meant to showcase how to use the Azure Go SDK (in the context of Azure Data Explorer) as a part of a larger application. It is not supposed to replace/substitute the Azure CLI which can be used to manage Azure Data Explorer resources

Pre-requisites

Install Go 1.13 or above

You will need a Microsoft Azure account. Go ahead and sign up for a free one!

Install the Azure CLI if you don’t have it already (should be quick!)

#tutorial #big data #azure #analytics #go #golang #azure data explorer clusters #azure go sdk