Message Queues play an important role in large-scale solutions, as the project becomes more and more sophisticated — micro-services are added and we become dependant of the message-queue functionality.

One day we wake up and discover that current message-queue implementation doesn’t meet our changing needs, we change it — and we fall into the endless refactor and regression tests loop.

So can we devise a framework that allows us to change the implementation without affecting the whole project?

In this post, I would share with you a framework design that enables just that, as I’ve developed in the final project of Dan Gittik’s great course “Advanced System Design” at Tel-Aviv University.

This post would be the first in a series in which I share with you some flexible yet SOLID and durable solutions to design challenges.

Detaching Message-Queue from its Implementation

The most important goal of the framework, is to enable us to change message-queue implementation (i.e. RabbitMQ to Kafka), with ease.

We want that such change would be friendly and require a minimal code change and that validating the architectural specifications would be understandable for non-coders too.

So, we end up creating a configuration file — just plug & play!

Such a configuration file would describe how to create for each micro-service it’s implementation message-queue:

• Publishers (which enable us to send messages to the message-queue)
• Consumers (which enable us to receive messages from the message queue)

The configuration file structure would be as the following:

## Message Queue configuration version tag.
	version: "1"

	## Default Header
	message_queue:
	    ## Type of micro-service (server, parser, saver, etc...)
	    micro_service_type:
	        ## Category : Publishers
	        publishers:
	            ## Item
	            - publisher_1:
	                ## Fields required to initialize implementation (Context fields values)
	                [field_1]   : [value_1]
	                [field_2]   : [value_2]
	                [field_3]   : [value_3]

	        ## Category : Consumers
	        consumers:   
	            ## Item
	            - consumer_1:
	                ## Fields required to initialize implementation (Context fields values)
	                [field_1]   : [value_1]
	                [field_2]   : [value_2]
	                [field_3]   : [value_3]

Publishers and Consumers describe the fields required to create their specific implementation (i.e. Specific RabbitMQ publisher/consumer exchange), those fields recipes are called “Context”.

NOTE: This is an example of a YAML configuration file, yet it could be any dictionary-structured file — as I’ve implemented in the project.

NOTE: An example of a RabbitMQ MessageQueue configuration file available here.

#software-design #microservices #programming #scale #software-development