Use Apache Kafka With Laravel

Laravel Kafka

Do you use Kafka in your laravel packages? All packages I've seen until today, including some built by myself, does not provide a nice syntax usage syntax or, if it does, the test process with these packages are very painful.

This package provides a nice way of producing and consuming kafka messages in your Laravel projects.

Follow these docs to install this package and start using kafka with ease.

Installation

To install this package, you must have installed PHP RdKafka extension. You can follow the steps here to install rdkafka in your system.

With RdKafka installed, require this package with composer:

composer require mateusjunges/laravel-kafka

Usage

After installing the package, you can start producing and consuming Kafka messages.

Producing Kafka Messages

To publish your messages to Kafka, you can use the publishOn method, of Junges\Kafka\Facades\Kafka class:

use Junges\Kafka\Facades\Kafka;

Kafka::publishOn('broker', 'topic');

This method returns a Junges\Kafka\Producers\ProducerBuilder::class instance, and you can configure your message.

The ProducerBuilder class contains a few methods to configure your kafka producer. The following lines describes these methods.

ProducerBuilder configuration methods

The withConfigOption method sets a \RdKafka\Conf::class option. You can check all available options here. This methods set one config per call, and you can use withConfigOptions passing an array of config name and config value as argument. Here's an example:

use Junges\Kafka\Facades\Kafka;

Kafka::publishOn('broker', 'topic')
    ->withConfigOption('property-name', 'property-value')
    ->withConfigOptions([
        'property-name' => 'property-value'
    ]);

While you are developing your application, you can enable debug with the withDebugEnabled method. To disable debug mode, you can use ->withDebugEnabled(false), or withDebugDisabled methods.

use Junges\Kafka\Facades\Kafka;

Kafka::publishOn('broker', 'topic')
    ->withConfigOption('property-name', 'property-value')
    ->withConfigOptions([
        'property-name' => 'property-value'
    ])
    ->withDebugEnabled() // To enable debug mode
    ->withDebugDisabled() // To disable debug mode
    ->withDebugEnabled(false) // Also to disable debug mode

Configuring the Kafka message payload

In kafka, you can configure your payload with a message, message headers and message key. All these configurations are available within ProducerBuilder class.

Configuring message headers

To configure the message headers, use the withHeaders method:

use Junges\Kafka\Facades\Kafka;

Kafka::publishOn('broker', 'topic')
    ->withHeaders([
        'header-key' => 'header-value'
    ])

Configure the message payload

You can configure the message with the withMessage or withMessageKey methods.

The withMessage sets the entire message, and it accepts a Junges\Kafka\Message::class instance as argument.

This is how you should use it:

use Junges\Kafka\Facades\Kafka;
use Junges\Kafka\Message;

$message = new Message(
    headers: ['header-key' => 'header-value'],
    message: ['key' => 'value'],
    key: 'kafka key here'  
)

Kafka::publishOn('broker', 'topic')->withMessage($message);

The withMessageKey method sets only a key in your message.

use Junges\Kafka\Facades\Kafka;

Kafka::publishOn('broker', 'topic')->withMessageKey('key', 'value');

Using Kafka Keys

In Kafka, keys are used to determine the partition within a log to which a message get's appended to. If you want to use a key in your message, you should use the withKafkaKey method:

use Junges\Kafka\Facades\Kafka;

Kafka::publishOn('broker', 'topic')->withKafkaKey('your-kafka-key');

Sending the message to Kafka

After configuring all your message options, you must use the send method, to send the message to kafka.

use Junges\Kafka\Facades\Kafka;

/** @var \Junges\Kafka\Producers\ProducerBuilder $producer */
$producer = Kafka::publishOn('broker', 'topic')
    ->withConfigOptions(['key' => 'value'])
    ->withKafkaKey('your-kafka-key')
    ->withKafkaKey('kafka-key')
    ->withHeaders(['header-key' => 'header-value']);

$producer->send();

Consuming Kafka Messages

If your application needs to read messages from a Kafka topic, you must create a consumer object, subscribe to the appropriate topic and start receiving messages.

To create a consumer using this package, you can use the createConsumer method, on Kafka facade:

use Junges\Kafka\Facades\Kafka;

$consumer = Kafka::createConsumer('broker');

This method returns a Junges\Kafka\Consumers\ConsumerBuilder::class instance, and you can use it to configure your consumer.

Subscribing to a topic

With a consumer created, you can subscribe to a kafka topic using the subscribe method:

use Junges\Kafka\Facades\Kafka;

$consumer = Kafka::createConsumer('broker')->subscribe('topic');

Of course, you can subscribe to more than one topic at once, either using an array of topics or specifying one by one:

use Junges\Kafka\Facades\Kafka;

$consumer = Kafka::createConsumer('broker')->subscribe('topic-1', 'topic-2', 'topic-n');

// Or, using array:
$consumer = Kafka::createConsumer('broker')->subscribe([
    'topic-1',
    'topic-2',
    'topic-n'
]);

Configuring consumer groups

Kafka consumers belonging to the same consumer group share a group id. THe consumers in a group divides the topic partitions as fairly amongst themselves as possible by establishing that each partition is only consumed by a single consumer from the group.

To attach your consumer to a consumer group, you can use the method withConsumerGroupId to specify the consumer group id:

use Junges\Kafka\Facades\Kafka;

$consumer = Kafka::createConsumer('broker')->withConsumerGroupId('foo');

Configuring message handlers

Now that you have created your kafka consumer, you must create a handler for the messages this consumer receives. By default, a consumer is any callable. You can use an invokable class or a simple callback. Use the withHandler method to specify your handler:

$consumer = \Junges\Kafka\Facades\Kafka::createConsumer('broker');

// Using callback:
$consumer->withHandler(function(\RdKafka\Message $message) {
    // Handle your message here
});

Or, using a invokable class:

class Handler
{
    public function __invoke(\RdKafka\Message $message){
        // Handle your message here
    }
}

$consumer = \Junges\Kafka\Facades\Kafka::createConsumer('broker')->withHandler(Handler::class)

Configuring max messages to be consumed

If you want to consume a limited amount of messages, you can use the withMaxMessages method to set the max number of messages to be consumed by a kafka consumer:

$consumer = \Junges\Kafka\Facades\Kafka::createConsumer('broker')->withMaxMessages(2);

Configuring a dead letter queue

In kafka, a Dead Letter Queue (or DLQ), is a simple kafka topic in the kafka cluster which acts as the destination for messages that were not able to make it to the desired destination due to some error.

To create a dlq in this package, you can use the withDlq method. If you don't specify the DLQ topic name, it will be created based on the topic you are consuming, adding the -dlq suffix to the topic name.

$consumer = \Junges\Kafka\Facades\Kafka::createConsumer('broker')->withDlq();

//Or, specifying the dlq topic name:
$consumer = \Junges\Kafka\Facades\Kafka::createConsumer('broker')->withDlq('your-dlq-topic-name')

Using SASL

SASL allows your producers and your consumers to authenticate to your Kafka cluster, which verifies their identity. It's also a secure way to enable your clients to endorse an identity. To provide SASL configuration, you can use the withSasl method, passing a Junges\Kafka\Config\Sasl instance as the argument:

$consumer = \Junges\Kafka\Facades\Kafka::createConsumer('broker')
    ->withSasl(new \Junges\Kafka\Config\Sasl(
        password: 'password',
        username: 'username'
        mechanisms: 'authentication mechanism'
    ));

Using middlewares

Middlewares provides a convenient way to filter and inspecting your Kafka messages. To write a middleware in this package, you can use the withMiddleware method. The middleware is a callable in which the first argument is the message itself and the second one is the next handler. The middlewares get executed in the order they are defined,

$consumer = \Junges\Kafka\Facades\Kafka::createConsumer('broker')
    ->withMiddleware(function($message, callable $next) {
        // Perform some work here
        return $next($message);
    });

Using auto commit

The auto-commit check is called in every poll and it checks that the time elapsed is greater than the configured time. To enable auto commit, use the withAutoCommit method:

$consumer = \Junges\Kafka\Facades\Kafka::createConsumer('broker')->withAutoCommit();

Setting Kafka configuration options

To set configuration options, you can use two methods: withOptions, passing an array of option and option value or, using the `withOption method and passing two arguments, the option name and the option value.

$consumer = \Junges\Kafka\Facades\Kafka::createConsumer('broker')
    ->withOptions([
        'option-name' => 'option-value'
    ]);
// Or:
$consumer = \Junges\Kafka\Facades\Kafka::createConsumer('broker')
    ->withOption('option-name', 'option-value');

Building the consumer

When you have finished configuring your consumer, you must call the build method, which returns a Junges\Kafka\Consumers\Consumer instance.

$consumer = \Junges\Kafka\Facades\Kafka::createConsumer('broker')
    // Configure your consumer here
    ->build();

Consuming the kafka messages

After building the consumer, you must call the consume method to consume the messages:

$consumer = \Junges\Kafka\Facades\Kafka::createConsumer()->build();

$consumer->consume();

Using Kafka::fake()

When testing your application, you may wish to "mock" certain aspects of the app, so they are not actually executed during a given test. This package provides convenient helpers for mocking the kafka producer out of the box. These helpers primarily provide a convenience layer over Mockery so you don't have to manually make complicated Mockery method calls.

The Kafka facade also provides methods to perform assertions over published messages, such as assertPublished, assertPublishedOn and assertNothingPublished.

use Junges\Kafka\Facades\Kafka;
use PHPUnit\Framework\TestCase;

class MyTest extends TestCase
{
     public function testMyAwesomeApp()
     {
         Kafka::fake();
         
         $producer = Kafka::publishOn('broker', 'topic')
             ->withHeaders(['key' => 'value'])
             ->withMessageKey('foo', 'bar');
             
         $producer->send();
             
         Kafka::assertPublished($producer->getMessage());       
     }
}

If you want to assert that a message was published in a specific kafka topic, you can use the assertPublishedOn method:

use PHPUnit\Framework\TestCase;
use Junges\Kafka\Facades\Kafka;

class MyTest extends TestCase
{
    public function testWithSpecificTopic()
    {
        Kafka::fake();
        
        $producer = Kafka::publishOn('broker', 'some-kafka-topic')
            ->withHeaders(['key' => 'value'])
            ->withMessageKey('key', 'value');
            
        $producer->send();
        
        Kafka::assertPublishedOn('some-kafka-topic', $producer->getMessage());
    }
}

You can also use a callback function to perform assertions within the message using a callback in which the argument is the published message itself.

use PHPUnit\Framework\TestCase;
use Junges\Kafka\Facades\Kafka;
use Junges\Kafka\Message;

class MyTest extends TestCase
{
    public function testWithSpecificTopic()
    {
        Kafka::fake();
        
        $producer = Kafka::publishOn('broker', 'some-kafka-topic')
            ->withHeaders(['key' => 'value'])
            ->withMessageKey('key', 'value');
            
        $producer->send();
        
        Kafka::assertPublishedOn('some-kafka-topic', $producer->getMessage(), function(Message $message) {
            return $message->getHeaders()['key'] === 'value';
        });
    }
} 

You can also assert that nothing was published at all, using the assertNothingPublished:

use PHPUnit\Framework\TestCase;
use Junges\Kafka\Facades\Kafka;
use Junges\Kafka\Message;

class MyTest extends TestCase
{
    public function testWithSpecificTopic()
    {
        Kafka::fake();
        
        if (false) {
            $producer = Kafka::publishOn('broker', 'some-kafka-topic')
                ->withHeaders(['key' => 'value'])
                ->withMessageKey('key', 'value');
                
            $producer->send();
        }
        
        Kafka::assertNothingPublished();
    }
} 

Testing

Run composer test to test this package.

Contributing

Thank you for considering contributing for the Laravel ACL package! The contribution guide can be found here.

Credits

Download Details:
Author: mateusjunges
Download Link: Download The Source Code
Official Website: https://github.com/mateusjunges/laravel-kafka
License: MIT

#laravel #php #webdev #apachekafka #kafka 

What is GEEK

Buddha Community

Use Apache Kafka With Laravel
Seamus  Quitzon

Seamus Quitzon

1595201363

Php how to delete multiple rows through checkbox using ajax in laravel

First thing, we will need a table and i am creating products table for this example. So run the following query to create table.

CREATE TABLE `products` (
 `id` int(10) unsigned NOT NULL AUTO_INCREMENT,
 `name` varchar(255) COLLATE utf8mb4_unicode_ci NOT NULL,
 `description` varchar(255) COLLATE utf8mb4_unicode_ci DEFAULT NULL,
 `created_at` timestamp NULL DEFAULT CURRENT_TIMESTAMP,
 `updated_at` datetime DEFAULT NULL,
 PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=7 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_unicode_ci

Next, we will need to insert some dummy records in this table that will be deleted.

INSERT INTO `products` (`name`, `description`) VALUES

('Test product 1', 'Product description example1'),

('Test product 2', 'Product description example2'),

('Test product 3', 'Product description example3'),

('Test product 4', 'Product description example4'),

('Test product 5', 'Product description example5');

Now we are redy to create a model corresponding to this products table. Here we will create Product model. So let’s create a model file Product.php file under app directory and put the code below.

<?php

namespace App;

use Illuminate\Database\Eloquent\Model;

class Product extends Model
{
    protected $fillable = [
        'name','description'
    ];
}

Step 2: Create Route

Now, in this second step we will create some routes to handle the request for this example. So opeen routes/web.php file and copy the routes as given below.

routes/web.php

Route::get('product', 'ProductController@index');
Route::delete('product/{id}', ['as'=>'product.destroy','uses'=>'ProductController@destroy']);
Route::delete('delete-multiple-product', ['as'=>'product.multiple-delete','uses'=>'ProductController@deleteMultiple']);

#laravel #delete multiple rows in laravel using ajax #laravel ajax delete #laravel ajax multiple checkbox delete #laravel delete multiple rows #laravel delete records using ajax #laravel multiple checkbox delete rows #laravel multiple delete

I am Developer

1597487833

Country State City Drop Down List using Ajax in Laravel

Here, i will show you how to create dynamic depedent country state city dropdown list using ajax in laravel.

Country State City Dropdown List using Ajax in php Laravel

Follow Below given steps to create dynamic dependent country state city dropdown list with jQuery ajax in laravel:

  • Step 1: Install Laravel App
  • Step 2: Add Database Details
  • Step 3: Create Country State City Migration and Model File
  • Step 4: Add Routes For Country State City
  • Step 5: Create Controller For Fetch Country State City
  • Step 6: Create Blade File For Show Dependent Country State City in Dropdown
  • Step 7: Run Development Server

https://www.tutsmake.com/ajax-country-state-city-dropdown-in-laravel/

#how to create dynamic dropdown list using laravel dynamic select box in laravel #laravel-country state city package #laravel country state city drop down #dynamic dropdown country city state list in laravel using ajax #country state city dropdown list using ajax in php laravel #country state city dropdown list using ajax in laravel demo

How to Send E-mail Using Queue in Laravel 7/8

Today I will show you How to Send E-mail Using Queue in Laravel 7/8, many time we can see some process take more time to load like payment gateway, email send, etc. Whenever you are sending email for verification then it load time to send mail because it is services. If you don’t want to wait to user for send email or other process on loading server side process then you can use queue.

Read More : How to Send E-mail Using Queue in Laravel 7/8

https://websolutionstuff.com/post/how-to-send-e-mail-using-queue-in-laravel-7-8


Read Also : Send Mail Example In Laravel 8

https://websolutionstuff.com/post/send-mail-example-in-laravel-8

#how to send e-mail using queue in laravel 7/8 #email #laravel #send mail using queue in laravel 7 #laravel 7/8 send mail using queue #laravel 7/8 mail queue example

I am Developer

1597469369

Crop and Resize Image Before Upload In Laravel Using with jQuery Copper JS

Crop and resize image size before upload in laravel using jquery copper js. In this post, i will show you how to crop and resize image size in laravel using jQuery copper js in laravel.

This laravel crop image before upload using cropper js looks like:

laravel crop image before upload

Laravel Crop Image Before Uploading using Cropper js Tutorial

Laravel crop image before upload tutorial, follow the following steps and learn how to use cropper js to crop image before uploading in laravel app:

  • Step 1: Install New Laravel App
  • Step 2: Add Database Details
  • Step 3: Create Migration & Model
  • Step 4: Add Route
  • Step 5: Create Controller By Artisan
  • Step 6: Create Blade View
  • Step 7: Make Upload Directory
  • Step 8: Start Development Server

Read More => https://www.tutsmake.com/laravel-crop-image-before-upload-using-jquery-copper-js/

Live Demo Laravel Crop image Before Upload.

#laravel crop image before upload, #laravel crop and resize image using cropper.js #ajax image upload and crop with jquery and laravel #crop and upload image ajax jquery laravel #crop image while uploading with jquery laravel #image crop and upload using jquery with laravel ajax

How To Create Zip File Using Ziparchive in Laravel

In this tutorial i will show you how to create zip file using zipArchive in laravel. Some times client’s have requirements to have functionalities like create zip file for documentation or images and download it.

So, at that time we can find many laravel packages to perform this task. But here, i am show you to how to create zip file in laravel using zipArchive without any package. Laravel provide ZipArchive class for create zip file in laravel,So i will use ZipArchive in laravel.

How To Create Zip File Using Ziparchive in Laravel

https://websolutionstuff.com/post/how-to-create-zip-file-using-ziparchive-in-laravel

#laravel #ziparchive #how to create zip file using ziparchive in laravel #ziparchive in laravel #make zip in laravel #ziparchive laravel example