flink pipeline from kafka to redis

flink pipeline from kafka to redis

I'm using flink to read from kafka and write to redis.

I'm using flink to read from kafka and write to redis.

For test, I just want to read the first 10 messages from kafka. So I use a counter and try to stop the consumer when the counter = 10

    AtomicInteger counter = new AtomicInteger(0);

FlinkKafkaConsumer08<String> kafkaConsumer =
        new FlinkKafkaConsumer08<>("my topic",
                new SimpleStringSchema() {
                    @Override
                    public boolean isEndOfStream(String nextElement) {
                        // It should only read 10 kafka message
                        return counter.getAndIncrement() > 9;
                    }
                },
                properties);

but I get 30 message in redis:

llen rtp:example
(integer) 30

When I change the condition to counter.getAndIncrement() > 8, it writes 27 messages to redis. Always triple.

Complete code:

public class FlinkEntry {

private final static JedisCluster JEDIS_CLUSTER;

static {
    Set<HostAndPort> hostAndPorts = new HashSet<>();
    hostAndPorts.add(new HostAndPort("localhost", 7001));
    JEDIS_CLUSTER = new JedisCluster(hostAndPorts);
}


public static void main(String[] args) throws Exception {
    StreamExecutionEnvironment environment = StreamExecutionEnvironment.getExecutionEnvironment();

    FlinkKafkaConsumer08<String> kafkaConsumer = createKafkaConsumer();
    DataStream<String> dataStream = environment.addSource(kafkaConsumer);

    SinkFunction<String> redisSink = createRedisSink();
    dataStream.addSink(redisSink);

    environment.execute();
}

private static FlinkKafkaConsumer08<String> createKafkaConsumer() {
    Properties properties = new Properties();
    //... set kafka property

    AtomicInteger counter = new AtomicInteger(0);

    FlinkKafkaConsumer08<String> kafkaConsumer =
            new FlinkKafkaConsumer08<>("my topic",
                    new SimpleStringSchema() {
                        @Override
                        public boolean isEndOfStream(String nextElement) {
                            // It should only read 10 kafka message
                            return counter.getAndIncrement() > 9;
                        }
                    },
                    properties);

    kafkaConsumer.setStartFromLatest();
    return kafkaConsumer;
}

private static SinkFunction<String> createRedisSink() {
    return new SinkFunction<String>() {
        @Override
        public void invoke(String value, Context context) {
            JEDIS_CLUSTER.lpush("rtp:example", value);
            JEDIS_CLUSTER.expire("rtp:example", 10 * 60);
        }
    };
}

}


java

Bootstrap 5 Complete Course with Examples

Bootstrap 5 Tutorial - Bootstrap 5 Crash Course for Beginners

Nest.JS Tutorial for Beginners

Hello Vue 3: A First Look at Vue 3 and the Composition API

Building a simple Applications with Vue 3

Deno Crash Course: Explore Deno and Create a full REST API with Deno

How to Build a Real-time Chat App with Deno and WebSockets

Convert HTML to Markdown Online

HTML entity encoder decoder Online

How to Install OpenJDK 11 on CentOS 8

What is OpenJDK? OpenJDk or Open Java Development Kit is a free, open-source framework of the Java Platform, Standard Edition (or Java SE).

Java Core (2020) | Java for beginners | Brush up your Java Skills | Get Hired

In this video we will learn about all the major concepts that come under Java Core . The course is very carefully designed and is made with keeping simplicit...

Java File Class Tutorial | Java.io File Class in Java Example

Java File class represents the path of directories and files. It provides the methods for renaming, deleting, and obtaining the properties of file or directory.

Best Ways To Improve Skills In Java and Python Programming

Join CETPA, India's best institute for Java online training and Certification Course. Enroll now to learn Core to Advanced Java & avail 50% discount on Java Online Course with Certificate.

Java Fundamentals: Learn Java for absolute beginners |Simpliv

Java Fundamentals: Learn Java for absolute beginners