Streaming Java CompletableFutures in Completion Order

Streaming Java CompletableFutures in Completion Order

Streaming Java CompletableFutures in Completion Order - Want to see what happens when you combine Stream and CompletableFutures in Java?...

Streaming Java CompletableFutures in Completion Order - Want to see what happens when you combine Stream and CompletableFutures in Java?...

Java 8 brought us tools like CompletableFuture and Stream API… let’s try to combine them both and create a Stream that returns values from a collection of CompletableFutures as they arrive.

This approach was also employed when developing 1.0.0 of parallel-collectors.

Streaming CompletableFutures

Essentially, what are trying to do is to implement a solution that would allow us to convert a collection of futures into a stream of values returned by those futures:

Collection<CompletableFuture<T>> -> Stream<T>

In the world of Java, that could be achieved by using, for example, a static method:

public static <T> Stream<T> inCompletionOrder(Collection<CompletableFuture<T>> futures) {
    // ...
}

To create a custom Stream, one needs to implement a custom java.util.Spliterator:

final class CompletionOrderSpliterator<T> 
  implements Spliterator<T> { ... }

And now, we can finish up with the implementation of our static method:

public static <T> Stream<T> completionOrder(Collection<CompletableFuture<T>> futures) {
    return StreamSupport.stream(
      new CompletionOrderSpliterator<>(futures), false);
}

That’s the easy part, let’s implement the CompletionOrderSpliterator now.

Implementing CompletionOrderSpliterator

To implement our Spliterator, we’ll need to fill in the blanks provide custom implementations of the following methods:

final class CompletionOrderSpliterator<T> implements Spliterator<T> {
    CompletionOrderSpliterator(Collection<CompletableFuture<T>> futures) {
        // TODO
    }
    @Override
    public boolean tryAdvance(Consumer<? super T> action) {
        // TODO
    }
    @Override
    public Spliterator<T> trySplit() {
        // TODO
    }
    @Override
    public long estimateSize() {
        // TODO
    }
    @Override
    public int characteristics() {
        // TODO
    }
}

Naturally, we need a proper constructor as well.

The most natural way of approaching the problem would involve making a working copy of the source collection, waiting for any future to complete, removing it from the collection and feeding it to the <strong>Spliterator</strong> itself.

Waiting for any future to complete can be quickly done using CompletableFuture#anyOf, and it handles exception propagation correctly out of the box.

However, there’s a slight complication.

If you look at the signature of CompletableFuture#anyOf, you will see that it’s not very practical because it accepts multiple CompletableFutures<?> and returns a single CompletableFuture< Object> , but this is not the main issue here (just a slight inconvenience).

The real problem is that the CompletableFuture<Object> returned by the method is not the future that completed first, but a new CompletableFuture instance that completes when any future completes.

This makes the whole idea of waiting for a future and then removing it from a list of remaining futures a bit complicated. We can’t rely on reference equality, so we can either do a linear scan after each signal from <strong>CompletableFuture#anyOf</strong>, or try to come up with something better.

The naive solution could look like:

private T takeNextCompleted() {
    anyOf(futureQueue.toArray(new CompletableFuture[0])).join();
    CompletableFuture<T> next = null;
    for (CompletableFuture<T> future : futureQueue) {
        if (future.isDone()) {
            next = future;
            break;
        }
    }
    futureQueue.remove(next);
    return next.join();
}

I’m doing a linear scan and store the index for the sake of constant-time removal. If you want to know why I’m passing 0 to the CompletableFuture[], although I know what the size is, check this article.

If you look at the problem from a pragmatic point of view, this should be good enough since no one would ever expect to iterate on a collection of futures that’s more than 10-20 thousand in size (although, it's absolutely possible since CompletableFutures are not bound to threads and millions of them can be completed by a single thread) because of the hardware thread-count limitations (actual number can vary a lot depending on multiple factors, for example, stack size).

However, that might change once Project Loom goes live.

Still, 20000 iterations would result in visiting anything between 20000 nodes optimistically(it’s always the first future that completes) to 200000000 nodes pessimistically.

What could we do about it if we can’t rely on referential equality or hashcodes of CompletableFutures?

We could assign our ids to them and store them in a map along with matching futures and then make futures identify themselves by returning indexes alongside actual values by returning a pair.

So, let’s store our futures in a map:

private final Map<Integer, CompletableFuture<Map.Entry<Integer, T>>> indexedFutures;

Now, we could manually assign ids from a monotonically increasing sequence, and make futures return them as well:

private static <T> Map<Integer, CompletableFuture<Map.Entry<Integer, T>>> toIndexedFutures(List<CompletableFuture<T>> futures) {
    Map<Integer, CompletableFuture<Map.Entry<Integer, T>>> map
      = new HashMap<>(futures.size(), 1); // presizing the HashMap since we know the capacity and expected collisions count (0)
    int seq = 0;
    for (CompletableFuture<T> future : futures) {
        int index = seq++;
        map.put(
          index, 
          future.thenApply(
            value -> new AbstractMap.SimpleEntry<>(index, value)));
    }
    return map;
}

And now, we can efficiently find and process the next completed future by waiting for it, reading the sequence number and then using it to remove the future from the list of remaining ones:

private T nextCompleted() {
    return anyOf(indexedFutures.values()
      .toArray(new CompletableFuture[0]))
        .thenApply(result -> ((Map.Entry<Integer, T>) result))
        .thenApply(result -> {
            indexedFutures.remove(result.getKey());
            return result.getValue();
        }).join();
}

Implementation of tryAdvance() becomes trivial:

@Override
public boolean tryAdvance(Consumer<? super T> action) {
    if (!indexedFutures.isEmpty()) {
        action.accept(nextCompleted());
        return true;
    } else {
        return false;
    }
}

The hardest part is behind us, now we need to implement three remaining methods:

@Override
public Spliterator<T> trySplit() {
    return null; // because splitting is not allowed
}
@Override
public long estimateSize() {
    return indexedFutures.size(); // because we know the size
}
@Override
public int characteristics() {
    return 
      SIZED       // because we know the size upfront
      | IMMUTABLE // because the source can be safely modified
      | NONNULL;  // because nulls in source are not accepted
}

And here we are.

Working Example

We can quickly validate that it works appropriately by introducing a random processing lag when going through a sorted sequence:

public static void main(String[] args) {
    ExecutorService executorService = Executors.newFixedThreadPool(10);
    List<CompletableFuture<Integer>> futures = Stream
      .iterate(0, i -> i + 1)
      .limit(100)
      .map(i -> CompletableFuture.supplyAsync(
        withRandomDelay(i), executorService))
      .collect(Collectors.toList());
    completionOrder(futures)
      .forEach(System.out::println);
}
private static Supplier<Integer> withRandomDelay(Integer i) {
    return () -> {
        try {
            Thread.sleep(ThreadLocalRandom.current()
              .nextInt(10000));
        } catch (InterruptedException e) {
            // ignore shamelessly, don't do this on production
        }
        return i;
    };
}

And you can see that values get returned not in the original order:

6
5
2
4
1
11
8
12
3

Streaming Futures in Original Order

What if we want different semantics and simply maintain the original order?

Luckily, that can be achieved quickly without any particular infrastructure:

public static <T> Stream<T> originalOrder(
  Collection<CompletableFuture<T>> futures) {
    return futures.stream().map(CompletableFuture::join);
}

A Complete Example
package com.pivovarit.collectors;
import java.util.AbstractMap;
import java.util.Collection;
import java.util.HashMap;
import java.util.Map;
import java.util.Spliterator;
import java.util.concurrent.CompletableFuture;
import java.util.function.Consumer;
import static java.util.concurrent.CompletableFuture.anyOf;
/**
 * @author Grzegorz Piwowarek
 */
final class CompletionOrderSpliterator<T> implements Spliterator<T> {
    private final Map<Integer, CompletableFuture<Map.Entry<Integer, T>>> indexedFutures;
    CompletionOrderSpliterator(Collection<CompletableFuture<T>> futures) {
        indexedFutures = toIndexedFutures(futures);
    }
    @Override
    public boolean tryAdvance(Consumer<? super T> action) {
        if (!indexedFutures.isEmpty()) {
            action.accept(nextCompleted());
            return true;
        } else {
            return false;
        }
    }
    private T nextCompleted() {
        return anyOf(indexedFutures.values().toArray(new CompletableFuture[0]))
          .thenApply(result -> ((Map.Entry<Integer, T>) result))
          .thenApply(result -> {
              indexedFutures.remove(result.getKey());
              return result.getValue();
          }).join();
    }
    @Override
    public Spliterator<T> trySplit() {
        return null;
    }
    @Override
    public long estimateSize() {
        return indexedFutures.size();
    }
    @Override
    public int characteristics() {
        return SIZED | IMMUTABLE | NONNULL;
    }
    private static <T> Map<Integer, CompletableFuture<Map.Entry<Integer, T>>> toIndexedFutures(Collection<CompletableFuture<T>> futures) {
        Map<Integer, CompletableFuture<Map.Entry<Integer, T>>> map = new HashMap<>(futures.size(), 1);
        int counter = 0;
        for (CompletableFuture<T> f : futures) {
            int index = counter++;
            map.put(index, f.thenApply(value -> new AbstractMap.SimpleEntry<>(index, value)));
        }
        return map;
    }
}

The complete working example can be found on GitHub as well.

Do you have an idea of how to make it better? Don’t hesitate, and let me know!

Apache Spark for Java Developers

Java Fundamentals: Learn Java for absolute beginners |Simpliv

Java Fundamentals: Learn Java for absolute beginners |Simpliv

Java Fundamentals: Learn Java for absolute beginners

Description
This is the best course to learn to program in Java in Spanish from scratch and without any experience in this fabulous programming language. This is the first course where we will study the Fundamentals of Java, and we will take you step by step until you acquire the bases of the Java language and you can start to study more advanced Java topics.

The content is divided into perfectly structured levels, each level supported by the previous one, with the aim of adding Java knowledge incrementally and so you can focus on mastering the issues little by little and gradually. So ensure the success of your Java training.

We will also offer support for any doubts about the didactic material included in this Java Fundamentals course.

We manage a new teaching methodology that we have called Speed ​​Learning. This methodology consists of concise videos that go directly to the point to be studied, complemented by eBooks with explanations and step-by-step images (which you can print, or search for any text you need, or use for your offline study), since As we know we can not do text search within a video. In addition, our methodology includes perfectly structured and very didactic exercises that will allow you to accelerate your eLearning learning. No loss of time in videos where you have to watch the instructor codify an exercise, too much theory, little practice or anything like that. Our Speed ​​Learning methodology guarantees that in the shortest possible time you will acquire the necessary knowledge for the Java professional and working world.

The Java Fundamentals course includes the following topics for study:

Lesson 1 - Starting with Java Technology

The amazing world of Java programming

What is Java technology (from a practical approach)

Our first Java program from scratch

Lesson 2 - Variables and Operators in Java

Use of Variables in Java and what we use them for

Types of Data in Java and how they are classified

Management and Classification of operators in Java

Lesson 3 - Control statements in Java

Using the if-else structure and where to use it

Handling the switch structure and when applying it

Lesson 4 - Handling Loops in Java

Use of the for loop and its use

Using the while loop and how to apply it

Use of the do-while loop and when to use it

Lesson 5 - Object Oriented Programming

Introduction to Object Oriented Programming (OOP)

Handling Classes in Java

Use of Objects in Java

Lesson 6 - Functions in Java

Declaration of Methods or Functions in Java

Use and call of functions in Java

Lesson 7 - Data Management in Java

Using Arrays in Java

Management of Matrices in Java

Lesson 8 - Inheritance in Java

Inheritance Management in Java

Use of superclasses and subclasses in Java

Final Level Laboratory

Final Exercise where everything learned in this Level is integrated

At the end you get a certificate of having completed the Java Fundamentals course.

We wait for you on the other side.

Ing. Ubaldo Acosta

Founder of Global Mentoring

Passion for Java Technology

Who this course is for:

Anyone who wants to learn how to program in Java
Basic knowledge
Basic knowledge of PC use
Basic management of an operating system such as Windows, Mac or Linux
It is not necessary to know how to program, we will start from scratch !!!
The attitude and desire to start coding and learning Java once and for all from scratch!
What will you learn
Have the basics of the programming language with Java
You will know the basic syntax of the Java language
Manage the concept of Variables and Operators in Java
We will study Object Oriented Programming with Java
You will learn the Control Statements and Loops in Java
We will see the concept of Functions with Java
We will study the concept of Inheritance in Java
We will learn to use Arrays in java
We will handle the concept of Matrices in Java
We will learn to Design Classes in Java
We will make a final application with everything learned in the course
To know more:

Java Essentials : Learn Core Java From Basic to Advance

Java Essentials : Learn Core Java From Basic to Advance

Learn Java Programming Using Practical Assignments. Start Building Back-end Web Applications Robust Test Automation Frameworks By End Of The Course. Learn More!

Description
This is only Java related course and it's great because it covers just the right amount of Java which is needed to leaning programming, java.

This is a comprehensive yet simple course on java programming language and it concentrates on Java programming concepts.

*************************** No Prior Coding Experience Needed ***************************

This course assumes that you have no programming background. If you have some experience then, it's just a bonus point. You have never code, have some experience or have a lot of experience any other programming language, this course is one stop place for you.

Java is one of the most and useful programming languages to learn You can build back-end of web applications and build robust test automation framework. Specially for Selenium WebDriver GUI automation, Java is most popular choice and has the largest community.

Each lecture consist of a video screencast and code files

There are quizzes, homework to test your knowledge

High focus on practice and asking questions

You will also learn coding best practices

Market is never short of jobs in Java programming language, there are ample of jobs in both Java development and Automation Testing using Java.

What are you waiting for? Enroll today and learn the powerful Java language !!!

Basic knowledge
Nothing else! It’s just you, your computer and your hunger to get started today
Java concepts are covered in the course, no experience needed
Windows/MAC computer
What will you learn
You will be able to EXPLAIN, DESIGN and IMPLEMENT efficient java Programs
You will be confident to clear test automation interviews
Understand the concepts of Object Oriented Programming Language
Complete understanding of java
Expert-level knowledge of Java code (+ advanced tips and tricks used by the pros)
Suitable for beginner programmers and ideal for users who learn faster when shown
To learn more:

Fundamentos de Java: Aprende Java desde cero, sin misterios | Simpliv

Fundamentos de Java: Aprende Java desde cero, sin misterios | Simpliv

Fundamentos de Java: Aprende Java desde cero, sin misterios

Description
This is the best course to learn to program in Java in Spanish from scratch and without any experience in this fabulous programming language . This is the first course where we will study the Java Fundamentals, and we will take you step by step until you acquire the basics of the Java language and so you can start studying more advanced Java topics.

The content is divided into perfectly structured levels , each level supported by the previous one, with the aim of adding Java knowledge incrementally so that you can focus on mastering the issues little by little and gradually. So ensure the success of your Java training.

In other offer support of any doubt teaching materials included in this course Fundamentals of Java.

To make matters worse, we handle a new teaching methodology that we have called Speed ​​Learning. This methodology consists of concise videos that go directly to the point to study, complemented with eBooks with explanations and step-by-step images (which you can print, or search for any text you need, or use for your offline study), since as we know we cannot do text search within a video. In addition, our methodology includes perfectly structured and very didactic exercises, which will allow you to accelerate your eLearning learning. Without wasting time on videos where you have to watch the instructor codify an exercise, too much theory, little practice or anything like that. Our Speed ​​Learning methodology guarantees that in the shortest possible time you will acquire the necessary knowledge for the professional and professional world of Java.

The Java Fundamentals course includes the following topics of study:

Level. Java basics

Lesson 1 - Starting with Java Technology

The amazing world of Java programming
What is Java technology (from a practical approach)
Our first Java program from scratch
Lesson 2 - Variables and Operators in Java

Use of Variables in Java and what we use them for
Data types in Java and how they are classified
Operator Management and Classification in Java
Lesson 3 - Control sentences in Java

Use of the if-else structure and where to use it
Management of the switch structure and when to apply it
Lesson 4 - Cycle Management in Java

Use of the for cycle and its use
Use of the while cycle and how to apply it
Use of the do-while cycle and when to use it
Lesson 5 - Object Oriented Programming

Introduction to Object Oriented Programming (OOP)
Class Management in Java
Using Objects in Java
Lesson 6 - Functions in Java

Declaration of Methods or Functions in Java
Use and call of functions in Java
Lesson 7 - Data Management in Java

Using Arrangements in Java
Matrix Management in Java
Lesson 8 - Inheritance in Java

Inheritance Management in Java
Use of superclasses and subclasses in Java
Final Level Laboratory

Final Exercise where everything learned in this Level is integrated
At the end you get a certificate of having completed the Java Fundamentals course.

We wait for you from the other side.

Ing. Ubaldo Acosta

Founder of Global Mentoring

Passion for Java Technology

Who this course is for:

Anyone who wants to learn to program in Java
Basic knowledge
Basic knowledge of PC use
Basic operation of an operating system such as Windows, Mac or Linux
It is not required to know how to program, we will start from scratch !!!
The attitude and desire to start coding and learning Java once and for all from scratch !!!
What will you learn
Have the basics of the programming language with Java
You will know the basic syntax of the Java language
Will handle the concept of Variables and Operators in Java
We will study Object Oriented Programming with Java
You will learn Control Sentences and Cycles in Java
We will see the concept of Functions with Java
We will study the concept of Inheritance in Java
We will learn to use Arrangements in java
We will handle the concept of Matrices in Java
We will learn to Design Classes in Java
We will make a final application with everything learned in the course
To continue: