Using a Java Based Kafka Client in a Node.js Application

Using a Java Based Kafka Client in a Node.js Application

A step by step guide for developing a Java based Kafka client in a Node.js application using GraalVM.

A step by step guide for developing a Java based Kafka client in a Node.js application using GraalVM.

The first time I heard about GraalVM, it totally blew my mind. Being able to combine multiple languages in a single application or business logic is an incredibly useful and powerful tool.

A real life need for a polyglot application emerged once we decided to switch from RabbitMQ to Kafka as our messaging system. Most of our RMQ consumers were written in Node.js, and moving to a different messaging system would force us either use a Node.js based library, or rewrite our entire business logic.

While there are several Node.js based Kafka clients, using them poses limitations such as the implemented Kafka API version, or the exposed interfaces and customization options. Using a Native Kafka client while maintaining the Node.js business logic would be a real win for us.

This tutorial builds on this awesome medium post on developing with Java and JavaScript together using GraalVM.

We will be using Docker Compose to build and create our images.

A working example can be found here.

Setting up Docker

The minimal needs of our environment are having GraalVM, Zookeeper and Kafka installed. The quickest way to achieve this is by using Docker and Docker Compose to create a complete running environment:

version: '3.3'
services:
  zookeeper:
    image: 'confluentinc/cp-zookeeper:5.0.0'
    hostname: zookeeper
    ports:
      - '2181:2181'
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_TICK_TIME: 2000
    volumes:
      - zk-data:/var/lib/zookeeper/data
      - zk-log:/var/lib/zookeeper/log
  kafka-broker:
    image: 'confluentinc/cp-kafka:5.0.0'
    ports:
      - '9092:9092'
      - '9093:9093'
    depends_on:
      - 'zookeeper'
    environment:
      KAFKA_BROKER_ID: 1
      KAFKA_ZOOKEEPER_CONNECT: 'zookeeper:2181'
      KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092,PLAINTEXT2://kafka-broker:9093
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT, PLAINTEXT2:PLAINTEXT
      KAFKA_TOPICS: "test_topic"
  graalvm:
    image: 'oracle/graalvm-ce:1.0.0-rc12'
    depends_on:
      - 'kafka-broker'
    volumes:
      - ./:/code
    environment:
      VM: 'graalvm'
      

volumes:
  zk-data:
  zk-log:

docker-compose.yml hosted with ❤ by GitHub

A Docker Compose file containing definitions for zookeeper, Kafka and GraalVM.

Running docker-compose up -d from the containing folder will perform the following:

  1. Download a Zookeeper image and run it on port 2181 along with persistent data and log volumes.
  2. Download a Kafka image containing a Kafka broker, and run it. The broker will connect to a Zookeeper on port 2181, and will allow client connections on ports 9092 and 9093.
  3. Download a GraalVM image. This image will have GraalVM and Node.js installed, and will have a shared volume with the host machine in the ./code folder.

All defined ports will be exposed on the local machine (localhost:port). Also, services will recognize each other based on their server name. Accessing Zookeeper from the broker machine will be using zookeeper:2181 as the host name. Same for kafka-broker:9092 for connecting with the Kafka broker.

Setting up the Java client

We are going to be using Java 1.8 and Maven to compile and run our Java client.

Even though the entire Kafka client will reside in a container, it will be helpful to run and debug our code directly from the host machine, using our favorite IDE. To do that, Maven and Java need to be installed on the host machine. Connection to other containers will be done using localhost as the host name.

Setting up Maven

You can use this tutorial to start a new Maven based Java project, or just use the following pom file:

<?xml version="1.0" encoding="UTF-8"?>

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>

  <groupId>your.group.id</groupId>
  <artifactId>kafka-client</artifactId>
  <version>1.0</version>

  <name>kafka-client</name>
  <!-- FIXME change it to the project's website -->
  <url>http://www.example.com</url>

  <properties>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    <maven.compiler.source>1.8</maven.compiler.source>
    <maven.compiler.target>1.8</maven.compiler.target>
  </properties>

  <dependencies>
    <dependency>
      <groupId>junit</groupId>
      <artifactId>junit</artifactId>
      <version>4.11</version>
      <scope>test</scope>
    </dependency>
    <!-- https://mvnrepository.com/artifact/org.apache.kafka/kafka-clients -->
    <dependency>
      <groupId>org.apache.kafka</groupId>
      <artifactId>kafka-clients</artifactId>
      <version>2.1.0</version>
    </dependency>
    <!-- https://mvnrepository.com/artifact/org.slf4j/slf4j-simple -->
    <dependency>
      <groupId>org.slf4j</groupId>
      <artifactId>slf4j-simple</artifactId>
      <version>1.7.25</version>
    </dependency>
    <!-- https://mvnrepository.com/artifact/org.json/json -->
    <dependency>
      <groupId>org.json</groupId>
      <artifactId>json</artifactId>
      <version>20180813</version>
    </dependency>
  </dependencies>

  <build>
    <plugins>
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-shade-plugin</artifactId>
        <executions>
          <execution>
            <phase>package</phase>
            <goals>
              <goal>shade</goal>
            </goals>
          </execution>
        </executions>
        <configuration>
          <finalName>uber-${project.artifactId}-${project.version}</finalName>
        </configuration>
      </plugin>
    </plugins>
            
  </build>
</project>

pom.xml hosted with ❤ by GitHub

The above pom file will create the required Java application file structure, along with all the required dependencies.

Notice the ‘maven-shade-plugin’ we are using to compile a single ‘uber-jar’ for the client and all of its dependencies. This will make it easier for us to add the client to the Node.js application later.

Make sure to change your.group.id to your desired package name.

Creating a Kafka client

Next step is creating our Kafka client (consumer and producer).

We will implement a basic Kafka producer and then a consumer.

Add a Producer.java file under /src/main/java/my/group/id:

package my.package.id;

import org.apache.kafka.clients.producer.*;
import org.apache.kafka.common.serialization.StringSerializer;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.util.Iterator;
import org.json.*;
import java.util.Properties;
import java.util.concurrent.ExecutionException;

public class Producer {

    public static void main(String[] args) {
        Producer p = new Producer("{\"bootstrap.servers\": \"localhost:9092\", }");
        try {
            p.put("test_topic", "msgKey", "msgData");
        }
        catch (Exception e) {
            System.out.println("Error Putting" + e);
        }
    }

    private Properties produceProperties;
    private final KafkaProducer<String, String> mProducer;
    private final Logger mLogger = LoggerFactory.getLogger(Producer.class);

    public Producer(String config) {
        extractPropertiesFromJson(config);
        mProducer = new KafkaProducer<>(produceProperties);

        mLogger.info("Producer initialized");
    }

    public void put(String topic, String key, String value) throws ExecutionException, InterruptedException {
        mLogger.info("Put value: " + value + ", for key: " + key);

        ProducerRecord<String, String> record = new ProducerRecord<>(topic, key, value);
        mProducer.send(record, (recordMetadata, e) -> {
        if (e != null) {
            mLogger.error("Error while producing", e);
            return;
        }

        mLogger.info("Received new meta. Topic: " + recordMetadata.topic()
            + "; Partition: " + recordMetadata.partition()
            + "; Offset: " + recordMetadata.offset()
            + "; Timestamp: " + recordMetadata.timestamp());
        }).get();
    }

    void close() {
        mLogger.info("Closing producer's connection");
        mProducer.close();
    }

    private void extractPropertiesFromJson(String jsonString) {
        produceProperties = new Properties();
        JSONObject jsonObject = new JSONObject(jsonString.trim());
        Iterator<String> keys = jsonObject.keys();
        while(keys.hasNext()) {
            String key = keys.next();
            produceProperties.setProperty(key, (String)jsonObject.get(key));
        }
        String deserializer = StringSerializer.class.getName();
        produceProperties.setProperty(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, deserializer);
        produceProperties.setProperty(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, deserializer);
    }
}

Producer.java hosted with ❤ by GitHub

The producer in the example above can receive its configuration in a JSON format, and sends a string type message.

The main function in the Producer is an easy way of running the code and sending a test message.

Add a Consumer.java file in the same folder:

package my.package.id;

import org.apache.kafka.clients.consumer.ConsumerConfig;
import org.apache.kafka.clients.consumer.ConsumerRecord;
import org.apache.kafka.clients.consumer.ConsumerRecords;
import org.apache.kafka.clients.consumer.KafkaConsumer;
import org.apache.kafka.common.errors.WakeupException;
import org.apache.kafka.common.serialization.StringDeserializer;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import java.util.Properties;
import java.util.Collections;
import java.util.Iterator;
import java.time.Duration;
import java.util.concurrent.CountDownLatch;
import org.json.*;
import java.util.Queue; 

public class Consumer {

    // a concurrent queue shared with Node
    private final Queue<Object> mQueue;     
    private Properties consumProperties;
    private final Logger mLogger = LoggerFactory.getLogger(Consumer.class.getName());
    
    public Consumer(Queue<Object> queue, String config){
      mQueue = queue;
      extractPropertiesFromJson(config);
    }

    public void start() {
        CountDownLatch latch = new CountDownLatch(1);

        ConsumerRunnable consumerRunnable = new ConsumerRunnable(consumProperties, latch, mQueue);
        Thread thread = new Thread(consumerRunnable);
        thread.start();
    
        Runtime.getRuntime().addShutdownHook(new Thread(() -> {
            mLogger.info("Caught shutdown hook");
            consumerRunnable.shutdown();
            await(latch);

            mLogger.info("Application has exited");
        }));
    }

    private void await(CountDownLatch latch) {
        try {
          latch.await();
        } catch (InterruptedException e) {
          mLogger.error("Application got interrupted", e);
        } finally {
          mLogger.info("Application is closing");
        }
      }
    
    private void extractPropertiesFromJson(String jsonString) {
        consumProperties = new Properties();
        JSONObject jsonObject = new JSONObject(jsonString.trim());
        Iterator<String> keys = jsonObject.keys();
        while(keys.hasNext()) {
            String key = keys.next();
            consumProperties.setProperty(key, (String)jsonObject.get(key));
        }
        String deserializer = StringDeserializer.class.getName();
        consumProperties.setProperty(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, deserializer);
        consumProperties.setProperty(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, deserializer);
    }

    private class ConsumerRunnable implements Runnable {

        private KafkaConsumer<String, String> mConsumer;
        private CountDownLatch mLatch;
        private Queue mQueue;

        ConsumerRunnable(Properties config, CountDownLatch latch, Queue queue) {
            mLatch = latch;
            mQueue = queue;
            String topic = (String)config.get("topic");
            config.remove("topic");
            mConsumer = new KafkaConsumer<>(config);
            mConsumer.subscribe(Collections.singletonList(topic));
        }

        @Override
        public void run() {
          try {
            while (true) {
              ConsumerRecords<String, String> records = mConsumer.poll(Duration.ofMillis(100));
    
              for (ConsumerRecord<String, String> record : records) {
                mLogger.info("Key: " + record.key() + ", Value: " + record.value());
                mLogger.info("Partition: " + record.partition() + ", Offset: " + record.offset());
                mQueue.offer(record);
              }
            }
          } catch (WakeupException e) {
            mLogger.info("Received shutdown signal!");
          } finally {
            mConsumer.close();
            mLatch.countDown();
          }
        }

        public void shutdown() {
            mConsumer.wakeup();
        }
    }
}

Consumer.java hosted with ❤ by GitHub

Same as the producer, this consumer receives its configuration in a JSON format.

After configuring our consumer, we start a new thread that connects to our Kafka broker and polls for messages. Each new message is pushed into a queue which will later be used in our Node.js application.

Compiling the code

Running mvn package form within the root folder, will compile the code into a single jar file named ‘uber-kafka-client-1.0.jar’. This file contains all required java code and dependencies, and will be used as a java library.

Setting up a Node.js Application

Last but not least is our Node.js application.

Add an index.js file under node/services/kafka-user:

const {Worker} = require('worker_threads');

function JavaToJSNotifier() {
    this.queue = new java.util.concurrent.LinkedBlockingDeque();
    this.worker = new Worker(`
        const { workerData, parentPort } = require('worker_threads');
        while (true) {
          // block the worker waiting for the next notification from Java
          var data = workerData.queue.take();
          // notify the main event loop that we got new data 
          parentPort.postMessage(data);
        }`,
        { eval: true, workerData: { queue: this.queue }, stdout: true, stderr: true });
}

const config = {
    "bootstrap.servers": (process.env.VM === 'graalvm') ?'kafka-broker:9093' : 'localhost:9092'
}

const Consumer = Java.type('my.package.id.Consumer');
config.topic = "test_topic";
config['group.id'] = 'Test_Group'

const asyncJavaEvents = new JavaToJSNotifier();
asyncJavaEvents.worker.on('message', (n) => {
    console.log(`Got new data from Java! ${n}`);
});

const mConsumer = new Consumer(asyncJavaEvents.queue, JSON.stringify(config));
mConsumer.start();

index.js hosted with ❤ by GitHub

The code above creates and configures a new Kafka consumer, and then uses node’s experimental workers to create a new thread that listens to messages from that consumer. The consumer thread notifies the main thread when a new message arrives.

Notice the this.queue = new java.util.concurrent.LinkedBlockingDeque()on line 4. This is possible due to using the GraalVM image. This queue will be a shared instance with the Java consumer we previously defined.

Also, notice the const Consumer = Java.type('my.pakcage.id.Consumer')in line 20. Again this is possible due to GraalVM, and will hold a reference to our Java based Kafka consumer.

Running the code

The previously installed GraalVM image already contains node and GraalVM setup. If one wishes to run the node application on the host machine instead, installing and configuring GraalVM is required (instructions).

To run our code inside the container, open a terminal from the root folder and type docker-compose run graalvm sh.

This will open a shell within the GraalVM image.

Due to our configuration all of our compiled code and scripts will be located under the ./code folder.

Run the following command:

node --polyglot --jvm --jvm.cp=code/target/uber-kafka-client-1.0.jar -- experimental-worker code/node/services/kafka-user/index.js

This command will run our node application as a polyglot application in a JVM. Notice the — jvm.cp parameter that tells JVM where to find our Java based Kafka client.

Trying it out

Keep the terminal open, go back to the Java IDE, and run the Producer.main procedure.

You should now see the following printed in you terminal:

Success!!

Summary

GraalVM makes writing polyglot applications easy. Adding a docker infrastructure, makes it even easier to develop and run cross-language applications just about anywhere.

The possibilities are virtually endless.

I hope this helps some of you and maybe inspires you to create some cross-language solutions to a real life problem you are facing.

Hetu Rajgor's answer to What are the major differences between Java, AngularJS, and JavaScript and Node JS? - Quora

<img src="https://moriohcdn.b-cdn.net/70b437cf37.png">Java is a programming language which is owned by Oracle. More than 3 Million devices are running in Java.&nbsp;JS is a client-side programming language used for creating dynamic websites and apps to run in the client's browser.

Java is a programming language which is owned by Oracle. More than 3 Million devices are running in Java. JS is a client-side programming language used for creating dynamic websites and apps to run in the client's browser.

Top 7 Most Popular Node.js Frameworks You Should Know

Top 7 Most Popular Node.js Frameworks You Should Know

Node.js is an open-source, cross-platform, runtime environment that allows developers to run JavaScript outside of a browser. In this post, you'll see top 7 of the most popular Node frameworks at this point in time (ranked from high to low by GitHub stars).

Node.js is an open-source, cross-platform, runtime environment that allows developers to run JavaScript outside of a browser.

One of the main advantages of Node is that it enables developers to use JavaScript on both the front-end and the back-end of an application. This not only makes the source code of any app cleaner and more consistent, but it significantly speeds up app development too, as developers only need to use one language.

Node is fast, scalable, and easy to get started with. Its default package manager is npm, which means it also sports the largest ecosystem of open-source libraries. Node is used by companies such as NASA, Uber, Netflix, and Walmart.

But Node doesn't come alone. It comes with a plethora of frameworks. A Node framework can be pictured as the external scaffolding that you can build your app in. These frameworks are built on top of Node and extend the technology's functionality, mostly by making apps easier to prototype and develop, while also making them faster and more scalable.

Below are 7of the most popular Node frameworks at this point in time (ranked from high to low by GitHub stars).

Express

With over 43,000 GitHub stars, Express is the most popular Node framework. It brands itself as a fast, unopinionated, and minimalist framework. Express acts as middleware: it helps set up and configure routes to send and receive requests between the front-end and the database of an app.

Express provides lightweight, powerful tools for HTTP servers. It's a great framework for single-page apps, websites, hybrids, or public HTTP APIs. It supports over fourteen different template engines, so developers aren't forced into any specific ORM.

Meteor

Meteor is a full-stack JavaScript platform. It allows developers to build real-time web apps, i.e. apps where code changes are pushed to all browsers and devices in real-time. Additionally, servers send data over the wire, instead of HTML. The client renders the data.

The project has over 41,000 GitHub stars and is built to power large projects. Meteor is used by companies such as Mazda, Honeywell, Qualcomm, and IKEA. It has excellent documentation and a strong community behind it.

Koa

Koa is built by the same team that built Express. It uses ES6 methods that allow developers to work without callbacks. Developers also have more control over error-handling. Koa has no middleware within its core, which means that developers have more control over configuration, but which means that traditional Node middleware (e.g. req, res, next) won't work with Koa.

Koa already has over 26,000 GitHub stars. The Express developers built Koa because they wanted a lighter framework that was more expressive and more robust than Express. You can find out more about the differences between Koa and Express here.

Sails

Sails is a real-time, MVC framework for Node that's built on Express. It supports auto-generated REST APIs and comes with an easy WebSocket integration.

The project has over 20,000 stars on GitHub and is compatible with almost all databases (MySQL, MongoDB, PostgreSQL, Redis). It's also compatible with most front-end technologies (Angular, iOS, Android, React, and even Windows Phone).

Nest

Nest has over 15,000 GitHub stars. It uses progressive JavaScript and is built with TypeScript, which means it comes with strong typing. It combines elements of object-oriented programming, functional programming, and functional reactive programming.

Nest is packaged in such a way it serves as a complete development kit for writing enterprise-level apps. The framework uses Express, but is compatible with a wide range of other libraries.

LoopBack

LoopBack is a framework that allows developers to quickly create REST APIs. It has an easy-to-use CLI wizard and allows developers to create models either on their schema or dynamically. It also has a built-in API explorer.

LoopBack has over 12,000 GitHub stars and is used by companies such as GoDaddy, Symantec, and the Bank of America. It's compatible with many REST services and a wide variety of databases (MongoDB, Oracle, MySQL, PostgreSQL).

Hapi

Similar to Express, hapi serves data by intermediating between server-side and client-side. As such, it's can serve as a substitute for Express. Hapi allows developers to focus on writing reusable app logic in a modular and prescriptive fashion.

The project has over 11,000 GitHub stars. It has built-in support for input validation, caching, authentication, and more. Hapi was originally developed to handle all of Walmart's mobile traffic during Black Friday.

Why is a Java guy so excited about Node.js and JavaScript?

Shouldn’t someone who worked 10+ years in the Java SE team at Sun Microsystems bleed Java bytecodes and be instantiating abstract interfaces until the last breath? For this former Java SE team member, learning the Node.js platform in 2011 was a breath of fresh air. After being laid off from Sun in January 2009 (just prior to the Oracle buyout), I learned about and became hooked on Node.js.

Shouldn’t someone who worked 10+ years in the Java SE team at Sun Microsystems bleed Java bytecodes and be instantiating abstract interfaces until the last breath? For this former Java SE team member, learning the Node.js platform in 2011 was a breath of fresh air. After being laid off from Sun in January 2009 (just prior to the Oracle buyout), I learned about and became hooked on Node.js.

Just how hooked? Since 2010, I have written extensively about Node.js programming. Namely, four editions of Node.js Web Development, plus other books and numerous tutorial blog posts on Node.js programming. That’s a LOT of time explaining Node.js and advancements in the JavaScript language.

While working for Sun Microsystems I believed in All Things Java. I presented sessions at JavaONE, co-developed the java.awt.Robot class, ran the Mustang Regressions Contest (the bug finding contest for the Java 1.6 release), helped to launch the “Distributions License for Java” that was the pre-OpenJDK answer for Linux distributions to distribute JDK builds, and later played a small role in launching the OpenJDK project. Along the way I landed a blog on java.net (a now-defunct website), writing 1–2 times a week for about 6 years discussing events in the Java ecosystem. A big topic was defending Java against those predicting Java’s death.


The Duke Award was handed out to employees who went above and beyond. I earned this after running the Mustang Regressions Contest, the bug-finding contest coinciding with the Java 1.6 release.

What happened to living and breathing Java byte-codes? What I’m here for is explaining how this died-in-the-wool Java advocate became a died-in-the-wool Node.js/JavaScript advocate.

It’s not like I’ve completely divorced myself of Java. I have written a significant amount of Java/Spring/Hibernate code in the last 3 years. While I fully enjoyed the work — I worked in the Solar Industry doing deeply soul-fulfilling things like writing database queries about kiloWatt-hours — coding in Java has lost its lustre.

Two years of Spring coding taught a very clear lesson: papering over complexity does not produce simplicity, it only produces more complexity.

TL;DR

  • Java is full of boilerplate code which obscures the programmers intention
  • Spring & Spring Boot teach a lesson: papering over complexity creates more complexity
  • Java EE was a “design by committee” project that covers everything of everything in Enterprise application development, hence is exceedingly complex
  • The experience of Spring programming is it’s great until it isn’t, until that day an obscure impossible-to-understand Exception appears out of the depths of a subsystem you’ve never heard of requiring 3+ days just to figure out the problem
  • What is the overhead required in the framework to allow coders to write zero code?
  • While IDE’s like Eclipse are powerful, they are a symptom of Java’s complexity
  • Node.js was the result of one guy honing and refining a vision of a lightweight event driven architecture, until Node.js revealed itself
  • The JavaScript community seems to appreciate removing boilerplate allowing the programmers intention to shine
  • The solution for Callback Hell, the async/await function, is an example of removing boilerplate so the programmers intention shines
  • Coding with Node.js is a joy
  • JavaScript lacks the strict type checking of Java, which is a blessing and a curse. Code is easier to write but requires more testing to ensure correctness
  • The npm/yarn package management system is excellent and joyful to use, versus the abomination that is Maven
  • Both Java and Node.js offer excellent performance, running counter to the myth that JavaScript is slow and therefore Node.js performance must be bad
  • Node.js performance is riding the coattails of Google investment in V8 to speed up the Chrome browser
  • The intense competition between browsers is making JavaScript more and more powerful every year, benefitting Node.js

Java has become a burden, coding with Node.js is joy-filled

Some tools or objects are the result of a designer spending years honing and refining the thing. They try different ideas, they remove unnecessary attributes, and they end up with an object with just the right attributes for the purpose. Often these objects have a kind of powerful simplicity that is very appealing. Java is not that kind of system.

Spring is a popular framework for developing Java-based web applications. The core purpose for Spring, and especially Spring Boot, is an easy to use preconfigured Java EE stack. The Spring programmer doesn’t have to hook up all the servlets, data persistence, application servers, and who knows what else, to get a complete system. Instead Spring takes care of all those details, while you focus on coding. For example, the JPA Repository class synthesizes database queries for methods with names like “findUserByFirstName” — you don’t write any code, simply add methods named this way to the Repository definition and Spring will handle the rest.

It’s a great story, and a nice experience, until it isn’t.

What’s it mean when you get a Hibernate PersistentObjectException about a “detached entity passed to persist”? That took several days to figure out — at the risk of oversimplification — it meant the JSON arriving at the REST endpoint had ID fields with values. Hibernate, oversimplying, wants control of the ID values, and throws this confusing exception. There are thousands of equally confusing and obtuse exception messages. With subsystem after subsystem in the Spring stack it’s like a nemesis sitting in wait for your tiniest mistake, and then it pounces on you with an application-crashing exception.

Then, there is the huge stack traces. They go on for several screens full of abstract method this and abstract method that. Spring is obviously working out the configuration required to implement what the code says. This level of abstraction obviously requires quite a bit of logic to find everything and to execute the requests. A long stack trace isn’t necessarily bad. Instead it points to a symptom: Just what is the memory/performance overhead cost?

How would “findUserByFirstName” execute when the programmer writes zero code? The framework has to parse the method name, guess the programmers intention, construct something like an abstract syntax tree, generate some SQL, etc.. Just what is the overhead for all that? Just so the coder doesn’t have to code?

After a few dozen times going through this, spending weeks learning arcana you should never have had to learn, you might arrive to the same conclusion I landed in: papering over complexity does not produce simplicity, it only produces more complexity.

The point goes to Node.js


“Compatibility Matters” was a very cool slogan, meaning that the Java Platform’s key value proposition was complete backwards compatibility. We took this seriously as well as plastered it on T-Shirts like this. Of course this degree of maintaining compatibility can be an albatross, and it’s useful at times to shirk off old ways of doing things that are no longer useful.

Node.js on the other hand …

Where Spring and Java EE is exceedingly complex, Node.js is a breath of fresh air. First is the design esthetic Ryan Dahl used in developing the core Node.js platform. Dahl’s experience was that using threads made for a heavyweight complex system. He sought something different, and spent a couple years honing and refining a set of core ideals arriving at Node.js. The result was a light-weight system, a single execution thread, an ingenious use of JavaScript anonymous functions for asynchronous callbacks, and a runtime library that ingeniously implemented asynchronicity. The initial pitch was high throughput event processing with event delivery to callback functions.

Then there is the JavaScript language itself. JavaScript programmers seem to have an esthetic of removing boilerplate so the programmers intention can shine clearly.

An example to contrast between Java and JavaScript is the implementation of listener functions. In Java. Listeners require creating a concrete instance of an abstract interface class. These entail a lot of verbiage obscuring what’s going on. How can the programmers intent be visible behind a veil of boilerplate?

In JavaScript one instead uses simple anonymous functions — closures. You don’t search for the correct abstract interface. Instead you simply write the required code, with no excess verbiage.

Another learning: Most programming languages obscure the programmers intent, making it harder to understand the code.

This point goes to Node.js — but with a caveat we must address: Callback Hell.


Solutions sometimes come with their own problems

In JavaScript we’ve long had two issues with asynchronous coding. One is what’s become called “Callback Hell” in Node.js. It is easy to fall into a trap of deeply nested callback functions, where each level of nesting complexifies the code, making error and results handling all the more difficult. A related issue is that the JavaScript language did not help the programmer properly express asynchronous execution.

Several libraries sprung up promising to simplify asynchronous execution. Another example of papering over complexity creating more complexity.

To give an example:

const async = require(‘async’);
const fs = require(‘fs’);
const cat = function(filez, fini) {
  async.eachSeries(filez, function(filenm, next) {
    fs.readFile(filenm, ‘utf8’, function(err, data) {
      if (err) return next(err);
      process.stdout.write(data, ‘utf8’, function(err) {
        if (err) next(err);
        else next();
      });
    });
  },
  function(err) {
    if (err) fini(err);
    else fini();
  });
};
cat(process.argv.slice(2), function(err) {
  if (err) console.error(err.stack);
});

This sample application is a pale cheap imitation of the Unix cat command. The async library is excellent at simplifying the sequencing of asynchronous execution. But its use required a pile of boilerplate code obscuring the programmers intention.

What we have here is a loop. It is not written as a loop, and it does not use natural looping constructs. Further, errors and results do not land in the natural place, but are inconveniently trapped inside callback functions. Before ES2015/2016 features landed in Node.js this was the best we could do.

The equivalent in Node.js 10.x is:

const fs = require(‘fs’).promises;
async function cat(filenmz) {
  for (var filenm of filenmz) {
    let data = await fs.readFile(filenm, ‘utf8’);
    await new Promise((resolve, reject) => {
      process.stdout.write(data, ‘utf8’, (err) => {
        if (err) reject(err);
        else resolve();
      });
    });
  }
}
cat(process.argv.slice(2)).catch(err => { 
    console.error(err.stack); 
});

This rewrite of the previous example uses async/await functions. It is the same asynchronous structure, but is written with normal looping structures. Errors and results are reported in a natural way. It’s easier to read, to code, and to understand the programmers intent.

The only wart is that process.stdout.write does not provide a Promise interface and therefore cannot be cleanly used in an async function without wrapping with a Promise.

The callback hell problem was not solved by papering over complexity. Instead a language and paradigm change solved both the problem and the excess verbiage forced on us by the interim solutions. With async functions our code became more beautiful.

While this started out as a point against Node.js the excellent solution converts this into a point for Node.js and JavaScript.


Supposed clarity through well defined types and interfaces

An issue I harped on as a died-in-the-wool Java advocate was that strict type checking enables writing huge applications. At that time the norm was developing monolithic systems (no microservices, no Docker, etc). Because Java has strict type checking, the Java compiler helps you avoid many kinds of bugs — by preventing you from compiling bad code.

JavaScript, by contrast, has loosey-goosey typing. The theory is obvious: The programmer has no certainty what kind of object they’ve received, so how can the programmer know what to do?

The flip side of strict typing in Java is more boilerplate. The programmer is constantly typecasting or otherwise working hard ensuring everything is precisely correct. The coder spends time coding, with extreme precision, using more boilerplate, hoping to save time by catching and fixing errors early.

The problem is so big that one more-or-less must use a big complex IDE. A simple programmers editor is not enough. The only way to keep the Java programmer sane (besides pizza) is with dropdowns showing the available fields on an object, describing method parameters, helping to construct classes, assisting with refactoring, and all the rest of the facilities offered by Eclipse, NetBeans, and IntelliJ.

And… don’t get me started on Maven. What a horrible tool.

In JavaScript, variable types are not declared, type-casting is generally not used, etc. Hence the code is cleaner to read, but there is a risk of uncaught coding errors.

Whether this point goes to Java or against Java depends on your point of view. My opinion ten years ago was this overhead is worth the cost by gaining more certainty. My opinion today is gosh that’s a lot of work, and it’s much easier to do things the JavaScript way.


Squashing bugs with small easily tested modules

Node.js encourages programmers to divide their programs into small units, the module. This may seem a small thing, but it partly addresses the problem just named.

A module is:

  • Self contained — as implied, it packages related code into a unit
  • Strong boundary — code inside the module is safe from intrusion from code elsewhere
  • Explicit exports — by default code and data in the module is not exported, with selected functions and data available to other code
  • Explicit imports — modules declare which modules they depend on
  • Potentially independent — it is easy to publicly publish modules to the npm repository, or to privately publish modules elsewhere, for easy sharing between applications
  • Easier to reason about — less code to read makes it easier to understand the purpose
  • Easier to test — if implemented correctly, the small module can be easily unit tested

All of this together contributes to Node.js modules being easier to test and to have a well defined scope.

The fear with JavaScript is that lacking strict type checking the code could easily do something wrong. In a small focused module with clear boundaries, the scope of affected code is mostly limited to that module. This keeps most concerns small, safely ensconced within the boundary of that module.

The other solution to the loosey-goosey problem is increased testing.

You must spend some of the productivity gains (it is easier to write JavaScript code) on increased testing. Your testing regime must catch the errors that might have been caught by the compiler. You do test your code don’t you?

For those wanting statically checked types in JavaScript, look at TypeScript. I’ve not used that language, but have heard great things said about it. It is directly compatible with JavaScript and adds useful type checking and other features.

Here the point goes to Node.js and JavaScript.


Package management

I get apoplectic just thinking about Maven, and simply can’t think straight enough to write anything about it. Supposedly one either loves Maven, or despises it, and that there is no middle ground.

An issue is that the Java ecosystem does not have a cohesive package management system. Maven packages exist and work fairly well, and supposedly work in Gradle as well. But it isn’t anywhere near as useful/usable/powerful as the package management system for Node.js.

In the Node.js world there are two excellent package management systems that work closely together. At first npm, and the npm repository, was the only such tool.

With npm we have an excellent schema for describing package dependencies. A dependency can be strict (exactly version 1.2.3) or specified with several gradations of looseness all the way to “*” which means the latest version. The Node.js community has published hundreds of thousands of packages into the npm repository. Using packages from outside the npm repository is just as easy as using packages from the npm repository.

It is so good that npm’s repository serves not just Node.js but front end engineers. Previously tools like Bower for package management were used. Bower has been deprecated and now one finds all the front-end JavaScript libraries available as npm packages. Many front end engineer toolchains, like the Vue.js CLI and Webpack, are written in Node.js.

The other package management system for Node.js, yarn, draws its packages from the npm repository, and uses the same configuration files as npm. The main advantage is that the yarn tool runs faster.

The npm repository, whether accessed with npm or yarn, is a strong part of what makes Node.js so easy and joyful to use.


Sometime after helping create java.awt.Robot, I was inspired to create this. Where the official Duke mascot was entirely curvy lines, RoboDuke is all about straight lines, except for the gearing in the elbows.

Performance

At times both Java and JavaScript were accused of being slow.

In both cases compilers convert source code to byte-code executed by a virtual machine implementation. The VM will often further compile byte-codes to native code, and use a variety of optimization techniques.

Both Java and JavaScript have huge incentives to run fast. In Java and Node.js the incentive is fast server-side code. In browser-JavaScript the incentive is better client side application performance — see the next section on Rich Internet Applications.

The Sun/Oracle JDK uses HotSpot, a super-dooper virtual machine with multiple byte code compilation strategies. Its name comes from detecting frequently executed code and applying more and more optimization the more a code section executes. HotSpot is highly optimized and produces very fast code.

On the JavaScript side, we used to wonder: How could we expect JavaScript code running in browsers to implement any kind of complex application? Surely an office document processing suite would be impossible in browser-based JavaScript? Today, well, the proof of the pudding is in the eating. I’m writing this article using Google Docs, and the performance is pretty good. Browser-JavaScript performance leaps forward every year.

Node.js directly benefits from this trend since it uses Chrome’s V8 engine.

An example is this talk by Peter Marshall, a Google engineer working on V8 whose primary job is performance enhancements for V8. Marshall is specifically assigned to work on Node.js performance. He describes why V8 switched from the Crankshaft virtual machine to the Turbofan virtual machine.


Machine Learning is an area involving lots of mathematics for which data scientists typically use R or Python. Several fields including Machine Learning depend on fast numerical computing. For various reasons JavaScript is poor at this, but work is underway to develop a standardized library for numerical computing in JavaScript.


Another video demonstrates using TensorFlow in JavaScript, using a new library: TensorFlow.js. It has an API similar to TensorFlow Python, and can import pretrained models. It can be used for things like analyzing live video to recognize trained objects, whilst running completely in a browser.


In another talk, Chris Bailey of IBM goes over Node.js performance and scalability issues, especially in regard to Docker/Kubernetes deployment. He starts with a set of benchmarks showing that Node.js has dramatically better performance than Spring Boot on I/O throughput, application startup time and memory footprint. Further, release-by-release performance is improving dramatically in Node.js, thanks in part to V8 improvements.


In that video Bailey says one should not run computational code in Node.js. The “why” of that is important to understand. Because of the single thread model long-running computations will block event execution. In my book Node.js Web Development, I talk about this issue and demonstrate three approaches:

  • Algorithmic refactoring — detect slow parts of an algorithm, refactoring for more speed
  • Splitting chunks of the computation through the event dispatcher, so that Node.js regularly returns to the execution thread
  • Shifting computations to a back-end server

If JavaScript improvements aren’t enough for your application, there are two ways to directly integrate native code into Node.js. The most straightforward method is a native-code Node.js module. The Node.js toolchain includes node-gyp which handles linking to native-code modules. This video demonstrates integrating a Rust library with Node.js:


WebAssembly offers the ability to compile other languages down to a JavaScript subset that runs very fast. WebAssembly is a portable format for executable code that runs inside a JavaScript engine. This video is a good overview, and demonstrates using WebAssembly to run code in Node.js.


Rich Internet Applications (RIA)

Ten years ago the software industry was buzzing about Rich Internet Applications implemented with fast (for the time) JavaScript engines that would make desktop applications irrelevant.

The story actually starts over twenty years ago. Sun and Netscape struck a deal for Java Applets in Netscape Navigator. The JavaScript language was, in part, developed as the scripting language for Java Applets. The hope was there would be Java Servlets on the server-side, and Java Applets on the client side, giving us all this nirvana condition of having the same programming language on both. That didn’t happen for various reasons.

Ten years ago JavaScript was starting to be powerful enough for implementing complex applications on its own. Hence the RIA buzzword, and RIA’s were supposed to kill Java as a client-side app platform.

Today we’re starting to see the RIA idea come to fruition. With Node.js on the server we can now have that nirvana, but with JavaScript at both ends of the wire.

Some examples:

  • Google Docs (where this article is being written) is like a typical Office Suite, but running in the browser
  • Powerful frameworks like React, Angular, Vue.js, simplify browser-based application development using HTML/CSS for styling
  • Electron is an amalgam of Node.js and the Chromium web browser supports cross-platform desktop application development. It’s good enough that several extremely popular applications like Visual Studio Code, Atom, GitKraken, and Postman are all written in Electron, all of which perform extremely well.
  • Because Electron/NW.js use a browser engine, frameworks like React/Angular/Vue can be used for desktop applications. For an example, see: https://blog.sourcerer.io/creating-a-markdown-editor-previewer-in-electron-and-vue-js-32a084e7b8fe

Java as a desktop application platform did not die because of JavaScript RIA’s. It died mostly because of neglect at Sun Microsystems for client technologies. Sun was focused on Enterprise customers demanding fast server-side performance. I was there and saw it first-hand. What really killed off Applets was a bad security bug discovered a few years ago in the Java Plugin and Java Web Start. That bug caused a worldwide alert to simply cease using Java applets and Webstart applications.

Other kinds of Java desktop applications can still be developed, and the competition between the NetBeans and Eclipse IDE’s is still alive and kicking as a result. But work in this area of Java is stagnant, and outside of developer tools there are very few Java-based applications.

The exception is JavaFX.

JavaFX was, 10 years ago, going to be Sun’s answer to the iPhone. It was going to support developing GUI-rich applications on the Java platform available in cell phones, and would single-handedly put Flash and iOS application development out of business. That didn’t happen. JavaFX is still used, but did not live up to its hype.

All of the excitement in this field is happening with React, Vue.js, and similar frameworks.

In this case JavaScript and Node.js won the point in a big way.


The Java Ring, handed out at an early Java ONE conference. These rings contain a chip with a complete Java implementation on-board. The primary use at JavaONE was unlocking the computers we installed in the lobby.


Java Ring instructions.

Conclusion

Today there are many choices for developing server-side code. We’re no longer limited to the “P languages” (Perl, PHP, Python) and Java, because there is Node.js, Ruby, Haskell, Go, Rust, and many more. Therefore we have an embarrassment of riches to enjoy.

In terms of why did this Java guy turn to Node.js, it’s clear that I prefer the feeling of freedom while coding with Node.js. Java became a burden, and with Node.js there is no such burden. If I were employed to write Java again I would do it, of course, because that’s what I’d be paid to do.

Each application has its authentic needs. It’s of course incorrect to always use Node.js because one prefers Node.js. There must be a technical reason to choose one language or framework over another. For example some work I’ve done recently involves XBRL documents. Since the best XBRL libraries are implemented in Python, it’s necessary to learn Python to continue with that project. Make an honest evaluation of your real needs and choose accordingly.

Thanks for reading ❤

If you liked this post, share it with all of your programming buddies!

Follow me on Facebook | Twitter

Learn More

Java Programming Masterclass for Software Developers

The Complete Node.js Developer Course (3rd Edition)

Angular & NodeJS - The MEAN Stack Guide

The Complete JavaScript Course 2019: Build Real Projects!

Vue JS 2 - The Complete Guide (incl. Vue Router & Vuex)

Google’s Go Essentials For Node.js / JavaScript Developers

Moving from NodeJS to Go

A Beginner’s Guide to JavaScript’s Prototype

Top 12 Javascript Tricks for Beginners

Spring Boot + JPA + Hibernate + Oracle

Top 5 Java Test Frameworks for Automation in 2019

A Study List for Java Developers