Arjun  Goodwin

Arjun Goodwin

1594142580

Setting Apache Nifi on Docker Containers

The system I work on is about to undergo a significant tech-refresh. The goal is to improve the current hindrances that prevent ingesting more data. The remedy requires a product that receives inputs from external sources, processes them, and disseminates the outcomes to their destinations.

Apache Nifi implements the flow-based programming (FBP) paradigm; it composes of black-box processes that exchange data across predefined connections (an excerpt from Wikipedia).

In short, Apache NiFi is a tool to process and distribute data. Its intuitive UI supports routing definitions, a variety of connectors (in/out), and many built-in processors. All these features combined together make it a suitable optional platform for our use case.

In view of our system’s future needs, we decided to evaluate Nifi thoroughly. The starting point is setting up an environment.

In this article, I’ll describe how to set up a Nifi environment using Docker images and run a simple predefined template; building a Nifi flow from scratch will be covered in another article. The main three parts of the article are:

  • Reviewing Apache Nifi concepts and building-blocks
  • Setting Nifi Flow and Nifi Registry (based on Docker images)
  • Loading a template and running it

Ready? Let’s start with the foundations.

Nifi Components and Concepts

Nifi is based on the following hierarchy:

  • Process Group
  • A collection of processors and their connections. A process group is the smallest unit to be saved in version control (Nifi Registry). A process group can have input and output ports that allow connecting Process Groups. With that, data flow can be composed of more than one Process Group.
  • Processor
  • A processing unit that (mostly) has input and output linked to another processor by a connector_._ Each processor is a black-box that executes a single operation; for example, processors can change the content or the attributes of the FlowFile (see below).
  • FlowFile
  • This is the logical set of data with two parts (content and attributes), which passes between the Nifi Processors. The FlowFile object is immutable, but its contents and attributes can change during the processing.
  • Connection
  • A Connection is a queue that routes FlowFiles between processors. The routing logic is based on conditions related to the processor’s result; a connection is associated with one or more result types. A connection’s conditions are the relationships between processors, which can be static or dynamic. While static relationships are fixed (for example — Success, Failure, Match, or Unmatch), the dynamic relationships are based on attributes of the FlowFile, defined by the user; the last section in this article exemplifies this feature with _RouteOnAttribute _processor.
  • Port
  • The entry and exit points of a Process Group. Each Process Group can have one or more input or output ports, distinguished by their names.
  • Funnel
  • Combines the data from several connections into a single connection.

The Nifi flow below depicts these components:

Process Group and Nifi elements

After reviewing Nifi data flow components, let’s see how to set up an environmen

#dataflow #software-development #apache-nifi #programming #docker #apache

What is GEEK

Buddha Community

Setting Apache Nifi on Docker Containers
Arjun  Goodwin

Arjun Goodwin

1594142580

Setting Apache Nifi on Docker Containers

The system I work on is about to undergo a significant tech-refresh. The goal is to improve the current hindrances that prevent ingesting more data. The remedy requires a product that receives inputs from external sources, processes them, and disseminates the outcomes to their destinations.

Apache Nifi implements the flow-based programming (FBP) paradigm; it composes of black-box processes that exchange data across predefined connections (an excerpt from Wikipedia).

In short, Apache NiFi is a tool to process and distribute data. Its intuitive UI supports routing definitions, a variety of connectors (in/out), and many built-in processors. All these features combined together make it a suitable optional platform for our use case.

In view of our system’s future needs, we decided to evaluate Nifi thoroughly. The starting point is setting up an environment.

In this article, I’ll describe how to set up a Nifi environment using Docker images and run a simple predefined template; building a Nifi flow from scratch will be covered in another article. The main three parts of the article are:

  • Reviewing Apache Nifi concepts and building-blocks
  • Setting Nifi Flow and Nifi Registry (based on Docker images)
  • Loading a template and running it

Ready? Let’s start with the foundations.

Nifi Components and Concepts

Nifi is based on the following hierarchy:

  • Process Group
  • A collection of processors and their connections. A process group is the smallest unit to be saved in version control (Nifi Registry). A process group can have input and output ports that allow connecting Process Groups. With that, data flow can be composed of more than one Process Group.
  • Processor
  • A processing unit that (mostly) has input and output linked to another processor by a connector_._ Each processor is a black-box that executes a single operation; for example, processors can change the content or the attributes of the FlowFile (see below).
  • FlowFile
  • This is the logical set of data with two parts (content and attributes), which passes between the Nifi Processors. The FlowFile object is immutable, but its contents and attributes can change during the processing.
  • Connection
  • A Connection is a queue that routes FlowFiles between processors. The routing logic is based on conditions related to the processor’s result; a connection is associated with one or more result types. A connection’s conditions are the relationships between processors, which can be static or dynamic. While static relationships are fixed (for example — Success, Failure, Match, or Unmatch), the dynamic relationships are based on attributes of the FlowFile, defined by the user; the last section in this article exemplifies this feature with _RouteOnAttribute _processor.
  • Port
  • The entry and exit points of a Process Group. Each Process Group can have one or more input or output ports, distinguished by their names.
  • Funnel
  • Combines the data from several connections into a single connection.

The Nifi flow below depicts these components:

Process Group and Nifi elements

After reviewing Nifi data flow components, let’s see how to set up an environmen

#dataflow #software-development #apache-nifi #programming #docker #apache

Mikel  Okuneva

Mikel Okuneva

1602317778

Ever Wondered Why We Use Containers In DevOps?

At some point we’ve all said the words, “But it works on my machine.” It usually happens during testing or when you’re trying to get a new project set up. Sometimes it happens when you pull down changes from an updated branch.

Every machine has different underlying states depending on the operating system, other installed programs, and permissions. Getting a project to run locally could take hours or even days because of weird system issues.

The worst part is that this can also happen in production. If the server is configured differently than what you’re running locally, your changes might not work as you expect and cause problems for users. There’s a way around all of these common issues using containers.

What is a container

A container is a piece of software that packages code and its dependencies so that the application can run in any computing environment. They basically create a little unit that you can put on any operating system and reliably and consistently run the application. You don’t have to worry about any of those underlying system issues creeping in later.

Although containers were already used in Linux for years, they became more popular in recent years. Most of the time when people are talking about containers, they’re referring to Docker containers. These containers are built from images that include all of the dependencies needed to run an application.

When you think of containers, virtual machines might also come to mind. They are very similar, but the big difference is that containers virtualize the operating system instead of the hardware. That’s what makes them so easy to run on all of the operating systems consistently.

What containers have to do with DevOps

Since we know how odd happenings occur when you move code from one computing environment to another, this is also a common issue with moving code to the different environments in our DevOps process. You don’t want to have to deal with system differences between staging and production. That would require more work than it should.

Once you have an artifact built, you should be able to use it in any environment from local to production. That’s the reason we use containers in DevOps. It’s also invaluable when you’re working with microservices. Docker containers used with something like Kubernetes will make it easier for you to handle larger systems with more moving pieces.

#devops #containers #containers-devops #devops-containers #devops-tools #devops-docker #docker #docker-image

Iliana  Welch

Iliana Welch

1597368540

Docker Tutorial for Beginners 8 - Build and Run C++ Applications in a Docker Container

Docker is an open platform that allows use package, develop, run, and ship software applications in different environments using containers.
In this course We will learn How to Write Dockerfiles, Working with the Docker Toolbox, How to Work with the Docker Machine, How to Use Docker Compose to fire up multiple containers, How to Work with Docker Kinematic, Push images to Docker Hub, Pull images from a Docker Registery, Push stacks of servers to Docker Hub.
How to install Docker on Mac.

#docker tutorial #c++ #docker container #docker #docker hub #devopstools

Iliana  Welch

Iliana Welch

1595249460

Docker Explained: Docker Architecture | Docker Registries

Following the second video about Docker basics, in this video, I explain Docker architecture and explain the different building blocks of the docker engine; docker client, API, Docker Daemon. I also explain what a docker registry is and I finish the video with a demo explaining and illustrating how to use Docker hub

In this video lesson you will learn:

  • What is Docker Host
  • What is Docker Engine
  • Learn about Docker Architecture
  • Learn about Docker client and Docker Daemon
  • Docker Hub and Registries
  • Simple demo to understand using images from registries

#docker #docker hub #docker host #docker engine #docker architecture #api

August  Murray

August Murray

1615124700

Docker Swarm: Container Orchestration Using Docker Swarm

Introduction

A swarm consists of multiple Docker hosts that run in swarm mode and act as managers (to manage membership and delegation) and workers (which run swarm services). A given Docker host can be a manager, a worker, or perform both roles.

When Docker is running in swarm mode, you can still run standalone containers on any of the Docker hosts participating in the swarm, as well as swarm services. A key difference between standalone containers and swarm services is that only swarm managers can manage a swarm, while standalone containers can be started on any daemon.

In this demonstration, we will see how to configure the docker swarm and how to perform basic tasks.

Pre-requisites

  1. For our demonstration, we will be using centos-07.
  2. We will be using 3 machines for our lab, 1 machine as a swarm Manager node and 2 swarm worker nodes. These servers have below IP details:

192.168.33.76 managernode.unixlab.com

192.168.33.77 workernode1.unixlab.com

192.168.33.78 workernode2.unixlab.com

3. The memory should be at least 2 GB and there should be at least 2 core CPUs for each node.

#docker #containers #container-orchestration #docker-swarm