How to Install Redis on Ubuntu 18.04?

How to Install and Secure Redis on Ubuntu 18.04?

How to Install and Secure Redis on Ubuntu 18.04?

In this Redis tutorial, you'll learn how to Install and Secure Redis on Ubuntu 18.04. You install and configure Redis, validate that your Redis installation is functioning correctly, and used its built-in security features to make it less vulnerable to attacks from malicious actors.

Introduction

Redis is an in-memory key-value store known for its flexibility, performance, and wide language support. This tutorial demonstrates how to install, configure, and secure Redis on an Ubuntu 18.04 server.

Step 1 — Installing and Configuring Redis

In order to get the latest version of Redis, we will use apt to install it from the official Ubuntu repositories.

Update your local apt package cache and install Redis by typing:

sudo apt update
sudo apt install redis-server

This will download and install Redis and its dependencies. Following this, there is one important configuration change to make in the Redis configuration file, which was generated automatically during the installation.

Open this file with your preferred text editor:

sudo nano /etc/redis/redis.conf

Inside the file, find the supervised directive. This directive allows you to declare an init system to manage Redis as a service, providing you with more control over its operation. The supervised directive is set to no by default. Since you are running Ubuntu, which uses the systemd init system, change this to systemd:

. . .

# If you run Redis from upstart or systemd, Redis can interact with your
# supervision tree. Options:
#   supervised no      - no supervision interaction
#   supervised upstart - signal upstart by putting Redis into SIGSTOP mode
#   supervised systemd - signal systemd by writing READY=1 to $NOTIFY_SOCKET
#   supervised auto    - detect upstart or systemd method based on
#                        UPSTART_JOB or NOTIFY_SOCKET environment variables
# Note: these supervision methods only signal "process is ready."
#       They do not enable continuous liveness pings back to your supervisor.
supervised systemd

. . .

That’s the only change you need to make to the Redis configuration file at this point, so save and close it when you are finished. Then, restart the Redis service to reflect the changes you made to the configuration file:

sudo systemctl restart redis.service

With that, you’ve installed and configured Redis and it’s running on your machine. Before you begin using it, though, it’s prudent to first check whether Redis is functioning correctly.

Step 2 — Testing Redis

As with any newly-installed software, it’s a good idea to ensure that Redis is functioning as expected before making any further changes to its configuration. We will go over a handful of ways to check that Redis is working correctly in this step.

Start by checking that the Redis service is running:

sudo systemctl status redis

If it is running without any errors, this command will produce output similar to the following:

Output● redis-server.service - Advanced key-value store
   Loaded: loaded (/lib/systemd/system/redis-server.service; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2018-06-27 18:48:52 UTC; 12s ago
     Docs: http://redis.io/documentation,
           man:redis-server(1)
  Process: 2421 ExecStop=/bin/kill -s TERM $MAINPID (code=exited, status=0/SUCCESS)
  Process: 2424 ExecStart=/usr/bin/redis-server /etc/redis/redis.conf (code=exited, status=0/SUCCESS)
 Main PID: 2445 (redis-server)
    Tasks: 4 (limit: 4704)
   CGroup: /system.slice/redis-server.service
           └─2445 /usr/bin/redis-server 127.0.0.1:6379
. . .

Here, you can see that Redis is running and is already enabled, meaning that it is set to start up every time the server boots.

Note: This setting is desirable for many common use cases of Redis. If, however, you prefer to start up Redis manually every time your server boots, you can configure this with the following command:

sudo systemctl disable redis

To test that Redis is functioning correctly, connect to the server using the command-line client:

redis-cli

In the prompt that follows, test connectivity with the ping command:

ping

OutputPONG

This output confirms that the server connection is still alive. Next, check that you’re able to set keys by running:

set test "It's working!"

OutputOK

Retrieve the value by typing:

get test

Assuming everything is working, you will be able to retrieve the value you stored:

Output"It's working!"

After confirming that you can fetch the value, exit the Redis prompt to get back to the shell:

exit

As a final test, we will check whether Redis is able to persist data even after it’s been stopped or restarted. To do this, first restart the Redis instance:

sudo systemctl restart redis

Then connect with the command-line client once again and confirm that your test value is still available:

redis-cli

get test

The value of your key should still be accessible:

Output"It's working!"

Exit out into the shell again when you are finished:

exit

With that, your Redis installation is fully operational and ready for you to use. However, some of its default configuration settings are insecure and provide malicious actors with opportunities to attack and gain access to your server and its data. The remaining steps in this tutorial cover methods for mitigating these vulnerabilities, as prescribed by the official Redis website. Although these steps are optional and Redis will still function if you choose not to follow them, it is strongly recommended that you complete them in order to harden your system’s security.

Step 3 — Binding to localhost

By default, Redis is only accessible from localhost. However, if you installed and configured Redis by following a different tutorial than this one, you might have updated the configuration file to allow connections from anywhere. This is not as secure as binding to localhost.

To correct this, open the Redis configuration file for editing:

sudo nano /etc/redis/redis.conf

Locate this line and make sure it is uncommented (remove the # if it exists):

bind 127.0.0.1 ::1

Save and close the file when finished (press CTRL + X, Y, then ENTER).

Then, restart the service to ensure that systemd reads your changes:

sudo systemctl restart redis

To check that this change has gone into effect, run the following netstat command:

sudo netstat -lnp | grep redis

Outputtcp        0      0 127.0.0.1:6379          0.0.0.0:*               LISTEN      14222/redis-server  
tcp6       0      0 ::1:6379                :::*                    LISTEN      14222/redis-server  

This output shows that the redis-server program is bound to localhost (127.0.0.1), reflecting the change you just made to the configuration file. If you see another IP address in that column (0.0.0.0, for example), then you should double check that you uncommented the correct line and restart the Redis service again.

Now that your Redis installation is only listening in on localhost, it will be more difficult for malicious actors to make requests or gain access to your server. However, Redis isn’t currently set to require users to authenticate themselves before making changes to its configuration or the data it holds. To remedy this, Redis allows you to require users to authenticate with a password before making changes via the Redis client (redis-cli).

Step 4 — Configuring a Redis Password

Configuring a Redis password enables one of its two built-in security features — the auth command, which requires clients to authenticate to access the database. The password is configured directly in Redis’s configuration file, /etc/redis/redis.conf, so open that file again with your preferred editor:

sudo nano /etc/redis/redis.conf

Scroll to the SECURITY section and look for a commented directive that reads:

/etc/redis/redis.conf
# requirepass foobared

Uncomment it by removing the #, and change foobared to a secure password.

Note: Above the requirepass directive in the redis.conf file, there is a commented warning:

# Warning: since Redis is pretty fast an outside user can try up to
# 150k passwords per second against a good box. This means that you should
# use a very strong password otherwise it will be very easy to break.
#

Thus, it’s important that you specify a very strong and very long value as your password. Rather than make up a password yourself, you can use the openssl command to generate a random one, as in the following example. By piping the output of the first command to the second openssl command, as shown here, it will remove any line breaks produced by that the first command:

openssl rand 60 | openssl base64 -A

Your output should look something like:

OutputRBOJ9cCNoGCKhlEBwQLHri1g+atWgn4Xn4HwNUbtzoVxAYxkiYBi7aufl4MILv1nxBqR4L6NNzI0X6cE

After copying and pasting the output of that command as the new value for requirepass, it should read:

/etc/redis/redis.confrequirepass RBOJ9cCNoGCKhlEBwQLHri1g+atWgn4Xn4HwNUbtzoVxAYxkiYBi7aufl4MILv1nxBqR4L6NNzI0X6cE

After setting the password, save and close the file, then restart Redis:

sudo systemctl restart redis.service

To test that the password works, access the Redis command line:

redis-cli

The following shows a sequence of commands used to test whether the Redis password works. The first command tries to set a key to a value before authentication:

set key1 10

That won’t work because you didn’t authenticate, so Redis returns an error:

Output(error) NOAUTH Authentication required.

The next command authenticates with the password specified in the Redis configuration file:

auth your_redis_password

Redis acknowledges:

OutputOK

After that, running the previous command again will succeed:

set key1 10

OutputOK

get key1 queries Redis for the value of the new key.

get key1

Output"10"

After confirming that you’re able to run commands in the Redis client after authenticating, you can exit the redis-cli:

quit

Next, we’ll look at renaming Redis commands which, if entered by mistake or by a malicious actor, could cause serious damage to your machine.

Step 5 — Renaming Dangerous Commands

The other security feature built into Redis involves renaming or completely disabling certain commands that are considered dangerous.

When run by unauthorized users, such commands can be used to reconfigure, destroy, or otherwise wipe your data. Like the authentication password, renaming or disabling commands is configured in the same SECURITY section of the /etc/redis/redis.conf file.

Some of the commands that are considered dangerous include: FLUSHDB, FLUSHALL, KEYS, PEXPIRE, DEL, CONFIG, SHUTDOWN, BGREWRITEAOF, BGSAVE, SAVE, SPOP, SREM, RENAME, and DEBUG. This is not a comprehensive list, but renaming or disabling all of the commands in that list is a good starting point for enhancing your Redis server’s security.

Whether you should disable or rename a command depends on your specific needs or those of your site. If you know you will never use a command that could be abused, then you may disable it. Otherwise, it might be in your best interest to rename it.

To enable or disable Redis commands, open the configuration file once more:

sudo nano  /etc/redis/redis.conf

Warning: The following steps showing how to disable and rename commands are examples. You should only choose to disable or rename the commands that make sense for you. You can review the full list of commands for yourself and determine how they might be misused at redis.io/commands.

To disable a command, simply rename it to an empty string (signified by a pair of quotation marks with no characters between them), as shown below:

. . .
# It is also possible to completely kill a command by renaming it into
# an empty string:
#
rename-command FLUSHDB ""
rename-command FLUSHALL ""
rename-command DEBUG ""
. . .

To rename a command, give it another name as shown in the examples below. Renamed commands should be difficult for others to guess, but easy for you to remember:

. . .
# rename-command CONFIG ""
rename-command SHUTDOWN SHUTDOWN_MENOT
rename-command CONFIG ASC12_CONFIG
. . .

Save your changes and close the file.

After renaming a command, apply the change by restarting Redis:

sudo systemctl restart redis.service

To test the new command, enter the Redis command line:

redis-cli

Then, authenticate:

auth your_redis_password

OutputOK

Let’s assume that you renamed the CONFIG command to ASC12_CONFIG, as in the preceding example. First, try using the original CONFIG command. It should fail, because you’ve renamed it:

config get requirepass

Output(error) ERR unknown command 'config'

Calling the renamed command, however, will be successful. It is not case-sensitive:

asc12_config get requirepass

Output1) "requirepass"
2) "your_redis_password"

Finally, you can exit from redis-cli:

exit

Note that if you’re already using the Redis command line and then restart Redis, you’ll need to re-authenticate. Otherwise, you’ll get this error if you type a command:

OutputNOAUTH Authentication required.

Regarding the practice of renaming commands, there’s a cautionary statement at the end of the SECURITY section in /etc/redis/redis.conf which reads:

Please note that changing the name of commands that are logged into the AOF file or transmitted to slaves may cause problems.

That means if the renamed command is not in the AOF file, or if it is but the AOF file has not been transmitted to slaves, then there should be no problem.

So, keep that in mind when you’re trying to rename commands. The best time to rename a command is when you’re not using AOF persistence, or right after installation, that is, before your Redis-using application has been deployed.

When you’re using AOF and dealing with a master-slave installation, consider this answer from the project’s GitHub issue page. The following is a reply to the author’s question:

The commands are logged to the AOF and replicated to the slave the same way they are sent, so if you try to replay the AOF on an instance that doesn’t have the same renaming, you may face inconsistencies as the command cannot be executed (same for slaves).

Thus, the best way to handle renaming in cases like that is to make sure that renamed commands are applied to all instances in master-slave installations.

Conclusion

In this tutorial, you installed and configured Redis, validated that your Redis installation is functioning correctly, and used its built-in security features to make it less vulnerable to attacks from malicious actors.

Originally published by Mark Drake at https://www.digitalocean.com

What is Redis - All You Need To Know About Redis

What is Redis - All You Need To Know About Redis

What is Redis and why use it? Our Redis guide tells you all you need to know about Redis, a highly performant in-memory data store and caching system. Redis is an open source (BSD licensed), in-memory data structure store, used as a database, cache and message broker.

Introduction

According to its GitHub repository, Redis (which stands for Remote Directory Server) is an in-memory data structure store. It is a disk-persistent key-value database with support for multiple data structures or data types.

This means that while Redis supports mapped key-value-based strings to store and retrieve data (analogous to the data model supported in traditional kinds of databases), it also supports other complex data structures like lists, sets, etc. As we proceed, we will look at the data structures supported by Redis. We will also get to learn about the unique features of Redis.

Redis is an open-source, highly replicated, performant, non-relational kind of database and caching server. It works by mapping keys to values with a sort of predefined data model. Its benefits include:

  • Mapped key-value-based caching system, almost comparable to memcached
  • No strict rules pertaining to defining strict schemas or tables for data (schemaless)
  • Support for multiple data models or types
  • Offers more advanced features when compared to other kinds of database systems
  • Ability to withstand multiple concurrent write requests or transactions per second, via a technique known as sharding
  • Can be used together with other databases as a support to reduce load and improve performance, but can also be used as a primary database. Note that this is usually based on individual needs and use cases
  • Can come in handy in the areas of quick data ingestion with data integrity in the mix, were features like high efficiency and replication are paramount

Note: Redis has a variety of use cases in large enterprise applications. Apart from acting as a caching server, it can also act as a message broker or used in publisher/subscriber kind of systems. For detailed information about other use cases, we can check this section of the documentation.

Prerequisites

Before we proceed, it is advisable to have Node and npm installed on our machines. Instructions on how to do so are available here. Furthermore, to install Redis locally, we can visit this section of the documentation.

While we will be making use of Redis’ cloud-hosted version — Redis Labs — we will also run through setting it up locally on our machines. This will enable us to learn while using the command line utility.

Installing Redis

To install Redis on our local machines, we can download the latest available binaries. Alternatively, we can use the following command to download as well:

wget http://download.redis.io/releases/redis-5.0.7.tar.gz

After downloading the binaries, we can go ahead and extract it:

tar xzf redis-5.0.7.tar.gz
cd redis-5.0.7
make
ls
cd src/

As seen above, after extracting the binaries, we can then navigate into the redis directory and run the make command, which compiles and builds all executables needed for Redis to function properly. Note that at the time of writing this article, the current Redis version is 5.0.7.

To start up Redis locally, all we have to do is run the following command:

./redis-server

To test that the Redis connection to the server is successful, we can open a new terminal window and run:

redis-cli ping

The output should be:

PONG

Finally, to start the Redis CLI, so as to experiment and start learning how to work with Redis commands, we can run:

./redis-cli

After starting the CLI, we can go ahead and run the info command to see all the parameters of the current running Redis instance. The output is shown below. Note that it is truncated here for brevity:

127.0.0.1:6379> info
# Server
redis_version:5.0.7
redis_git_sha1:00000000
redis_git_dirty:0
redis_build_id:e3be448653a99bb8
redis_mode:standalone
os:Darwin 18.7.0 x86_64
arch_bits:64
multiplexing_api:kqueue
atomicvar_api:atomic-builtin
gcc_version:4.2.1
process_id:98832
run_id:c63164944a269066f81b9bbc553296614fcb3df6
tcp_port:6379
uptime_in_seconds:374
uptime_in_days:0
hz:10
configured_hz:10
lru_clock:443996
executable:/Users/alexander.nnakwue/Downloads/redis-5.0.7/src/./redis-server
config_file:

# Clients
connected_clients:1
client_recent_max_input_buffer:2
client_recent_max_output_buffer:0
blocked_clients:0

Let’s experiment on the redis-cli with the most basic Redis commands, SET and GET, just to be certain we are connected successfully to the server:

SET mykey "Hello world"

Here we are setting the key mykey to the string value "Hello world". The output of this command is shown below:

OK

Now, we can run a get command with the key to get the value back:

get mykey

The output is shown below:

"Hello world"

For a complete list of all the available Redis commands, a reference is available in this section of the Redis documentation.

Note: To install Redis properly, check this section of the documentation. This is absolutely necessary if we intend not to go with a managed Redis server/instance in a live or production application. However, for hacking around and for development purposes, our previous installation setup works fine.

Cloud alternative with Redis Labs

To begin, we can visit Redis Labs, a fully cloud-managed alternative to working with Redis, to set up our account. In this tutorial, we will be making use of the free tier, which is basically a limited option meant for development and testing purposes.


Figure 1: Creating a new subscription for the free-tier option.

After we are done with the current configuration, we can then go ahead and set up our database as shown below:


Figure 2: Interface for database creation.

After we are done setting up our database, we should now be able to view our current subscription on the dashboard:


Figure 3: Subscription that contains the endpoint URL for the 30MB free tier.

Note that we can connect to the Redis database using the redis-cli or telnet. The command for connecting to the cluster via the CLI is shown below:

redis-cli -h <redic-host>  -p <redis-port> -a <password>

More information on the setup can be found here.

Use cases for Redis

Redis data stays in memory, as opposed to traditional kinds of databases that persist to disk. This gives Redis an edge over other kinds of storage systems and makes it ultra fast with high throughput and low latency. Therefore, Redis can be used in real-time applications and message queuing systems. Other use cases include:

Session management

Redis also comes in handy in managing user sessions at the application level. Typical web applications store user session information about the user login, user IDs, recent user actions, and so on. Details on implementing session management in Redis can found here.

Caching

For data that is frequently needed or retrieved by app users, a cache would serve as a temporary data store for quick and fast retrieval without the need for extra database round trips. Note that data stored in a cache is usually data from an earlier query or copy of data stored somewhere else. This feature is vital because the more data we can fetch from a cache, the faster and more efficiently the system performs overall.

More detailed information of Redis use cases like publisher/subscriber systems, fraud detection, leaderboard and data ingestion, etc. can be found in this section of the Redis enterprise documentation.

Redis data structures and associated commands

Redis is very versatile when it comes to how we model data based on particular use cases. As mentioned earlier, it allows us to map keys to strings and other available supported data types. The other supported data types/models include lists, sets, hashes, sorted sets, streams, and so on.

Having other data models supported in Redis eliminates or reduces to the barest minimum the time needed to convert one data type to another, contrary to traditional databases.

For detailed information about the definition and use cases for each data type, you can check this section of the Redis documentation, which explains thoroughly how and when to issue Redis commands on these data types against a Redis server. A complete listing of all Redis commands can also be found here here.

In the documentation/guide, each command is mapped to the respective data types it can operate on. We will be experimenting with just a few of these commands in the next section. Now let’s learn about some of the data structures.

String

Redis has commands for operating on some parts or the whole of the string data type. The string data type can also store integers and floating point numbers. Here is the link for the documentation on the commands used for operating on strings. We have previously seen an example of how to use the string data type.

Sets

Redis sets store unique sets of numbers, almost similar to sets in most programming languages. We can add and remove items to a set, check if an item exists, and so on. Details about using sets can be found here. For the associated commands on the set data structure, we can check this section of the documentation.

Sorted sets

Sorted sets in Redis are like regular sets, but are ordered by a score. Apart from normal operation on sets like addition and deletion, we can also fetch details in a sorted set by their scores. For the associated commands on the sorted set data structure, we can check this section of the documentation.

Lists

Redis also support the list data structure. Lists hold a collection of strings in order of insertion, i.e., FIFO (first-in, first-out). We can pop and push items from both ends, etc. For the associated commands on the list data structure, we can check this section of the documentation.

Hashes

Redis hashes store a set of field-value pair. We can perform operations like add, remove, and fetch individual items from the hash. We can also use the hash as a counter, using the INCRBY() command and others. For the associated commands on the hash data structure, we can check this section of the documentation.

Note that a link to learn about other data structures available in Redis like bitmaps, streams, HyperLogLog, bitfields, etc. can be found here.

For keys, we can check if they exist. Also, we can delete or expire a key and even increase/decrease a key.

In the next section on Node.js, we will learn how to use and apply these commands in real-life applications. Stay tuned!

Redis and Node.js

To begin, we can create a directory and call it any name we want. We then navigate into the directory and run the npm init command. After all the required setup, we can install all the dependencies for our application by running:

npm install redis concurrently dotenv --save

We can then go ahead and create all the necessary files and folders as shown below:


Figure 4: Folder structure for our project.

From our subscription details, we can get the parameters needed to connect to our Redis instance. A sample is shown below, which is located in our sample.env file. The real-life credentials are stored in the env file, which are referenced in the config file. The files are shown below:

app_port=6006
host=host
port=port
password=password

The content of the config file is shown below:

const path = require('path');
require('dotenv').config();
const config= {
    port: process.env.APP_PORT,
    redis: {
    host: process.env.REDIS_HOST,
    port: process.env.REDIS_PORT,
    password: process.env.REDIS_PASSWORD
    }
}
module.exports = config;

As seen in the files above, we have the redisClientConfig file, where we handle the connection to our Redis server. The contents of the file are shown below:

const redis = require('redis');
const config = require('./config');
const client = redis.createClient(config.redis.port, config.redis.host, {no_ready_check: true});
client.auth(config.redis.password);
client.on('error', error => console.error('Error Connecting to the Redis Cluster', error));
client.on('connect', () => {
    console.log('Successfully connected to the Redis cluster!');
  });

module.exports = client;

Furthermore, the samples for the various test cases in applying Redis commands are available in the test.js file located in the app directory. Note that we can also choose to test these commands against the data structures using the redis-cli.

The file’s contents are shown below:

const redisConnection = require('../redisClientConnection');

// test redis server
redisConnection.set('foo', 'bar');
    redisConnection.get('foo', function (err, res) {
        console.log(res.toString()); 
    });

// Retrieving a string value from Redis if it already exists for this key - Redis cache example 
redisConnection.get('myStringKey', (err, value) => {
    if (value) {
    console.log(`The value associated with this key is:${value}`)
    }
    else { 
    // Storing a simple string in the Redis store
    redisConnection.set('myStringKey', 'LogRocket Redis Tutorial');
    }
});

As seen in the snippet above, we are importing the connection to the Redis cluster. Then, we are using the commands against the data structures based on use cases or need.

A bunch of exhaustive Redis examples with real-world use cases are available in this GitHub repository. Note that this is meant for interacting with Redis using the Node.js client library. Finally, here’s the repo for this project.

Note that we can test our code using the redis-cli or we can as well start our Redis server. We can do so by running the npm run dev command, we get the following output:

Figure 5: Output from the console when we start the app.

Conclusion

In summary, Redis offers highly performant and efficient read and write via the optimizations outlined in this tutorial. For more information on learning Redis, with available resources outlined, you can visit the documentation.

We touched on basic Redis commands in this tutorial, as well as some more complex commands and how to use them in our application. Instructions to do so using Node.js has been mentioned earlier.

How to setup TensorFlow on Ubuntu

How to setup TensorFlow on Ubuntu

How to setup TensorFlow on Ubuntu - This tutorial will help you set up TensorFlow 1.12 on Ubuntu 16.04 with a GPU using Docker and nvidia-docker.

TensorFlow is one of the most popular deep-learning libraries. It was created by Google and was released as an open-source project in 2015. TensorFlow is used for both research and production environments. Installing TensorFlow can be cumbersome. The difficulty varies based on your environment constraints, and more when you’re a data scientist that just wants to build your neural networks.

When using TensorFlow on GPU — setting up requires a few steps. In the following tutorial, we will go over the process required to set up TensorFlow.


You may also like:5 TensorFlow and ML Courses for Programmers


Requirements:

Step 1 — Prepare your environment with Docker and Nvidia-Docker

Docker is a tool designed to make it easier to create, deploy, and run applications by using containers. What exactly is a container? Containers allow data scientists and developers to wrap up an environment with all of the parts it needs — such as libraries and other dependencies — and ship it all out in one package.

To use docker with GPUs and to be able to use TensorFlow in your application, you’ll need to install Docker with Nvidia-Docker. If you already have those installed, move to the next step. Otherwise, you can follow our previous guide to installing nvidia docker.

Prerequisites: Step 2 — Dockerfile

Docker can build images (environments) automatically by reading the instructions from a Dockerfile. A Dockerfile is a text document that contains all the commands a user could call on the command line to assemble an image.

In our case, those commands will describe the installation of Python 3.6, CUDA 9 and CUDNN 7.2.1 — and of course the installation of TensorFlow 1.12 from source.

For this environment, we will use the following Dockerfile

FROM nvidia/cuda:9.0-base-ubuntu16.04

RUN apt-get update && apt-get install -y --no-install-recommends \
        build-essential \
        cuda-command-line-tools-9-0 \
        cuda-cublas-dev-9-0 \
        cuda-cudart-dev-9-0 \
        cuda-cufft-dev-9-0 \
        cuda-curand-dev-9-0 \
        cuda-cusolver-dev-9-0 \
        cuda-cusparse-dev-9-0 \
        curl \
        git \
        libcudnn7=7.2.1.38-1+cuda9.0 \
        libcudnn7-dev=7.2.1.38-1+cuda9.0 \
	libnccl2=2.4.2-1+cuda9.0 \
	libnccl-dev=2.4.2-1+cuda9.0 \
        libcurl3-dev \
        libfreetype6-dev \
        libhdf5-serial-dev \
        libpng12-dev \
        libzmq3-dev \
        pkg-config \
        rsync \
        software-properties-common \
        unzip \
        zip \
        zlib1g-dev \
        wget \
        && \
    rm -rf /var/lib/apt/lists/* && \
    find /usr/local/cuda-9.0/lib64/ -type f -name 'lib*_static.a' -not -name 'libcudart_static.a' -delete && \
    rm /usr/lib/x86_64-linux-gnu/libcudnn_static_v7.a

# install python 3.6 and pip

RUN apt-get update
RUN apt-get install -y software-properties-common vim
RUN add-apt-repository ppa:jonathonf/python-3.6
RUN apt-get update

RUN apt-get install -y build-essential python3.6 python3.6-dev python3-pip python3.6-venv
RUN apt-get install -y git

RUN apt-get update && \
        apt-get install nvinfer-runtime-trt-repo-ubuntu1604-4.0.1-ga-cuda9.0 && \
        apt-get update && \
        apt-get install libnvinfer4=4.1.2-1+cuda9.0 && \
        apt-get install libnvinfer-dev=4.1.2-1+cuda9.0

RUN python3.6 -m pip install pip --upgrade
RUN python3.6 -m pip install wheel 
RUN python3.6 -m pip install six numpy wheel mock
RUN python3.6 -m pip install keras_applications
RUN python3.6 -m pip install keras_preprocessing

RUN ln -s /usr/bin/python3.6 /usr/bin/python


# Set up Bazel.

# Running bazel inside a `docker build` command causes trouble, cf:
#   https://github.com/bazelbuild/bazel/issues/134
# The easiest solution is to set up a bazelrc file forcing --batch.
RUN echo "startup --batch" >>/etc/bazel.bazelrc
# Similarly, we need to workaround sandboxing issues:
#   https://github.com/bazelbuild/bazel/issues/418
RUN echo "build --spawn_strategy=standalone --genrule_strategy=standalone" \
    >>/etc/bazel.bazelrc
# Install the most recent bazel release.
ENV BAZEL_VERSION 0.15.0
WORKDIR /
RUN mkdir /bazel && \
    cd /bazel && \
    curl -H "User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.133 Safari/537.36" -fSsL -O https://github.com/bazelbuild/bazel/releases/download/$BAZEL_VERSION/bazel-$BAZEL_VERSION-installer-linux-x86_64.sh && \
    curl -H "User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.133 Safari/537.36" -fSsL -o /bazel/LICENSE.txt https://raw.githubusercontent.com/bazelbuild/bazel/master/LICENSE && \
    chmod +x bazel-*.sh && \
    ./bazel-$BAZEL_VERSION-installer-linux-x86_64.sh && \
    cd / && \
    rm -f /bazel/bazel-$BAZEL_VERSION-installer-linux-x86_64.sh




# Download and build TensorFlow.
WORKDIR /tensorflow
RUN git clone --branch=r1.12 --depth=1 https://github.com/tensorflow/tensorflow.git .

# Configure the build for our CUDA configuration.
ENV CI_BUILD_PYTHON python3.6
ENV LD_LIBRARY_PATH /usr/local/cuda/extras/CUPTI/lib64:$LD_LIBRARY_PATH
ENV TF_NEED_CUDA 1
ENV TF_NEED_TENSORRT 1
ENV TF_CUDA_COMPUTE_CAPABILITIES=3.5,5.2,6.0,6.1,7.0
ENV TF_CUDA_VERSION=9.0
ENV TF_CUDNN_VERSION=7

RUN ln -s /usr/local/cuda/lib64/stubs/libcuda.so /usr/local/cuda/lib64/stubs/libcuda.so.1 && \
    LD_LIBRARY_PATH=/usr/local/cuda/lib64/stubs:${LD_LIBRARY_PATH} \
    tensorflow/tools/ci_build/builds/configured GPU \
    bazel build -c opt --copt=-mavx --config=cuda \
	--cxxopt="-D_GLIBCXX_USE_CXX11_ABI=0" \
        tensorflow/tools/pip_package:build_pip_package && \
    rm /usr/local/cuda/lib64/stubs/libcuda.so.1 && \
    bazel-bin/tensorflow/tools/pip_package/build_pip_package /tmp/pip && \
    pip --no-cache-dir install --upgrade /tmp/pip/tensorflow-*.whl && \
    rm -rf /tmp/pip && \
    rm -rf /root/.cache
# Clean up pip wheel and Bazel cache when done.

WORKDIR /root

# TensorBoard
EXPOSE 6006

Step 3 — Running Dockerfile

To build the image from the Dockerfile, simply run the docker build command. Keep in mind that this build process might take a few hours to complete. We recommend using nohup utility so that if your terminal hangs — it will still run.

$ docker build -t deeplearning -f Dockerfile

This should output the setup process and should end with something similar to:

>> Successfully built deeplearning (= the image ID)

Your image is ready to use. To start the environment, simply type in the below command. But, don’t forget to replace your image id:

$ docker run --runtime=nvidia -it deeplearning /bin/bashStep 4 — Validating TensorFlow & start building!

Validate that TensorFlow is indeed running in your Dockerfile

$ python
import tensorflow as tf
sess = tf.Session(config=tf.ConfigProto(log_device_placement=True))
2019-02-23 07:34:14.592926: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports 
instructions that this TensorFlow binary was not compiled to use: AVX2 FMA2019-02-23 07:34:17.452780: I 
tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:964] successful NUMA
node read from SysFS had negative value (-1), but there must be at leastone NUMA node, so returning NUMA node zero
2019-02-23 07:34:17.453267: I
tensorflow/core/common_runtime/gpu/gpu_device.cc:1432] Found device 0 with 
properties: 
name: Tesla K80 major: 3 minor: 7 memoryClockRate(GHz): 0.8235pciBusID: 0000:00:1e.0
totalMemory: 11.17GiB freeMemory: 11.10GiB
2019-02-23 07:34:17.453306: I
tensorflow/core/common_runtime/gpu/gpu_device.cc:1511] Adding visible gpu
devices: 0
2019-02-23 07:34:17.772969: I
tensorflow/core/common_runtime/gpu/gpu_device.cc:982] Device interconnect
StreamExecutor with strength 1 edge matrix:
2019-02-23 07:34:17.773032: I
tensorflow/core/common_runtime/gpu/gpu_device.cc:988]      0
2019-02-23 07:34:17.773054: I
tensorflow/core/common_runtime/gpu/gpu_device.cc:1001] 0:   N
2019-02-23 07:34:17.773403: I
tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] Created TensorFlow
device (/job:localhost/replica:0/task:0/device:GPU:0 with 10757 MB memory)
-> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:1e.0,
compute capability: 3.7)
Device mapping:
/job:localhost/replica:0/task:0/device:XLA_CPU:0 -> device: XLA_CPU device
/job:localhost/replica:0/task:0/device:XLA_GPU:0 -> device: XLA_GPU device
/job:localhost/replica:0/task:0/device:GPU:0 -> device: 0, name: Tesla K80,
pci bus id: 0000:00:1e.0, compute capability: 3.7
2019-02-23 07:34:17.774289: I
tensorflow/core/common_runtime/direct_session.cc:307] Device mapping:
/job:localhost/replica:0/task:0/device:XLA_CPU:0 -> device: XLA_CPU device
/job:localhost/replica:0/task:0/device:XLA_GPU:0 -> device: XLA_GPU device
/job:localhost/replica:0/task:0/device:GPU:0 -> device: 0, name: Tesla K80, pci bus id: 0000:0

Congrats! Your new TensorFlow environment is set up and ready to start training, testing and deploying your deep learning models!

Conclusion

Tensorflow has truly disrupted the machine learning world by offering a solution to build production-ready models at scale. But Tensorflow is not always the most user-friendly. It can be difficult to smoothly incorporate into your machine learning pipeline. cnvrg.io data science platform leverages Tensorflow and other open-source tools so that data scientists can focus on the magic — the algorithms. You can find more tutorials on how to easily leverage open-source tools like TensorflowHow to set up Kubernetes for your machine learning workflows, and How to run Spark on Kubernetes. Finding simple ways to integrate these useful tools will get your models closer to production.

Further Reading

How To Set Up Django with Postgres, Nginx, and Gunicorn on Ubuntu 16.04

How To Install Python 3 and Set Up a Programming Environment on Ubuntu 18.04

How To Set Up Jupyter Notebook with Python 3 on Ubuntu 18.04

Originally published by*** *yochze **at towardsdatascience.com


Thanks for reading :heart: If you liked this post, share it with all of your programming buddies! Follow me on Facebook | Twitter