Udocker: A Basic User tool To Execute Simple Docker Containers

udocker is a basic user tool to execute simple docker containers in user space without requiring root privileges. Enables download and execution of docker containers by non-privileged users in Linux systems where docker is not available. It can be used to pull and execute docker containers in Linux batch systems and interactive clusters that are managed by other entities such as grid infrastructures or externally managed batch or interactive systems.

udocker does not require any type of privileges nor the deployment of services by system administrators. It can be downloaded and executed entirely by the end user. The limited root functionality provided by some of the udocker execution modes is either simulated or provided via user namespaces.

udocker is a wrapper around several tools and libraries to mimic a subset of the docker capabilities including pulling images and running containers with minimal functionality.

How does it work

udocker is written in Python, it has a minimal set of dependencies so that can be executed in a wide range of Linux systems.

udocker does not make use of docker nor requires its presence.

udocker "executes" the containers by simply providing a chroot like environment over the extracted container. The current implementation supports different methods to mimic chroot thus enabling execution of containers under a chroot like environment without requiring privileges. udocker transparently supports several methods to execute the containers based on external tools and libraries such as:

PRoot
Fakechroot
runc
crun
Singularity

With the exception of Singularity the tools and libraries to support execution are downloaded and deployed by udocker during the installation process. This installation is performed in the user home directory and does not require privileges. The udocker related files such as libraries, executables, documentation, licenses, container images and extracted directory trees are placed by default under $HOME/.udocker.

Advantages

Can be deployed by the end-user
Does not require privileges for installation
Does not require privileges for execution
Does not require compilation, just transfer the Python code
Encapsulates several tools and execution methods
Includes the required tools already statically compiled to work across systems
Provides a docker like command line interface
Supports a subset of docker commands: search, pull, import, export, load, save, login, logout, create and run
Understands docker container metadata
Allows loading of docker and OCI containers
Supports NVIDIA GPGPU applications
Can execute in systems and environments where Linux namespaces support is unavailable
Runs both on new and older Linux distributions including: CentOS 6, CentOS 7, CentOS 8, Ubuntu 14, Ubuntu 16, Ubuntu 18, Ubuntu 20, Ubuntu 21, Alpine, Fedora, etc

Python 2 and Python 3

Since v1.3.0 (or the development v1.2.x), udocker supports Python 2.6, 2.7 and Python >= 3.6. The original udocker v1.1.x for Python 2 only is still available here.

Syntax

        Commands:
          search <repo/expression>      :Search dockerhub for container images
          pull <repo/image:tag>         :Pull container image from dockerhub
          create <repo/image:tag>       :Create container from a pulled image
          run <container>               :Execute container

          images -l                     :List container images
          ps -m -s                      :List created containers
          name <container_id> <name>    :Give name to container
          rmname <name>                 :Delete name from container
          rename <name> <new_name>      :Change container name 
          clone <container_id>          :Duplicate container
          rm <container-id>             :Delete container
          rmi <repo/image:tag>          :Delete image

          import <tar> <repo/image:tag> :Import tar file (exported by docker)
          import - <repo/image:tag>     :Import from stdin (exported by docker)
          export -o <tar> <container>   :Export container directory tree
          export - <container>          :Export container directory tree
          load -i <imagefile>           :Load image from file (saved by docker)
          load                          :Load image from stdin (saved by docker)
          save -o <imagefile> <repo/image:tag>  :Save image with layers to file

          inspect -p <repo/image:tag>   :Return low level information on image
          verify <repo/image:tag>       :Verify a pulled or loaded image

          protect <repo/image:tag>      :Protect repository
          unprotect <repo/image:tag>    :Unprotect repository
          protect <container>           :Protect container
          unprotect <container>         :Unprotect container

          mkrepo <top-repo-dir>         :Create another repository in location
          setup                         :Change container execution settings
          login                         :Login into docker repository
          logout                        :Logout from docker repository

          help                          :This help
          run --help                    :Command specific help
          version                       :Shows udocker version

        Options common to all commands must appear before the command:
          -D                            :Debug
          --quiet                       :Less verbosity
          --repo=<directory>            :Use repository at directory
          --insecure                    :Allow insecure non authenticated https
          --allow-root                  :Allow execution by root NOT recommended

Examples

Some examples of usage:

Search container images in dockerhub.

udocker search  fedora
udocker search  ubuntu
udocker search  indigodatacloud

Pull from dockerhub and list the pulled images.

udocker pull   fedora:29
udocker pull   busybox
udocker pull   iscampos/openqcd
udocker images

Pull from a registry other than dockerhub.

udocker search  quay.io/bio
udocker search  --list-tags  quay.io/biocontainers/scikit-bio
udocker pull    quay.io/biocontainers/scikit-bio:0.2.3--np112py35_0
udocker images

Create a container from a pulled image, assign a name to the created container and run it. A created container can be run multiple times until it is explicitely removed.

udocker create --name=myfed  fedora:29
udocker run  myfed  cat /etc/redhat-release

The three steps of pulling, creating and running can be also achieved in a single command, however this will be much slower for multiple invocations of the same container, as a new container will be created for each invocation. This approach will also consume more storage space. The following example creates a new container for each invocation.

udocker run  fedora:29  cat /etc/redhat-release

Execute mounting the host /home/u457 into the container directory /home/cuser. Notice that you can "mount" any host directory inside the container. Depending on the execution mode the "mount" is implemented differently and may have restrictions.

udocker run -v /home/u457:/home/cuser -w /home/user myfed  /bin/bash
udocker run -v /var -v /proc -v /sys -v /tmp  myfed  /bin/bash

Place a script in your host /tmp and execute it in the container. Notice that the behavior of --entrypoint changed from the previous versions for better compatibility with docker.

udocker run  -v /tmp  --entrypoint="" myfed  /bin/bash -c 'cd /tmp; ./myscript.sh'

udocker run  -v /tmp  --entrypoint=/bin/bash  myfed  -c 'cd /tmp; ./myscript.sh'

Execute mounting the host /var, /proc, /sys and /tmp in the same container directories. Notice that the content of these container directories will be obfuscated.

udocker run -v /var -v /proc -v /sys -v /tmp  myfed  /bin/bash

Install software inside the container.

udocker run  --user=root myfed  yum install -y firefox pulseaudio gnash-plugin

Run as some user. The usernames should exist in the container.

udocker run --user 1000:1001  myfed  /bin/id
udocker run --user root   myfed  /bin/id
udocker run --user jorge  myfed  /bin/id

Running Firefox.

udocker run --bindhome --hostauth --hostenv \
   -v /sys -v /proc -v /var/run -v /dev --user=jorge --dri myfed  firefox

Change execution engine mode from PRoot to Fakechroot and run.

udocker setup  --execmode=F3  myfed

udocker run --bindhome --hostauth --hostenv \
   -v /sys -v /proc -v /var/run -v /dev --user=jorge --dri myfed  firefox

Change execution engine mode to accelerated PRoot.

udocker setup  --execmode=P1  myfed

Change execution engine to runc.

udocker setup  --execmode=R1  myfed

Change execution engine to Singularity. Requires the availability of Singularity in the host system.

./udocker setup  --execmode=S1  myfed

Install software running as root emulation in Singularity:

udocker setup  --execmode=S1  myfed
udocker run  --user=root myfed  yum install -y firefox pulseaudio gnash-plugin

Security

By default udocker via PRoot offers the emulation of the root user. This emulation mimics a real root user (e.g getuid will return 0). This is just an emulation no root privileges are involved. This feature makes possible the execution of some tools that do not require actual privileges but which refuse to work if the username or id are not root or 0. This enables for instance software installation using rpm, yum or dnf inside the container.

udocker does not offer robust isolation features such as the ones offered by docker. Therefore if the containers content is not trusted then these containers should not be executed with udocker as they will run inside the user environment. For this reason udocker should not be run by privileged users.

Container images and filesystems will be unpacked and stored in the user home directory under $HOME/.udocker or other location of choice. Therefore the containers data will be subjected to the same filesystem protections as other files owned by the user. If the containers have sensitive information the files and directories should be adequately protected by the user.

udocker does not require privileges and runs under the identity of the user invoking it. Users can downloaded udocker and execute it without requiring system administrators intervention.

udocker also provides execution with runc, crun and Singularity, these modes make use of rootless namespaces and enable a normal user to execute as root with the limitations that apply to user namespaces and to these tools.

When executed by normal unprivileged users, udocker limits privilege escalation issues since it does not use or require system privileges.

General Limitations

Since root privileges are not involved any operation that really requires such privileges will not be possible. The following are examples of operations that are not possible:

accessing host protected devices and files
listening on TCP/IP privileged ports (range below 1024)
mount file-systems
the su command will not work
change the system time
changing routing tables, firewall rules, or network interfaces

If the containers require such privilege capabilities then docker should be used instead.

udocker is not meant to create containers. Creation of containers is better performed using docker and dockerfiles.

udocker does not provide all the docker features, and is not intended as a docker replacement.

udocker is mainly oriented at providing a run-time environment for containers execution in user space. udocker is particularly suited to run user applications encapsulated in docker containers.

Debugging inside of udocker with the PRoot engine will not work due to the way PRoot implements the chroot environment

Execution mode specific limitations

udocker offers multiple execution modes leveraging several external tools such as PRoot (P mode), Fakechroot (F mode), runC (R mode), crun (R mode) and Singularity (S mode).

When using execution Fakechroot modes such as F2, F3 and F4 the created containers cannot be moved across hosts. In this case convert back to a Pn mode before transfer. This is not needed if the hosts are part of an homogeneous cluster where the mount points and directory structure is the same. This limitation applies whenever the absolute realpath to the container directory changes.

The default accelerated mode of PRoot (mode P1) may exhibit problems in Linux kernels above 4.0 due to kernel changes and upstream issues, in this case use mode P2 or any of the other execution modes.

./udocker setup  --execmode=P2  my-container-id

The Fakechroot modes (Fn modes) require shared libraries compiled against the libc shipped with the container. udocker provides these libraries for several Linux distributions, these shared libraries are installed by udocker under:

$HOME/.udocker/lib/libfakechroot-*

The runc and crun modes (R modes) require a kernel with user namespaces enabled.

The singularity mode (S mode) requires the availability of Singularity in the host system. Singularity is not shipped with udocker.

Metadata generation

The codemeta.json metadata file was initially generated with codemetapy package:

codemetapy udocker --with-orcid --affiliation "LIP Lisbon" \
  --buildInstructions "https://https://github.com/indigo-dc/udocker/blob/master/docs/installation_manual.md#3-source-code-and-build" \
  --citation "https://doi.org/10.1016/j.cpc.2018.05.021" \
  --codeRepository "https://github.com/indigo-dc/udocker" \
  --contIntegration "https://jenkins.eosc-synergy.eu/job/indigo-dc/job/udocker/job/master/" --contributor "Mario David" \
  --copyrightHolder "LIP"  --copyrightYear "2016" --creator "Jorge Gomes" \
  --dateCreated "2021-05-26" --maintainer "Jorge Gomes" \
  --readme "https://github.com/indigo-dc/udocker/blob/master/README.md" \
  --referencePublication "https://doi.org/10.1016/j.cpc.2018.05.021" \
  --releaseNotes "https://github.com/indigo-dc/udocker/blob/master/changelog" \
  -O codemeta.json

Further updates may be needed to add the correct values in the metadata file.

Documentation

The full documentation is available at:

udocker documentation

Contributing

See: Contributing

Citing

See: Citing

When citing udocker please use the following:

Jorge Gomes, Emanuele Bagnaschi, Isabel Campos, Mario David, Luís Alves, João Martins, João Pina, Alvaro López-García, Pablo Orviz, Enabling rootless Linux Containers in multi-user environments: The udocker tool, Computer Physics Communications, Available online 6 June 2018, ISSN 0010-4655, https://doi.org/10.1016/j.cpc.2018.05.021

Licensing

Redistribution, commercial use and code changes must regard all licenses shipped with udocker. These include the udocker license and the individual licences of the external tools and libraries packaged for use with udocker. For further information see the software licenses section of the installation manual.

Acknowledgements

Docker https://www.docker.com/
PRoot https://proot-me.github.io/
Fakechroot https://github.com/dex4er/fakechroot/wiki
runC https://runc.io/
crun https://github.com/containers/crun
Singularity https://www.sylabs.io/
Open Container Initiative https://www.opencontainers.org/
INDIGO DataCloud https://www.indigo-datacloud.eu
DEEP-Hybrid-DataCloud https://deep-hybrid-datacloud.eu
EOSC-hub https://eosc-hub.eu
EGI-ACE https://www.egi.eu/projects/egi-ace/
EOSC-Synergy https://www.eosc-synergy.eu/
LIP https://www.lip.pt
INCD https://www.incd.pt

This work was performed in the framework of the H2020 project INDIGO-Datacloud (RIA 653549) and further developed with co-funding by the projects EOSC-hub (Horizon 2020) under Grant number 777536 and DEEP-Hybrid-DataCloud (Horizon 2020) under Grant number 777435. Software Quality Assurance is performed with the support of by the project EOSC-Synergy (Horizon 2020). The authors wish to acknowleadge the support of INCD-Infraestrutura Nacional de Computação Distribuída (funded by FCT, P2020, Lisboa2020, COMPETE and FEDER under the project number 22153-01/SAICT/2016).

Download Details:

Author: indigo-dc
Source code: https://github.com/indigo-dc/udocker

License: Apache-2.0 license
#docker

What is GEEK

Buddha Community

Udocker: A Basic User tool To Execute Simple Docker Containers

Grace Lesch

1639778400

PySQL Tutorial: A Database Framework for Python

PySQL

PySQL is database framework for Python (v3.x) Language, Which is based on Python module mysql.connector, this module can help you to make your code more short and more easier. Before using this framework you must have knowledge about list, tuple, set, dictionary because all codes are designed using it. It's totally free and open source.

Tutorial Video in English (Watch Now)

IMAGE ALT TEXT HERE

Installation

Before we said that this framework is based on mysql.connector so you have to install mysql.connector first on your system. Then you can import pysql and enjoy coding!

python -m pip install mysql-connector-python

After Install mysql.connector successfully create Python file download/install pysql on the same dir where you want to create program. You can clone is using git or npm command, and you can also downlaod manually from repository site.

PyPi Command

Go to https://pypi.org/project/pysql-framework/ or use command

pip install pysql-framework

Git Command

git clone https://github.com/rohit-chouhan/pysql

Npm Command

Go to https://www.npmjs.com/package/pysql or use command

$ npm i pysql

Snippet Extention for VS Code

Install From Here https://marketplace.visualstudio.com/items?itemName=rohit-chouhan.pysql

IMAGE ALT TEXT HERE

Table of contents

Connecting a Server
Create a Database in Server
Drop Database
Connecting a Database
Creating Table in Database
Drop Table in Database
Selecting data from Table
Add New Column to Table
Modify Column to Table
Drop Column from Table
Manual Execute Query
Inserting data
Updating data
Deleting data

Connecting a Server

To connect a database with localhost server or phpmyadmin, use connect method to establish your python with database server.

import pysql

db = pysql.connect(
    "host",
    "username",
    "password"
 )

Create a Database in Server

Creating database in server, to use this method

import pysql

db = pysql.connect(
    "host",
    "username",
    "password"
 )
 pysql.createDb(db,"demo")
 #execute: CREATE DATABASE demo

Drop Database

To drop database use this method .

Syntex Code -

pysql.dropDb([connect_obj,"table_name"])

Example Code -

pysql.dropDb([db,"demo"])
#execute:DROP DATABASE demo

Connecting a Database

To connect a database with localhost server or phpmyadmin, use connect method to establish your python with database server.

import pysql

db = pysql.connect(
    "host",
    "username",
    "password",
    "database"
 )

Creating Table in Database

To create table in database use this method to pass column name as key and data type as value.

Syntex Code -


pysql.createTable([db,"table_name_to_create"],{
    "column_name":"data_type", 
    "column_name":"data_type"
})

Example Code -


pysql.createTable([db,"details"],{
    "id":"int(11) primary", 
     "name":"text", 
    "email":"varchar(50)",
    "address":"varchar(500)"
})

2nd Example Code -

Use can use any Constraint with Data Value


pysql.createTable([db,"details"],{
    "id":"int NOT NULL PRIMARY KEY", 
     "name":"varchar(20) NOT NULL", 
    "email":"varchar(50)",
    "address":"varchar(500)"
})

Drop Table in Database

To drop table in database use this method .

Syntex Code -

pysql.dropTable([connect_obj,"table_name"])

Example Code -

pysql.dropTable([db,"users"])
#execute:DROP TABLE users

Selecting data from Table

For Select data from table, you have to mention the connector object with table name. pass column names in set.

Syntex For All Data (*)-

records = pysql.selectAll([db,"table_name"])
for x in records:
  print(x)

Example - -

records = pysql.selectAll([db,"details"])
for x in records:
  print(x)
#execute: SELECT * FROM details

Syntex For Specific Column-

records = pysql.select([db,"table_name"],{"column","column"})
for x in records:
  print(x)

Example - -

records = pysql.select([db,"details"],{"name","email"})
for x in records:
  print(x)
#execute: SELECT name, email FROM details

Syntex Where and Where Not-

#For Where Column=Data
records = pysql.selectWhere([db,"table_name"],{"column","column"},("column","data"))

#For Where Not Column=Data (use ! with column)
records = pysql.selectWhere([db,"table_name"],{"column","column"},("column!","data"))
for x in records:
  print(x)

Example - -

records = pysql.selectWhere([db,"details"],{"name","email"},("county","india"))
for x in records:
  print(x)
#execute: SELECT name, email FROM details WHERE country='india'

Add New Column to Table

To add column in table, use this method to pass column name as key and data type as value. Note: you can only add one column only one call

Syntex Code -


pysql.addColumn([db,"table_name"],{
    "column_name":"data_type"
})

Example Code -


pysql.addColumn([db,"details"],{
    "email":"varchar(50)"
})
#execute: ALTER TABLE details ADD email varchar(50);

Modify Column to Table

To modify data type of column table, use this method to pass column name as key and data type as value.

Syntex Code -

pysql.modifyColumn([db,"table_name"],{
    "column_name":"new_data_type"
})

Example Code -

pysql.modifyColumn([db,"details"],{
    "email":"text"
})
#execute: ALTER TABLE details MODIFY COLUMN email text;

Drop Column from Table

Note: you can only add one column only one call

Syntex Code -

pysql.dropColumn([db,"table_name"],"column_name")

Example Code -

pysql.dropColumn([db,"details"],"name")
#execute: ALTER TABLE details DROP COLUMN name

Manual Execute Query

To execute manual SQL Query to use this method.

Syntex Code -

pysql.query(connector_object,your_query)

Example Code -

pysql.query(db,"INSERT INTO users (name) VALUES ('Rohit')")

Inserting data

For Inserting data in database, you have to mention the connector object with table name, and data as sets.

Syntex -

data =     {
    "db_column":"Data for Insert",
    "db_column":"Data for Insert"
}
pysql.insert([db,"table_name"],data)

Example Code -

data =     {
    "name":"Komal Sharma",
    "contry":"India"
}
pysql.insert([db,"users"],data)

Updating data

For Update data in database, you have to mention the connector object with table name, and data as tuple.

Syntex For Updating All Data-

data = ("column","data to update")
pysql.updateAll([db,"users"],data)

Example - -

data = ("name","Rohit")
pysql.updateAll([db,"users"],data)
#execute: UPDATE users SET name='Rohit'

Syntex For Updating Data (Where and Where Not)-

data = ("column","data to update")
#For Where Column=Data
where = ("column","data")

#For Where Not Column=Data (use ! with column)
where = ("column!","data")
pysql.update([db,"users"],data,where)

Example -

data = ("name","Rohit")
where = ("id",1)
pysql.update([db,"users"],data,where)
#execute: UPDATE users SET name='Rohit' WHERE id=1

Deleting data

For Delete data in database, you have to mention the connector object with table name.

Syntex For Delete All Data-

pysql.deleteAll([db,"table_name"])

Example - -

pysql.deleteAll([db,"users"])
#execute: DELETE FROM users

Syntex For Deleting Data (Where and Where Not)-

where = ("column","data")

pysql.delete([db,"table_name"],where)

Example -

#For Where Column=Data
where = ("id",1)

#For Where Not Column=Data (use ! with column)
where = ("id!",1)
pysql.delete([db,"users"],where)
#execute: DELETE FROM users WHERE id=1

--- Finish ---

Change Logs

[19/06/2021]
 - ConnectSever() removed and merged to Connect()
 - deleteAll() [Fixed]
 - dropTable() [Added]
 - dropDb() [Added]
 
[20/06/2021]
 - Where Not Docs [Added]

The module is designed by Rohit Chouhan, contact us for any bug report, feature or business inquiry.

Author: rohit-chouhan
Source Code: https://github.com/rohit-chouhan/pysql
License: Apache-2.0 License

#python

Christa Stehr

1602964260

50+ Useful Kubernetes Tools for 2020 - Part 2

Introduction

Last year, we provided a list of Kubernetes tools that proved so popular we have decided to curate another list of some useful additions for working with the platform—among which are many tools that we personally use here at Caylent. Check out the original tools list here in case you missed it.

According to a recent survey done by Stackrox, the dominance Kubernetes enjoys in the market continues to be reinforced, with 86% of respondents using it for container orchestration.

(State of Kubernetes and Container Security, 2020)

And as you can see below, more and more companies are jumping into containerization for their apps. If you’re among them, here are some tools to aid you going forward as Kubernetes continues its rapid growth.

(State of Kubernetes and Container Security, 2020)

#blog #tools #amazon elastic kubernetes service #application security #aws kms #botkube #caylent #cli #container monitoring #container orchestration tools #container security #containers #continuous delivery #continuous deployment #continuous integration #contour #developers #development #developments #draft #eksctl #firewall #gcp #github #harbor #helm #helm charts #helm-2to3 #helm-aws-secret-plugin #helm-docs #helm-operator-get-started #helm-secrets #iam #json #k-rail #k3s #k3sup #k8s #keel.sh #keycloak #kiali #kiam #klum #knative #krew #ksniff #kube #kube-prod-runtime #kube-ps1 #kube-scan #kube-state-metrics #kube2iam #kubeapps #kubebuilder #kubeconfig #kubectl #kubectl-aws-secrets #kubefwd #kubernetes #kubernetes command line tool #kubernetes configuration #kubernetes deployment #kubernetes in development #kubernetes in production #kubernetes ingress #kubernetes interfaces #kubernetes monitoring #kubernetes networking #kubernetes observability #kubernetes plugins #kubernetes secrets #kubernetes security #kubernetes security best practices #kubernetes security vendors #kubernetes service discovery #kubernetic #kubesec #kubeterminal #kubeval #kudo #kuma #microsoft azure key vault #mozilla sops #octant #octarine #open source #palo alto kubernetes security #permission-manager #pgp #rafay #rakess #rancher #rook #secrets operations #serverless function #service mesh #shell-operator #snyk #snyk container #sonobuoy #strongdm #tcpdump #tenkai #testing #tigera #tilt #vert.x #wireshark #yaml

Mikel Okuneva

1602317778

Ever Wondered Why We Use Containers In DevOps?

At some point we’ve all said the words, “But it works on my machine.” It usually happens during testing or when you’re trying to get a new project set up. Sometimes it happens when you pull down changes from an updated branch.

Every machine has different underlying states depending on the operating system, other installed programs, and permissions. Getting a project to run locally could take hours or even days because of weird system issues.

The worst part is that this can also happen in production. If the server is configured differently than what you’re running locally, your changes might not work as you expect and cause problems for users. There’s a way around all of these common issues using containers.

What is a container

A container is a piece of software that packages code and its dependencies so that the application can run in any computing environment. They basically create a little unit that you can put on any operating system and reliably and consistently run the application. You don’t have to worry about any of those underlying system issues creeping in later.

Although containers were already used in Linux for years, they became more popular in recent years. Most of the time when people are talking about containers, they’re referring to Docker containers. These containers are built from images that include all of the dependencies needed to run an application.

When you think of containers, virtual machines might also come to mind. They are very similar, but the big difference is that containers virtualize the operating system instead of the hardware. That’s what makes them so easy to run on all of the operating systems consistently.

What containers have to do with DevOps

Since we know how odd happenings occur when you move code from one computing environment to another, this is also a common issue with moving code to the different environments in our DevOps process. You don’t want to have to deal with system differences between staging and production. That would require more work than it should.

Once you have an artifact built, you should be able to use it in any environment from local to production. That’s the reason we use containers in DevOps. It’s also invaluable when you’re working with microservices. Docker containers used with something like Kubernetes will make it easier for you to handle larger systems with more moving pieces.

#devops #containers #containers-devops #devops-containers #devops-tools #devops-docker #docker #docker-image

Mary Turcotte

1661628180