Jammie Yost

Jammie Yost

1661526240

Custom Docker Registry That Allows For Deployments Through Bittorrent

Docket

DOCKEr + torrenT = Docket

Docket is a custom docker registry that allows for deployments through bittorrent. It allows for lightning fast Docker image deploys across large number of machines.

This was designed and built in 48 hours as part of the Gopher Gala Golang 48 hour hackathon. Hence kindly forgive me for the hackish code, and lack of tests.

[On hind sight - container rush or docker flood would have been a better name.]

Screencast

Problem Statement

As containers get more mainstream, server deployments are soon going to be done through a containerization solution like docker [or rocket or lxd].

Large scale deploys are going to choke your docker registry. Imagine pulling/deploying a 800mb base image (for example the official perl image) across 200 machines in one go. That's 800*200 = 160GB of data that's going to be deployed and it'll definitely choke your private docker registry (and will take a while pulling in from the public image)

Solution

The solution is BitTorrent. Any technology good for distributing music and movies, is usually good for doing server deploys. Instead of downloading from one point source, we can now distribute the load across the machines downloading themselves. All of this transparently and automatically handled by the docket client.

Silly Totally Non-scientific Benchmark

Pulling a 800 mb image across 5 machines simultaneously via Docket resulted in a 90% speedup.

Features

  • Written in Golang :)
  • Very easy to use
  • Works along side your private docker registry

Concepts

Docket constitutes of 3 components:

  1. Docket Registry

A REST service that acts as a registry. It receives docker image tarballs from the client, stores metadata into a database, creates torrents out of them and seeds them.

  1. Docket Client

The client itnerface in which the end user will be interacting. Can view available images in the registry, push an image to the registry and pull an image (which triggers a bittorrent deploy) from the registry.

  1. Bittorrent Tracker

Docket allows you to BYOT (bring your own tracker), but I suggest installing opentracker

Docket uses ctorrent for the actual downloading of torrents.

Installation

Step 0: You need to already have Docker installed

Step 1: Install ctorrent and opentracker

$ sudo apt-get update
$ sudo apt-get install zlib1g-dev make g++ ctorrent
$ git clone https://github.com/gebi/libowfat
$ make
$ git clone git://erdgeist.org/opentracker
$ cd opentracker
$ make

Step 2: Download the Docket binaries

$ wget http://storage.googleapis.com/docket/docket.zip
$ unzip docket.zip

Step 3: That's all

Getting Started

Step 1: Fire up the Tracker

$ cd opentracker
$ ./opentracker 8940

Step 2: Fire up the Registry

$ cd registry
# Note you need to put an ip address in which other machines can contact the tracker
# You cannot put in localhost or 127.0.0.1
$ sudo ./registry --tracker "10.240.101.85:8940"

Usage

Push an image to the Registry

$ cd client
#Note: You will have to to explicitly mention the tag ":latest"
$ sudo ./client -h "http://10.240.101.85" push netvarun/test:latest
Found image:  netvarun/test:latest
ID:  353b94eb357ddb343ebe054ccc80b49bb6d0828522e9f2eff313406363449d17
RepoTags:  [netvarun/test:latest]
Created:  1422145581
Size:  0
VirtualSize:  188305056
ParentId:  d7d8be71d422a83c97849c4a8e124fcbe42170d5ce508f339ce52be9954dc3b4
Exporting image to tarball...
Successively exported tarball...
key =  image  val =  netvarun/test:latest
key =  id  val =  353b94eb357ddb343ebe054ccc80b49bb6d0828522e9f2eff313406363449d17
key =  created  val =  1422145581
Successfully uploaded image:  netvarun/test:latest  to the Docket registry.

List all images available in the Registry

$ cd client
$ sudo ./client -h "http://10.240.101.85" images
netvarun/test:latest
perl:latest

Do a deploy

$ cd client
$ sudo ./client -h "http://10.240.101.85" pull perl:latest
Downloading the torrent file for image:  perl:latest
Downloading from the torrent file...
META INFO
Announce: http://10.240.101.85:8940/announce
Piece length: 524288
Created with: docket-registry
FILES INFO
<1> 14f61693dd2db6380755a662d6e4e3583b5214fad9032bd983ce6c70df2144bc_perl_latest.tar [838467072]
Total: 799 MB
Found bit field file; verifying previous state.
Listening on 0.0.0.0:2706
Press 'h' or '?' for help (display/control client options).
Checking completed.
- 0/0/1 [0/1600/0] 0MB,0MB | 0,0K/s | 0,0K E:0,0 Connecting
End of input reached.
Input channel is now off
\ 1/0/2 [1572/1600/1600] 785MB,0MB | 36422,0K/s | 38896,0K E:0,1
Download complete.
Total time used: 0 minutes.
Seed for others 0 hours
| 0/0/2 [1600/1600/1600] 799MB,0MB | 0,0K/s | 14320,0K E:0,1 Connecting

Tarball path =  /tmp/docket/14f61693dd2db6380755a662d6e4e3583b5214fad9032bd983ce6c70df2144bc_perl_latest.tar
Exporting image to tarball...

Reference (Docket Registry)

./registry --help

usage: registry [<flags>]

Docket Registry

Flags:
  --help             Show help.
  -t, --tracker="10.240.101.85:8940"  
                     Set host and port of bittorrent tracker. Example: -host 10.240.101.85:8940 Note: This cannot be set to localhost, since this is the tracker in which all the torrents will be created with. They have to be some accessible ip
                     address from outside
  -p, --port="8000"  Set port of docket registry.
  -l, --location="/var/local/docket"  
                     Set location to save torrents and docker images.

Reference (Docket Client)

./client 

usage: client [<flags>] <command> [<flags>] [<args> ...]

Docket Client

Flags:
  --help             Show help.
  -h, --host="http://127.0.0.1"  
                     Set host of docket registry.
  -p, --port="8000"  Set port of docket registry.
  -l, --location="/tmp/docket"  
                     Set location to store torrents and tarballs.

Commands:
  help [<command>]
    Show help for a command.

  push <push>
    Push to the docket registry.

  pull <pull>
    pull to the docket registry.

  images [<flags>]
    display images in the docket registry.

Author

Sivamani Varun (varun@semantics3.com)

Download Details:

Author: netvarun
Source code: https://github.com/netvarun/docket

License: MIT license
#docker 

What is GEEK

Buddha Community

Custom Docker Registry That Allows For Deployments Through Bittorrent
Iliana Welch

Iliana Welch

1595249460

Docker Explained: Docker Architecture | Docker Registries

Following the second video about Docker basics, in this video, I explain Docker architecture and explain the different building blocks of the docker engine; docker client, API, Docker Daemon. I also explain what a docker registry is and I finish the video with a demo explaining and illustrating how to use Docker hub

In this video lesson you will learn:

  • What is Docker Host
  • What is Docker Engine
  • Learn about Docker Architecture
  • Learn about Docker client and Docker Daemon
  • Docker Hub and Registries
  • Simple demo to understand using images from registries

#docker #docker hub #docker host #docker engine #docker architecture #api

Alex Tyler

Alex Tyler

1600580050

How to Set up Docker Registry and a Docker RegUI

In this story, we will install Docker Registry on a VM and the cloud and we will add a Docker container to browser the registries from a web browser. From my side, I created an ECS instance on Alibaba Cloud. But we could do it everywhere. After creating the ECS instance, I got its public IP address (8.208.91.39) and I created a security group to authorize the following ports:

  • 5000 for the Docker Registry,
  • 8086 for the Docker Registry UI.

#docker #devops #docker-compose #docker-registry

Jammie Yost

Jammie Yost

1661526240

Custom Docker Registry That Allows For Deployments Through Bittorrent

Docket

DOCKEr + torrenT = Docket

Docket is a custom docker registry that allows for deployments through bittorrent. It allows for lightning fast Docker image deploys across large number of machines.

This was designed and built in 48 hours as part of the Gopher Gala Golang 48 hour hackathon. Hence kindly forgive me for the hackish code, and lack of tests.

[On hind sight - container rush or docker flood would have been a better name.]

Screencast

Problem Statement

As containers get more mainstream, server deployments are soon going to be done through a containerization solution like docker [or rocket or lxd].

Large scale deploys are going to choke your docker registry. Imagine pulling/deploying a 800mb base image (for example the official perl image) across 200 machines in one go. That's 800*200 = 160GB of data that's going to be deployed and it'll definitely choke your private docker registry (and will take a while pulling in from the public image)

Solution

The solution is BitTorrent. Any technology good for distributing music and movies, is usually good for doing server deploys. Instead of downloading from one point source, we can now distribute the load across the machines downloading themselves. All of this transparently and automatically handled by the docket client.

Silly Totally Non-scientific Benchmark

Pulling a 800 mb image across 5 machines simultaneously via Docket resulted in a 90% speedup.

Features

  • Written in Golang :)
  • Very easy to use
  • Works along side your private docker registry

Concepts

Docket constitutes of 3 components:

  1. Docket Registry

A REST service that acts as a registry. It receives docker image tarballs from the client, stores metadata into a database, creates torrents out of them and seeds them.

  1. Docket Client

The client itnerface in which the end user will be interacting. Can view available images in the registry, push an image to the registry and pull an image (which triggers a bittorrent deploy) from the registry.

  1. Bittorrent Tracker

Docket allows you to BYOT (bring your own tracker), but I suggest installing opentracker

Docket uses ctorrent for the actual downloading of torrents.

Installation

Step 0: You need to already have Docker installed

Step 1: Install ctorrent and opentracker

$ sudo apt-get update
$ sudo apt-get install zlib1g-dev make g++ ctorrent
$ git clone https://github.com/gebi/libowfat
$ make
$ git clone git://erdgeist.org/opentracker
$ cd opentracker
$ make

Step 2: Download the Docket binaries

$ wget http://storage.googleapis.com/docket/docket.zip
$ unzip docket.zip

Step 3: That's all

Getting Started

Step 1: Fire up the Tracker

$ cd opentracker
$ ./opentracker 8940

Step 2: Fire up the Registry

$ cd registry
# Note you need to put an ip address in which other machines can contact the tracker
# You cannot put in localhost or 127.0.0.1
$ sudo ./registry --tracker "10.240.101.85:8940"

Usage

Push an image to the Registry

$ cd client
#Note: You will have to to explicitly mention the tag ":latest"
$ sudo ./client -h "http://10.240.101.85" push netvarun/test:latest
Found image:  netvarun/test:latest
ID:  353b94eb357ddb343ebe054ccc80b49bb6d0828522e9f2eff313406363449d17
RepoTags:  [netvarun/test:latest]
Created:  1422145581
Size:  0
VirtualSize:  188305056
ParentId:  d7d8be71d422a83c97849c4a8e124fcbe42170d5ce508f339ce52be9954dc3b4
Exporting image to tarball...
Successively exported tarball...
key =  image  val =  netvarun/test:latest
key =  id  val =  353b94eb357ddb343ebe054ccc80b49bb6d0828522e9f2eff313406363449d17
key =  created  val =  1422145581
Successfully uploaded image:  netvarun/test:latest  to the Docket registry.

List all images available in the Registry

$ cd client
$ sudo ./client -h "http://10.240.101.85" images
netvarun/test:latest
perl:latest

Do a deploy

$ cd client
$ sudo ./client -h "http://10.240.101.85" pull perl:latest
Downloading the torrent file for image:  perl:latest
Downloading from the torrent file...
META INFO
Announce: http://10.240.101.85:8940/announce
Piece length: 524288
Created with: docket-registry
FILES INFO
<1> 14f61693dd2db6380755a662d6e4e3583b5214fad9032bd983ce6c70df2144bc_perl_latest.tar [838467072]
Total: 799 MB
Found bit field file; verifying previous state.
Listening on 0.0.0.0:2706
Press 'h' or '?' for help (display/control client options).
Checking completed.
- 0/0/1 [0/1600/0] 0MB,0MB | 0,0K/s | 0,0K E:0,0 Connecting
End of input reached.
Input channel is now off
\ 1/0/2 [1572/1600/1600] 785MB,0MB | 36422,0K/s | 38896,0K E:0,1
Download complete.
Total time used: 0 minutes.
Seed for others 0 hours
| 0/0/2 [1600/1600/1600] 799MB,0MB | 0,0K/s | 14320,0K E:0,1 Connecting

Tarball path =  /tmp/docket/14f61693dd2db6380755a662d6e4e3583b5214fad9032bd983ce6c70df2144bc_perl_latest.tar
Exporting image to tarball...

Reference (Docket Registry)

./registry --help

usage: registry [<flags>]

Docket Registry

Flags:
  --help             Show help.
  -t, --tracker="10.240.101.85:8940"  
                     Set host and port of bittorrent tracker. Example: -host 10.240.101.85:8940 Note: This cannot be set to localhost, since this is the tracker in which all the torrents will be created with. They have to be some accessible ip
                     address from outside
  -p, --port="8000"  Set port of docket registry.
  -l, --location="/var/local/docket"  
                     Set location to save torrents and docker images.

Reference (Docket Client)

./client 

usage: client [<flags>] <command> [<flags>] [<args> ...]

Docket Client

Flags:
  --help             Show help.
  -h, --host="http://127.0.0.1"  
                     Set host of docket registry.
  -p, --port="8000"  Set port of docket registry.
  -l, --location="/tmp/docket"  
                     Set location to store torrents and tarballs.

Commands:
  help [<command>]
    Show help for a command.

  push <push>
    Push to the docket registry.

  pull <pull>
    pull to the docket registry.

  images [<flags>]
    display images in the docket registry.

Author

Sivamani Varun (varun@semantics3.com)

Download Details:

Author: netvarun
Source code: https://github.com/netvarun/docket

License: MIT license
#docker 

Turner Crona

Turner Crona

1595743680

Full support for popular Docker Registry implementations in werf

Container registries tend to support the Docker Registry HTTP API, allowing their users to rely on the same tools to operate them. However, some implementations have their peculiarities and limitations. Thus, you have to take into account their specifics when using them as part of your CI/CD toolchain. That is exactly what happened when we decided to improve the way our werf GitOps utility manages the lifecycle of images.

In this article, we will discuss the main peculiarities of Docker Registry implementations supported by werf as well as resulting improvements in our tool.

Storing images

As you know, you need a registry to store and distribute Docker images. Strictly speaking, a registry is just a service to store various repositories (AWS ECR, Azure CR, Docker Hub, and so on). The repository stores images grouped by name.

When building applications and/or deploying them to Kubernetes with werf, you can use --images-repo and --images-repo-mode parameters. They allow you to specify where and how (in a single or multiple repositories*) application images will be stored in the registry.

* You can learn more about these parameters (including the problem definition and history of development) in our “Monorepo/multirepo support in werf (and what does it have to do with Docker Registry?)” article.

The --images-repo parameter can be either a registry address or a repository address. In essence, its value serves as the basis for composing the name of an image and not necessarily specifies the repository where the images will be stored (take a look at templates below to clarify this point).

The --images-repo-mode parameter supports two values that define the template for composing the final image name:

  • IMAGES_REPO:IMAGE_NAME-TAG is a template for the monorepo mode;
  • IMAGES_REPO/IMAGE_NAME:TAG is a template for the multirepo mode.

#container-registry #werf #docker #docker-registry

Fynzo Survey

Fynzo Survey

1622049211

Fynzo Customer Feedback Software For Cafes, Hotels, Saloons, Spa!

Customer Feedback Tool | Fynzo online customer feedback comes with Android, iOS app. Collect feedback from your customers with tablets or send them feedback links.

Visit page for more information: https://www.fynzo.com/feedback

#CustomerFeedbackSystem
#PowerfulCustomerFeedbackSystem
#freecustomerfeedbacktools
#automatedcustomerfeedbacksystem
#customerfeedbacktools
#customerratingsystem
#Customerfeedbackmanagement

#customer feedback system #powerful customer feedback system #free customer feedback tools #automated customer feedback system #customer feedback tools #customer rating system