Libcompose: Go library for Docker Compose

⚠️ Deprecation Notice: This project and repository is now deprecated and is no longer under active development. Please use compose-go instead.

libcompose

A Go library for Docker Compose. It does everything the command-line tool does, but from within Go -- read Compose files, start them, scale them, etc.

Note: This is not really maintained anymore — the reason are diverse but mainly lack of time from the maintainers

The current state is the following :

The libcompose CLI should considered abandonned. The v2 parsing is incomplete and v3 parsing is missing.
The official compose Go parser implementation is on docker/cli but only support v3 version of the compose format.

What is the work that is needed:

Remove the cli code (thus removing dependencies to docker/cli )
Clearer separation of packages : parsing, conversion (to docker api or swarm api), execution (Up, Down, … behaviors)
Add support for all compose format version (v1, v2.x, v3.x)
Switch to either golang/dep or go mod for dependencies (removing the vendor folder)
(bonus) extract the docker/cli code here and vendor this library into docker/cli.

If you are interested to work on libcompose, feel free to ping me (over twitter @vdemeest), I'll definitely do code reviews and help as much as I can 😉.

Note: This is experimental and not intended to replace the Docker Compose command-line tool. If you're looking to use Compose, head over to the Compose installation instructions to get started with it.

Here is a list of known project that uses libcompose:

rancher-compose and rancher os (by Rancher)
openshift (by Red Hat)
henge (by Red Hat) [Deprecated in favour of kompose]
kompose (by skippbox)
compose2kube (by kelseyhightower)
amazon-ecs-cli (by Amazon AWS)
libkermit (by vdemeester)

Usage

package main

import (
    "log"

    "golang.org/x/net/context"

    "github.com/docker/libcompose/docker"
    "github.com/docker/libcompose/docker/ctx"
    "github.com/docker/libcompose/project"
    "github.com/docker/libcompose/project/options"
)

func main() {
    project, err := docker.NewProject(&ctx.Context{
        Context: project.Context{
            ComposeFiles: []string{"docker-compose.yml"},
            ProjectName:  "my-compose",
        },
    }, nil)

    if err != nil {
        log.Fatal(err)
    }

    err = project.Up(context.Background(), options.Up{})

    if err != nil {
        log.Fatal(err)
    }
}

Building

You need either Docker and make, or go in order to build libcompose.

Building with `docker`

You need Docker and make and then run the binary target. This will create binary for all platform in the bundles folder.

$ make binary
docker build -t "libcompose-dev:refactor-makefile" .
# […]
---> Making bundle: binary (in .)
Number of parallel builds: 4

-->      darwin/386: github.com/docker/libcompose/cli/main
-->    darwin/amd64: github.com/docker/libcompose/cli/main
-->       linux/386: github.com/docker/libcompose/cli/main
-->     linux/amd64: github.com/docker/libcompose/cli/main
-->       linux/arm: github.com/docker/libcompose/cli/main
-->     windows/386: github.com/docker/libcompose/cli/main
-->   windows/amd64: github.com/docker/libcompose/cli/main

$ ls bundles
libcompose-cli_darwin-386*    libcompose-cli_linux-amd64*      libcompose-cli_windows-amd64.exe*
libcompose-cli_darwin-amd64*  libcompose-cli_linux-arm*
libcompose-cli_linux-386*     libcompose-cli_windows-386.exe*

Building with `go`

You need go v1.11 or greater
you need to set export GO111MODULE=on environment variable
If your working copy is not in your GOPATH, you need to set it accordingly.

$ go generate
# Generate some stuff
$ go build -o libcompose ./cli/main

Running

A partial implementation of the libcompose-cli CLI is also implemented in Go. The primary purpose of this code is so one can easily test the behavior of libcompose.

Run one of these:

libcompose-cli_darwin-386
libcompose-cli_linux-amd64
libcompose-cli_windows-amd64.exe
libcompose-cli_darwin-amd64
libcompose-cli_linux-arm
libcompose-cli_linux-386
libcompose-cli_windows-386.exe

Tests (unit & integration)

You can run unit tests using the test-unit target and the integration test using the test-integration target. If you don't use Docker and make to build libcompose, you can use go test and the following scripts : hack/test-unit and hack/test-integration.

$ make test-unit
docker build -t "libcompose-dev:refactor-makefile" .
#[…]
---> Making bundle: test-unit (in .)
+ go test -cover -coverprofile=cover.out ./docker
ok      github.com/docker/libcompose/docker     0.019s  coverage: 4.6% of statements
+ go test -cover -coverprofile=cover.out ./project
ok      github.com/docker/libcompose/project    0.010s  coverage: 8.4% of statements
+ go test -cover -coverprofile=cover.out ./version
ok      github.com/docker/libcompose/version    0.002s  coverage: 0.0% of statements

Test success

Current status

The project is still being kickstarted... But it does a lot. Please try it out and help us find bugs.

Contributing

Want to hack on libcompose? Docker's contributions guidelines apply.

If you have comments, questions, or want to use your knowledge to help other, come join the conversation on IRC. You can reach us at #libcompose on Freenode.

Download Details:

Author: docker
Source code: https://github.com/docker/libcompose

License: Apache-2.0 license
#docker #go #golang

What is GEEK

Buddha Community

Libcompose: Go library for Docker Compose

Jammie Yost

1661671140

Libcompose: Go library for Docker Compose

⚠️ Deprecation Notice: This project and repository is now deprecated and is no longer under active development. Please use compose-go instead.

libcompose

A Go library for Docker Compose. It does everything the command-line tool does, but from within Go -- read Compose files, start them, scale them, etc.

Note: This is not really maintained anymore — the reason are diverse but mainly lack of time from the maintainers

The current state is the following :

The libcompose CLI should considered abandonned. The v2 parsing is incomplete and v3 parsing is missing.
The official compose Go parser implementation is on docker/cli but only support v3 version of the compose format.

What is the work that is needed:

Remove the cli code (thus removing dependencies to docker/cli )
Clearer separation of packages : parsing, conversion (to docker api or swarm api), execution (Up, Down, … behaviors)
Add support for all compose format version (v1, v2.x, v3.x)
Switch to either golang/dep or go mod for dependencies (removing the vendor folder)
(bonus) extract the docker/cli code here and vendor this library into docker/cli.

If you are interested to work on libcompose, feel free to ping me (over twitter @vdemeest), I'll definitely do code reviews and help as much as I can 😉.

Here is a list of known project that uses libcompose:

rancher-compose and rancher os (by Rancher)
openshift (by Red Hat)
henge (by Red Hat) [Deprecated in favour of kompose]
kompose (by skippbox)
compose2kube (by kelseyhightower)
amazon-ecs-cli (by Amazon AWS)
libkermit (by vdemeester)

Usage

package main

import (
    "log"

    "golang.org/x/net/context"

    "github.com/docker/libcompose/docker"
    "github.com/docker/libcompose/docker/ctx"
    "github.com/docker/libcompose/project"
    "github.com/docker/libcompose/project/options"
)

func main() {
    project, err := docker.NewProject(&ctx.Context{
        Context: project.Context{
            ComposeFiles: []string{"docker-compose.yml"},
            ProjectName:  "my-compose",
        },
    }, nil)

    if err != nil {
        log.Fatal(err)
    }

    err = project.Up(context.Background(), options.Up{})

    if err != nil {
        log.Fatal(err)
    }
}

Building

You need either Docker and make, or go in order to build libcompose.

Building with `docker`

You need Docker and make and then run the binary target. This will create binary for all platform in the bundles folder.

$ make binary
docker build -t "libcompose-dev:refactor-makefile" .
# […]
---> Making bundle: binary (in .)
Number of parallel builds: 4

-->      darwin/386: github.com/docker/libcompose/cli/main
-->    darwin/amd64: github.com/docker/libcompose/cli/main
-->       linux/386: github.com/docker/libcompose/cli/main
-->     linux/amd64: github.com/docker/libcompose/cli/main
-->       linux/arm: github.com/docker/libcompose/cli/main
-->     windows/386: github.com/docker/libcompose/cli/main
-->   windows/amd64: github.com/docker/libcompose/cli/main

$ ls bundles
libcompose-cli_darwin-386*    libcompose-cli_linux-amd64*      libcompose-cli_windows-amd64.exe*
libcompose-cli_darwin-amd64*  libcompose-cli_linux-arm*
libcompose-cli_linux-386*     libcompose-cli_windows-386.exe*

Building with `go`

You need go v1.11 or greater
you need to set export GO111MODULE=on environment variable
If your working copy is not in your GOPATH, you need to set it accordingly.

$ go generate
# Generate some stuff
$ go build -o libcompose ./cli/main

Running

A partial implementation of the libcompose-cli CLI is also implemented in Go. The primary purpose of this code is so one can easily test the behavior of libcompose.

Run one of these:

libcompose-cli_darwin-386
libcompose-cli_linux-amd64
libcompose-cli_windows-amd64.exe
libcompose-cli_darwin-amd64
libcompose-cli_linux-arm
libcompose-cli_linux-386
libcompose-cli_windows-386.exe

Tests (unit & integration)

$ make test-unit
docker build -t "libcompose-dev:refactor-makefile" .
#[…]
---> Making bundle: test-unit (in .)
+ go test -cover -coverprofile=cover.out ./docker
ok      github.com/docker/libcompose/docker     0.019s  coverage: 4.6% of statements
+ go test -cover -coverprofile=cover.out ./project
ok      github.com/docker/libcompose/project    0.010s  coverage: 8.4% of statements
+ go test -cover -coverprofile=cover.out ./version
ok      github.com/docker/libcompose/version    0.002s  coverage: 0.0% of statements

Test success

Current status

The project is still being kickstarted... But it does a lot. Please try it out and help us find bugs.

Contributing

Want to hack on libcompose? Docker's contributions guidelines apply.

If you have comments, questions, or want to use your knowledge to help other, come join the conversation on IRC. You can reach us at #libcompose on Freenode.

Download Details:

Author: docker
Source code: https://github.com/docker/libcompose

License: Apache-2.0 license
#docker #go #golang

Iliana Welch

1595249460

Docker Explained: Docker Architecture | Docker Registries

Following the second video about Docker basics, in this video, I explain Docker architecture and explain the different building blocks of the docker engine; docker client, API, Docker Daemon. I also explain what a docker registry is and I finish the video with a demo explaining and illustrating how to use Docker hub

In this video lesson you will learn:

What is Docker Host
What is Docker Engine
Learn about Docker Architecture
Learn about Docker client and Docker Daemon
Docker Hub and Registries
Simple demo to understand using images from registries

#docker #docker hub #docker host #docker engine #docker architecture #api

Fannie Zemlak

1599854400

What's new in the go 1.15

Go announced Go 1.15 version on 11 Aug 2020. Highlighted updates and features include Substantial improvements to the Go linker, Improved allocation for small objects at high core counts, X.509 CommonName deprecation, GOPROXY supports skipping proxies that return errors, New embedded tzdata package, Several Core Library improvements and more.

As Go promise for maintaining backward compatibility. After upgrading to the latest Go 1.15 version, almost all existing Golang applications or programs continue to compile and run as older Golang version.

#go #golang #go 1.15 #go features #go improvement #go package #go new features

Cayla Erdman

1599914520

Apache/Airflow and PostgreSQL with Docker and Docker Compose

Hello, in this post I will show you how to set up official Apache/Airflow with PostgreSQL and LocalExecutor using docker and docker-compose. In this post, I won’t be going through Airflow, what it is, and how it is used. Please checktheofficial documentation for more information about that.

Before setting up and running Apache Airflow, please install Docker and Docker Compose.

For those in hurry…

In this chapter, I will show you files and directories which are needed to run airflow and in the next chapter, I will go file by file, line by line explaining what is going on.

Firstly, in the root directory create three more directories: dags, logs, and scripts. Further, create following files: **.env, docker-compose.yml, entrypoint.sh **and **dummy_dag.py. **Please make sure those files and directories follow the structure below.

#project structure

root/
├── dags/
│   └── dummy_dag.py
├── scripts/
│   └── entrypoint.sh
├── logs/
├── .env
└── docker-compose.yml

Created files should contain the following:

#docker-compose.yml

version: '3.8'
services:
    postgres:
        image: postgres
        environment:
            - POSTGRES_USER=airflow
            - POSTGRES_PASSWORD=airflow
            - POSTGRES_DB=airflow
    scheduler:
        image: apache/airflow
        command: scheduler
        restart_policy:
            condition: on-failure
        depends_on:
            - postgres
        env_file:
            - .env
        volumes:
            - ./dags:/opt/airflow/dags
            - ./logs:/opt/airflow/logs
    webserver:
        image: apache/airflow
        entrypoint: ./scripts/entrypoint.sh
        restart_policy:
            condition: on-failure
        depends_on:
            - postgres
            - scheduler
        env_file:
            - .env
        volumes:
            - ./dags:/opt/airflow/dags
            - ./logs:/opt/airflow/logs
            - ./scripts:/opt/airflow/scripts
        ports:
            - "8080:8080"

#entrypoint.sh
#!/usr/bin/env bash
airflow initdb
airflow webserver

#.env
AIRFLOW__CORE__SQL_ALCHEMY_CONN=postgresql+psycopg2://airflow:airflow@postgres/airflow
AIRFLOW__CORE__EXECUTOR=LocalExecutor

#dummy_dag.py
from airflow import DAG
from airflow.operators.dummy_operator import DummyOperator
from datetime import datetime
with DAG('example_dag', start_date=datetime(2016, 1, 1)) as dag:
    op = DummyOperator(task_id='op')

Positioning in the root directory and executing “docker-compose up” in the terminal should make airflow accessible on localhost:8080. Image bellow shows the final result.

If you encounter permission errors, please run “chmod -R 777” on all subdirectories, e.g. “chmod -R 777 logs/”

For the curious ones...

In Leyman’s terms, docker is used when managing individual containers and docker-compose can be used to manage multi-container applications. It also moves many of the options you would enter on the docker run into the docker-compose.yml file for easier reuse. It works as a front end "script" on top of the same docker API used by docker. You can do everything docker-compose does with docker commands and a lot of shell scripting.

Before running our multi-container docker applications, docker-compose.yml must be configured. With that file, we define services that will be run on docker-compose up.

The first attribute of docker-compose.yml is version, which is the compose file format version. For the most recent version of file format and all configuration options click here.

Second attribute is services and all attributes one level bellow services denote containers used in our multi-container application. These are postgres, scheduler and webserver. Each container has image attribute which points to base image used for that service.

For each service, we define environment variables used inside service containers. For postgres it is defined by environment attribute, but for scheduler and webserver it is defined by .env file. Because .env is an external file we must point to it with env_file attribute.

By opening .env file we can see two variables defined. One defines executor which will be used and the other denotes connection string. Each connection string must be defined in the following manner:

dialect+driver://username:password@host:port/database

Dialect names include the identifying name of the SQLAlchemy dialect, a name such as sqlite, mysql, postgresql, oracle, or mssql. Driver is the name of the DBAPI to be used to connect to the database using all lowercase letters. In our case, connection string is defined by:

postgresql+psycopg2://airflow:airflow@postgres/airflow

Omitting port after host part denotes that we will be using default postgres port defined in its own Dockerfile.

Every service can define command which will be run inside Docker container. If one service needs to execute multiple commands it can be done by defining an optional .sh file and pointing to it with entrypoint attribute. In our case we have entrypoint.sh inside the scripts folder which once executed, runs airflow initdb and airflow webserver. Both are mandatory for airflow to run properly.

Defining depends_on attribute, we can express dependency between services. In our example, webserver starts only if both scheduler and postgres have started, also the scheduler only starts after postgres have started.

In case our container crashes, we can restart it by restart_policy. The restart_policy configures if and how to restart containers when they exit. Additional options are condition, delay, max_attempts, and window.

Once service is running, it is being served on containers defined port. To access that service we need to expose the containers port to the host's port. That is being done by ports attribute. In our case, we are exposing port 8080 of the container to TCP port 8080 on 127.0.0.1 (localhost) of the host machine. Left side of : defines host machines port and the right-hand side defines containers port.

Lastly, the volumes attribute defines shared volumes (directories) between host file system and docker container. Because airflows default working directory is /opt/airflow/ we need to point our designated volumes from the root folder to the airflow containers working directory. Such is done by the following command:

#general case for airflow
- ./<our-root-subdir>:/opt/airflow/<our-root-subdir>
#our case
- ./dags:/opt/airflow/dags
- ./logs:/opt/airflow/logs
- ./scripts:/opt/airflow/scripts
           ...

This way, when the scheduler or webserver writes logs to its logs directory we can access it from our file system within the logs directory. When we add a new dag to the dags folder it will automatically be added in the containers dag bag and so on.

Originally published by Ivan Rezic at Towardsdatascience

#docker #how-to #apache-airflow #docker-compose #postgresql

August Murray

1615008840

Top 24 Docker Commands Explained with Examples

In my previous blog post, I have explained in detail how you can Install Docker and Docker-compose on Ubuntu

In this guide, I have explained the Top 24 Docker Commands with examples.

Make sure you have sudo or root privileges to the system.

Docker Commands

The command to check the version of Docker installed.
To look/search for available docker images from the Docker registry.
To pull docker images from the Docker registry.
Listing all the docker images
Creating / Running docker container from Docker image.
To list the actively running docker containers.
To list all the docker containers
To stop a Container
To start a Container
To restart a Docker container
To login to running Docker container
To delete the stopped Docker containers
To delete Docker images from the Local system
To check logs of a running Docker container
Killing docker containers
Log in to Docker Hub registry (hub.docker.com)
Removing docker hub registry login from the system.
Check active resource usage by each containers
Rename a Docker container
To display system wide information of Docker
Inspecting a Docker container
Building docker images from Docker file
Creating new docker images from a Container
Pushing Docker images from Local to Docker registry.

#docker #docker-command #containers #docker-compose #docker-image

Libcompose: Go library for Docker Compose

⚠️ Deprecation Notice: This project and repository is now deprecated and is no longer under active development. Please use compose-go instead.

Usage

Building

Building with docker

Building with go

Running

Tests (unit & integration)

Current status

Contributing

Download Details:

What is GEEK

Buddha Community

Libcompose: Go library for Docker Compose

⚠️ Deprecation Notice: This project and repository is now deprecated and is no longer under active development. Please use compose-go instead.

Usage

Building

Building with docker

Building with go

Running

Tests (unit & integration)

Current status

Contributing

Download Details:

Docker Explained: Docker Architecture | Docker Registries

What's new in the go 1.15

Apache/Airflow and PostgreSQL with Docker and Docker Compose

For those in hurry…

Top 24 Docker Commands Explained with Examples

Docker Commands

Building with `docker`

Building with `go`

Building with `docker`

Building with `go`