Elian  Harber

Elian Harber

1665512040

Fabio: Consul Load-Balancing Made Simple

Fabio

Consul Load-Balancing made simple


Notes

From release 1.6.1 onward, the minimum golang version supported is 1.16.

From release 1.6.0 onward, metrics backend statsd is no longer supported. statsd_raw works similarly, though it actually resets counters appropriately. If you are using datadog, you should consider using the new dogstatsd backend, which has support for tags now. Graphite histogram functionality has changed slightly since switching to gokit framework, so something to be aware of. Prometheus functionality is now supported natively.

From release 1.5.15 onward, fabio changes the default GOGC from 800 back to the golang default of 100. Apparently this made some sense back in the golang 1.5 days, but with changes introduced with golang 1.12 and others, this is probably no longer a very good default. This is still configurable, as always, but the new default should make the most sense for most users.

From release 1.5.14, release hashes are signed with a new PGP key. See details here.

From release 1.5.14 onward, fabio binary releases are compiled with golang 1.15+.
This means that the fabio will no longer validate upstream https certificates that do not have SAN extensions matching the server name. This may be a concern if fabio is communicating with https backends with misconfigured certificates. If this is a problem, you can specify tlsskipverify=true on the route.


fabio is a fast, modern, zero-conf load balancing HTTP(S) and TCP router for deploying applications managed by consul.

Register your services in consul, provide a health check and fabio will start routing traffic to them. No configuration required. Deployment, upgrading and refactoring has never been easier.

fabio is developed and maintained by The Fabio Authors.

It powers some of the largest websites in Australia (gumtree.com.au) and Italy (www.kijiji.it). It delivers 23.000 req/sec every day since Sep 2015 without problems.

It integrates with Consul, Vault, Amazon ELB, Amazon API Gateway and more.

It supports (Full feature list)

Watch Kelsey Hightower demo Consul, Nomad, Vault and fabio at HashiConf EU 2016.

The full documentation is on fabiolb.net

Getting started

Install from source, binary, Docker or Homebrew.

# go 1.15 or higher is required
go install github.com/fabiolb/fabio@latest          (>= go1.15)

brew install fabio                                  (OSX/macOS stable)
brew install --devel fabio                          (OSX/macOS devel)

docker pull fabiolb/fabio                           (Docker)

https://github.com/fabiolb/fabio/releases           (pre-built binaries)

Register your service in consul.

Make sure that each instance registers with a unique ServiceID and a service name without spaces.

Register a health check in consul as described here.

By default fabio only watches services which have a passing health check, unless overridden with registry.consul.service.status.

Register one urlprefix- tag per host/path prefix it serves, e.g.:

# HTTP/S examples
urlprefix-/css                                     # path route
urlprefix-i.com/static                             # host specific path route
urlprefix-mysite.com/                              # host specific catch all route
urlprefix-/foo/bar strip=/foo                      # path stripping (forward '/bar' to upstream)
urlprefix-/foo/bar proto=https                     # HTTPS upstream
urlprefix-/foo/bar proto=https tlsskipverify=true  # HTTPS upstream and self-signed cert

# TCP examples
urlprefix-:3306 proto=tcp                          # route external port 3306

Make sure the prefix for HTTP routes contains at least one slash (/).

See the full list of options in the Documentation.

Start fabio without a config file (assuming a running consul agent on localhost:8500) Watch the log output how fabio picks up the route to your service. Try starting/stopping your service to see how the routing table changes instantly.

Send all your HTTP traffic to fabio on port 9999. For TCP proxying see TCP proxy.

Done

Author and Founder

Maintainers

Contributors

This project exists thanks to all the people who contribute. [Contribute].

Download Details:

Author: Fabiolb
Source Code: https://github.com/fabiolb/fabio 
License: MIT license

#go #golang #docker #consul 

Fabio: Consul Load-Balancing Made Simple

How to Set Up and Use Hashicorp's Vault and Consul Projects

The following tutorial details how to set up and use Hashicorp's Vault and Consul projects to securely store and manage secrets.

Source: https://testdriven.io

#vault #consul 

How to Set Up and Use Hashicorp's Vault and Consul Projects
Thai  Son

Thai Son

1660640700

Cách Thiết Lập Và Sử Dụng Các Dự Án Vault Và Consul Của Hashicorp

Hướng dẫn sau đây trình bày chi tiết cách thiết lập và sử dụng các dự án Vault vàConsul của Hashicorp để lưu trữ và quản lý bí mật một cách an toàn.

Chúng tôi sẽ bắt đầu bằng cách xoay một phiên bản Vault duy nhất trong vùng chứa Docker và sau đó chuyển sang quản lý cả bí mật tĩnh và động cùng với tính năng "mã hóa dưới dạng dịch vụ" của Vault. Sau đó, chúng tôi sẽ thêm Consul vào hỗn hợp và xem xét cách mở rộng quy mô Vault.

Phụ thuộc chính:

  • Docker v20.10.8
  • Docker-Compose v1.29.2
  • Vault v1.8.2
  • Consul v1.10.2

Mục tiêu

Đến cuối hướng dẫn này, bạn sẽ có thể:

  1. Giải thích Vault là gì và tại sao bạn có thể muốn sử dụng nó
  2. Mô tả kiến ​​trúc Vault cơ bản cùng với các bí mật động và tĩnh, các phụ trợ khác nhau (lưu trữ, bí mật, xác thực, kiểm tra) và cách Vault có thể được sử dụng như một "mã hóa như một dịch vụ"
  3. Định cấu hình và chạy Vault and Consul với Docker
  4. Tăng tốc Vault với chương trình phụ trợ Hệ thống tập tin
  5. Init và unseal Vault
  6. Xác thực chống lại Vault
  7. Định cấu hình chương trình phụ trợ Kiểm tra để ghi lại tất cả các tương tác với Vault
  8. Làm việc với các bí mật tĩnh và động thông qua CLI, API HTTP và giao diện người dùng
  9. Tạo chính sách Vault để giới hạn quyền truy cập vào một đường dẫn cụ thể
  10. Sử dụng phần phụ trợ Chuyển tuyến làm "mã hóa dưới dạng dịch vụ"
  11. Thiết lập Consul để làm việc với Vault dưới dạng phụ trợ Storage để biết bí mật
  12. Xác định thời hạn thuê tùy chỉnh cho một bí mật và thu hồi một bí mật trước khi kết thúc thời hạn đó

Vault là gì?

Vault là một công cụ mã nguồn mở được sử dụng để lưu trữ và quản lý các bí mật một cách an toàn.

What is a secret? Secrets, in the context of this tutorial, are securely-sensitive or personally identifiable info like database credentials, SSH keys, usernames and passwords, AWS IAM credentials, API tokens, Social Security Numbers, credit card numbers, just to name a few.

Take a moment to think about how your team currently manages and distributes secrets:

  1. Who has access to them?
  2. Who manages them?
  3. How do you control who has access to them?
  4. How do your apps get them?
  5. How are they updated?
  6. How are they revoked?

Vault provides answers to those questions and helps to solve the following problems with regard to secret management:

ProblemsVault's Goals
Secrets are everywhere.Vault is the single source of truth for all secrets.
They are generally unencrypted.Vault manages encryption (during transit and at rest) out of the box.
It's difficult to dynamically generate them.Secrets can be dynamically generated.
It's even more difficult to lease and revoke them.Secrets can be leased and revoked.
There's no audit trail.There's an audit trail for generating and using secrets.

Vault has a number of moving pieces so it can take some time to get up to speed with the overall architecture. Take a moment to review the Architecture guide, taking note of the following backends:

BackendUseExamples
StorageWhere secrets are storedConsul*, Filesystem*, In-Memory, PostgreSQL, S3
SecretHandles static or dynamic secretsAWS*, Databases, Key/Value*, RabbitMQ, SSH
AuthHandles authentication and authorizationAWS, Azure, Google Cloud, GitHub, Tokens*, Username & Password
AuditLogs all requests and responsesFile*, Syslog, Socket

* used in this tutorial

With that, let's start using Vault.

Filesystem Backend

To get up and running quickly, we'll use the Filesystem backend to store secrets at rest.

The filesystem backend should only be used for local development or a single-server Vault deployment since it does not support high availability.

Create a new project directory:

$ mkdir vault-consul-docker && cd vault-consul-docker

Then add the following folders:

└── vault
    ├── config
    ├── data
    ├── logs
    └── policies

Add a Dockerfile to the "vault" directory:

# base image
FROM alpine:3.14

# set vault version
ENV VAULT_VERSION 1.8.2

# create a new directory
RUN mkdir /vault

# download dependencies
RUN apk --no-cache add \
      bash \
      ca-certificates \
      wget

# download and set up vault
RUN wget --quiet --output-document=/tmp/vault.zip https://releases.hashicorp.com/vault/${VAULT_VERSION}/vault_${VAULT_VERSION}_linux_amd64.zip && \
    unzip /tmp/vault.zip -d /vault && \
    rm -f /tmp/vault.zip && \
    chmod +x /vault

# update PATH
ENV PATH="PATH=$PATH:$PWD/vault"

# add the config file
COPY ./config/vault-config.json /vault/config/vault-config.json

# expose port 8200
EXPOSE 8200

# run vault
ENTRYPOINT ["vault"]

Next, add a docker-compose.yml file to the project root:

version: '3.8'

services:

  vault:
    build:
      context: ./vault
      dockerfile: Dockerfile
    ports:
      - 8200:8200
    volumes:
      - ./vault/config:/vault/config
      - ./vault/policies:/vault/policies
      - ./vault/data:/vault/data
      - ./vault/logs:/vault/logs
    environment:
      - VAULT_ADDR=http://127.0.0.1:8200
      - VAULT_API_ADDR=http://127.0.0.1:8200
    command: server -config=/vault/config/vault-config.json
    cap_add:
      - IPC_LOCK

Add a config file called vault-config.json to "vault/config":

{
  "backend": {
    "file": {
      "path": "vault/data"
    }
  },
  "listener": {
    "tcp":{
      "address": "0.0.0.0:8200",
      "tls_disable": 1
    }
  },
  "ui": true
}

Here, we configured Vault to use the Filesystem backend, defined the listener for Vault, disabled TLS, and enabled the Vault UI. Review the docs for more info on configuring Vault.

Now we can build the image and spin up the container:

$ docker-compose up -d --build

Pull up the Docker logs to make sure there were no errors in the build:

$ docker-compose logs

You should see something similar to:

Attaching to vault-consul-docker_vault_1
vault_1  | ==> Vault server configuration:
vault_1  |
vault_1  |              Api Address: http://127.0.0.1:8200
vault_1  | 2021-09-08T14:48:35.014Z [INFO]  proxy environment: http_proxy="" https_proxy="" no_proxy=""
vault_1  |                      Cgo: disabled
vault_1  |          Cluster Address: https://127.0.0.1:8201
vault_1  |               Go Version: go1.16.7
vault_1  |               Listener 1: tcp (addr: "0.0.0.0:8200", cluster address: "0.0.0.0:8201", max_request_duration: "1m30s", max_request_size: "33554432", tls: "disabled")
vault_1  |                Log Level: info
vault_1  |                    Mlock: supported: true, enabled: true
vault_1  |            Recovery Mode: false
vault_1  |                  Storage: file
vault_1  |                  Version: Vault v1.8.2
vault_1  |              Version Sha: aca76f63357041a43b49f3e8c11d67358496959f
vault_1  |
vault_1  | ==> Vault server started! Log data will stream in below:
vault_1  |

Initializing and Unsealing

Start a bash session within the running container:

$ docker-compose exec vault bash

Within the shell, initialize Vault:

bash-5.1# vault operator init

Take note of the unseal keys and the initial root token. You will need to provide three of the unseal keys every time the Vault server is resealed or restarted.

Why 3 keys? Review Shamir's Secret Sharing.

Now you can unseal Vault using three of the keys:

bash-5.1# vault operator unseal
Unseal Key (will be hidden):

Run this command two more times, using different keys each time. Once done, make sure Sealed is false:

Key             Value
---             -----
Seal Type       shamir
Initialized     true
Sealed          false
Total Shares    5
Threshold       3
Version         1.8.2
Storage Type    file
Cluster Name    vault-cluster-8fcf9d05
Cluster ID      d86e0274-ad9c-d2c1-d6ec-baeab410797b
HA Enabled      false

Using the root token, you can now authenticate:

bash-5.1# vault login
Token (will be hidden):

You should see something similar to:

Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.

Key                  Value
---                  -----
token                s.c0kYHWiOTqQvtR8JuSeTz6sZ
token_accessor       3FQJVxOY5C1brzlHHQSFaCdZ
token_duration       ∞
token_renewable      false
token_policies       ["root"]
identity_policies    []
policies             ["root"]

Keep in mind that this uses the root policy. In production you'll want to set up policies with different levels of access. We'll look at how to do this shortly.

vault init

Vault is now unsealed and ready for use.

Auditing

Before we test out the functionality, let's enable an Audit Device:

bash-5.1# vault audit enable file file_path=/vault/logs/audit.log

Success! Enabled the file audit device at: file/

You should now be able to view the logs locally in "vault/logs". To test, run the following command to view all enabled Audit Devices:

bash-5.1# vault audit list

Path     Type    Description
----     ----    -----------
file/    file    n/a

The request and subsequent response should be logged in vault/logs/audit.log. Take a look.

Secrets

There are two types of secrets in Vault: static and dynamic.

Static secrets (think encrypted Redis or Memcached) have refresh intervals but they do not expire unless explicitly revoked. They are defined ahead of time with the Key/Value backend (formerly the "generic" backend) and then shared.

lưu trữ bí mật an toàn

Dynamic secrets are generated on demand. They have enforced leases and generally expire after a short period of time. Since they do not exist until they are accessed, there's less exposure -- so dynamic secrets are much more secure. Vault ships with a number of dynamic backends -- i.e., AWS, Databases, Google Cloud, Consul, and RabbitMQ.

Review the Why We Need Dynamic Secrets blog post for more info on the advantages of using dynamic secrets.

Static Secrets

Vault can be managed through the CLI, HTTP API, or UI.

CLI

Still within the bash session in the container, we can create, read, update, and delete secrets. We'll also look at how to version and roll back secrets.

Enable secrets with following command:

bash-5.1# vault secrets enable kv

Success! Enabled the kv secrets engine at: kv/

Create a new secret with a key of bar and value of precious within the kv/foo path:

bash-5.1# vault kv put kv/foo bar=precious

Success! Data written to: kv/foo

Read:

bash-5.1# vault kv get kv/foo

=== Data ===
Key    Value
---    -----
bar    precious

To work with different versions of a specific key, we'll need to upgrade to v2 of the Key/Value backend:

bash-5.1# vault kv enable-versioning kv/

Success! Tuned the secrets engine at: kv/

Add version 2 by updating the value to copper:

bash-5.1# vault kv put kv/foo bar=copper

Key              Value
---              -----
created_time     2021-09-08T18:23:14.4154928Z
deletion_time    n/a
destroyed        false
version          2

Read version 1:

bash-5.1# vault kv get -version=1 kv/foo

====== Metadata ======
Key              Value
---              -----
created_time     2021-09-08T18:22:37.2548824Z
deletion_time    n/a
destroyed        false
version          1

=== Data ===
Key    Value
---    -----
bar    precious

Read version 2:

bash-5.1# vault kv get -version=2 kv/foo

====== Metadata ======
Key              Value
---              -----
created_time     2021-09-08T18:23:14.4154928Z
deletion_time    n/a
destroyed        false
version          2

=== Data ===
Key    Value
---    -----
bar    copper

Delete the latest version (e.g., version 2):

bash-5.1# vault kv delete kv/foo

Success! Data deleted (if it existed) at: kv/foo

Delete version 1:

bash-5.1# vault kv delete -versions=1 kv/foo

Success! Data deleted (if it existed) at: kv/foo

You can undelete as well:

bash-5.1# vault kv undelete -versions=1 kv/foo

Success! Data written to: kv/undelete/foo

Delete is akin to a soft delete. If you want to remove the underlying metadata, you'll have to use the destroy command:

bash-5.1# vault kv destroy -versions=1 kv/foo

Success! Data written to: kv/destroy/foo

Review v1 and v2 to view all the available commands.

Take note of the audit log. Each of the above requests were logged!

API

You can also interact with Vault via the HTTP API. We'll make requests against v2 of the API. Open a new terminal tab, and then set the root token as an environment variable:

$ export VAULT_TOKEN=your_token_goes_here

Create a new secret called foo with a value of world:

$ curl \
    -H "X-Vault-Token: $VAULT_TOKEN" \
    -H "Content-Type: application/json" \
    -X POST \
    -d '{ "data": { "foo": "world" } }' \
    http://127.0.0.1:8200/v1/kv/data/hello

Read the secret:

$ curl \
    -H "X-Vault-Token: $VAULT_TOKEN" \
    -X GET \
    http://127.0.0.1:8200/v1/kv/data/hello

The JSON response should contain a data key with a value similar to:

"data": {
  "data":{
    "foo": "world"
  },
  "metadata": {
    "created_time": "2021-09-08T18:30:32.5140484Z",
    "deletion_time": "",
    "destroyed": false,
    "version": 1
  }
}

vault api

Try adding new versions, deleting, and destroying on your own.

UI

The UI should be up at running at http://localhost:8200/ui/vault. Use the root token to login. Then, explore the Key/Value backend on your own:

vault ui

Policies

Thus far we've been using the root policy to interact with the API. Let's set up a policy that only has read access.

Add a new config file called app-policy.json to "vault/policies":

{
  "path": {
    "kv/data/app/*": {
      "policy": "read"
    }
  }
}

Create a new policy back in the bash session:

bash-5.1# vault policy write app /vault/policies/app-policy.json

Success! Uploaded policy: app

Then, create a new token:

bash-5.1# vault token create -policy=app

Key                  Value
---                  -----
token                s.ZOUMx3RIhVRhI4ijlZg8KXRQ
token_accessor       TT53xOxbIfGjI7l4392gjXcg
token_duration       768h
token_renewable      true
token_policies       ["app" "default"]
identity_policies    []
policies             ["app" "default"]

Within another new terminal tab (you should now have three), add the VAULT_TOKEN environment variable with the new token:

$ export VAULT_TOKEN=your_token_goes_here

Try to read the foo secret that we previously set:

$ curl \
    -H "X-Vault-Token: $VAULT_TOKEN" \
    -X GET \
    http://127.0.0.1:8200/v1/kv/data/hello

You should not have the correct permissions to view that secret:

{
  "errors":[
    "1 error occurred:\n\t* permission denied\n\n"
  ]
}

Why can't we even read it? Jump back to the policy config in vault-config.json. kv/data/app/* indicates that the policy can only read from the app path.

As you've probably already noticed, nearly everything in Vault is path-based.

Back within the bash session in the container, add a new secret to the app/test path:

bash-5.1# vault kv put kv/app/test ping=pong

Key              Value
---              -----
created_time     2021-09-08T18:40:35.2694047Z
deletion_time    n/a
destroyed        false
version          1

You should be able to view the secret using the token associated with the app policy:

$ curl \
    -H "X-Vault-Token: $VAULT_TOKEN" \
    -X GET \
    http://127.0.0.1:8200/v1/kv/data/app/test

Policies can be managed from the UI as well:

vault ui

Encryption as a Service

Before we look at dynamic secrets, let's quickly review the Transit backend, which can be used as an "encryption as a service" for:

  • Encrypting and decrypting data "in-transit" without storing it inside Vault
  • Easily integrating encryption into your application workflow

Back within the bash session in the container, enable Transit:

bash-5.1# vault secrets enable transit

Success! Enabled the transit secrets engine at: transit/

Configure a named encryption key:

bash-5.1# vault write -f transit/keys/foo

Success! Data written to: transit/keys/foo

Encrypt:

bash-5.1# vault write transit/encrypt/foo plaintext=$(base64 <<< "my precious")

Key           Value
---           -----
ciphertext    vault:v1:cFnk5AQLE9Mg+mZ7Ej17vRmYT5aqheikdZQ1FC4vre5jAod0L/uHDA==

Decrypt:

bash-5.1# vault write transit/decrypt/foo ciphertext=vault:v1:cFnk5AQLE9Mg+mZ7Ej17vRmYT5aqheikdZQ1FC4vre5jAod0L/uHDA==

Key          Value
---          -----
plaintext    bXkgcHJlY2lvdXMK

Decode:

bash-5.1# base64 -d <<< "bXkgcHJlY2lvdXMK"

my precious

Test it out in the UI as well:

vault ui

Dynamic Secrets

As mentioned, Vault supports a number of dynamic secret backends for generating secrets dynamically when needed. For example, with the AWS and Google Cloud backends, you can create access credentials based on IAM policies. The Databases backend, meanwhile, generates database credentials based on configured roles.

Dynamic Secrets:

  • are generated on demand
  • have limited access based on role
  • are leased for a period of time
  • can be revoked
  • come with an audit trail

Let's look at how to generate AWS credentials using the AWS backend.

AWS Credentials

Enable the AWS secrets backend:

bash-5.1# vault secrets enable -path=aws aws

Success! Enabled the aws secrets engine at: aws/

Authenticate:

bash-5.1# vault write aws/config/root access_key=foo secret_key=bar

Success! Data written to: aws/config/root

Make sure to replace foo and bar with your AWS access key id and secret key, respectively.

Create role:

bash-5.1# vault write aws/roles/ec2-read credential_type=iam_user policy_document=-<<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "Stmt1426528957000",
      "Effect": "Allow",
      "Action": [
        "ec2:*"
      ],
      "Resource": [
        "*"
      ]
    }
  ]
}
EOF

Success! Data written to: aws/roles/ec2-read

Here, we created a new role based on AmazonEC2ReadOnlyAccess, which is an AWS-managed policy. As the name suggests, it give users read-only access to the EC2 console; they cannot perform any actions or create new resources. You can also use an inline policy to create a custom role based on your individual needs. We'll look at an example of this shortly. Refer to the AWS Secrets Engine docs for more info.

Remember: Dynamic Secrets are generated only when they are requested (i.e., a web app requests access to S3). They are not available in the store before this.

Create a new set of credentials:

bash-5.1# vault read aws/creds/ec2-read

Key                Value
---                -----
lease_id           aws/creds/ec2-read/9KdO6J7KVBiSwOPEvwrqqALG
lease_duration     768h
lease_renewable    true
access_key         AKIAZ4DZAKZKEULSDW5A
secret_key         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
security_token     <nil>

You should now be able to see the user within the "Users" section on the IAM console on AWS:

tôi xin lỗi

Leases and Revocation

In this section, we'll take a quick look at how to define a custom lease period and revoke a secret before the end of that period.

Create a new AWS role:

bash-5.1# vault write aws/roles/foo credential_type=iam_user policy_document=-<<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "Stmt1426528957000",
      "Effect": "Allow",
      "Action": [
        "ec2:*"
      ],
      "Resource": [
        "*"
      ]
    }
  ]
}
EOF

Success! Data written to: aws/roles/foo

Take note of the lease_duration when you create a new AWS credential:

bash-5.1# vault read aws/creds/foo

Key                Value
---                -----
lease_id           aws/creds/foo/F0oBbnBIHEoz0ywVVtbuJB7r
lease_duration     768h
lease_renewable    true
access_key         AKIAZ4DZAKZKLJKB7CPX
secret_key         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
security_token     <nil>

What if you only wanted the lease period for all AWS IAM dynamic secrets to be 30 minutes?

bash-5.1# vault write aws/config/lease lease=1800s lease_max=1800s

In this example, since lease_max is the same as lease, you won't be able to renew the token. If you set the lease_max to 3600s, you'd be able to renew the lease once. For more, review the Tokens and Leases guide.

Create a new credential:

bash-5.1# vault read aws/creds/foo

Key                Value
---                -----
lease_id           aws/creds/foo/xQlJpKDS1ljE9Awz0aywXgbB
lease_duration     30m
lease_renewable    true
access_key         AKIAZ4DZAKZKJPL5OM5W
secret_key         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
security_token     <nil>

Want to quickly revoke this credential? Grab the lease_id and then run:

bash-5.1# vault lease revoke aws/creds/foo/xQlJpKDS1ljE9Awz0aywXgbB

Want to revoke all AWS creds?

bash-5.1# vault lease revoke -prefix aws/

Refer to the Lease, Renew, and Revoke guide for more info these concepts.

Consul Backend

Cho đến nay, chúng tôi đã sử dụng chương trình phụ trợ Hệ thống tệp. Điều này sẽ không mở rộng ra ngoài một máy chủ duy nhất, vì vậy nó không tận dụng được tính khả dụng cao của Vault. May mắn thay, có một số phần mềm phụ trợ Lưu trữ khác , như phần phụ trợ Consul , được thiết kế cho các hệ thống phân tán.

Để thiết lập Consul , hãy bắt đầu bằng cách cập nhật tệp docker-compost.yml :

version: '3.8'

services:

  vault:
    build:
      context: ./vault
      dockerfile: Dockerfile
    ports:
      - 8200:8200
    volumes:
      - ./vault/config:/vault/config
      - ./vault/policies:/vault/policies
      - ./vault/data:/vault/data
      - ./vault/logs:/vault/logs
    environment:
      - VAULT_ADDR=http://127.0.0.1:8200
      - VAULT_API_ADDR=http://127.0.0.1:8200
    command: server -config=/vault/config/vault-config.json
    cap_add:
      - IPC_LOCK
    depends_on:
      - consul

  consul:
    build:
      context: ./consul
      dockerfile: Dockerfile
    ports:
      - 8500:8500
    command: agent -server -bind 0.0.0.0 -client 0.0.0.0 -bootstrap-expect 1 -config-file=/consul/config/config.json
    volumes:
      - ./consul/config/consul-config.json:/consul/config/config.json
      - ./consul/data:/consul/data

Thêm một thư mục mới trong thư mục gốc của dự án được gọi là "consul", sau đó thêm một Dockerfile mới vào thư mục mới tạo đó:

# base image
FROM alpine:3.14

# set consul version
ENV CONSUL_VERSION 1.10.2

# create a new directory
RUN mkdir /consul

# download dependencies
RUN apk --no-cache add \
      bash \
      ca-certificates \
      wget

# download and set up consul
RUN wget --quiet --output-document=/tmp/consul.zip https://releases.hashicorp.com/consul/${CONSUL_VERSION}/consul_${CONSUL_VERSION}_linux_amd64.zip && \
    unzip /tmp/consul.zip -d /consul && \
    rm -f /tmp/consul.zip && \
    chmod +x /consul/consul

# update PATH
ENV PATH="PATH=$PATH:$PWD/consul"

# add the config file
COPY ./config/consul-config.json /consul/config/config.json

# expose ports
EXPOSE 8300 8400 8500 8600

# run consul
ENTRYPOINT ["consul"]

Tiếp theo, trong thư mục "consul" thêm hai thư mục mới: "config" và "data". Sau đó, trong "config", thêm một tệp cấu hình có tên là consul-config.json :

{
  "datacenter": "localhost",
  "data_dir": "/consul/data",
  "log_level": "DEBUG",
  "server": true,
  "ui": true,
  "ports": {
    "dns": 53
  }
}

Hãy nhớ xem lại các tùy chọn Cấu hình từ tài liệu của Consul để biết thêm thông tin về các tùy chọn trên.

Thư mục "Consul" bây giờ sẽ giống như sau:

├── Dockerfile
├── config
│   └── consul-config.json
└── data

Thoát khỏi phiên bash. Đặt vùng chứa xuống, sau đó cập nhật tệp cấu hình Vault:

{
  "backend": {
    "consul": {
      "address": "consul:8500",
      "path": "vault/"
    }
  },
  "listener": {
    "tcp":{
      "address": "0.0.0.0:8200",
      "tls_disable": 1
    }
  },
  "ui": true
}

Vì vậy, bây giờ chúng tôi đang sử dụng chương trình phụ trợ Consul thay vì Hệ thống tệp. Chúng tôi đã sử dụng tên của dịch vụ consul, như một phần của địa chỉ. Khóa pathxác định đường dẫn trong kho lưu trữ khóa / giá trị của Consul nơi dữ liệu Vault sẽ được lưu trữ.

Xóa tất cả các tệp và thư mục trong thư mục "vault / data" để xóa phần phụ trợ Hệ thống tệp. Xây dựng các hình ảnh mới và quay các vùng chứa:

$ docker-compose down
$ docker-compose up -d --build

Đảm bảo tất cả đều tốt bằng cách điều hướng trong trình duyệt của bạn đến http: // localhost: 8500 / ui :

lãnh sự ui

Kiểm tra điều này từ CLI hoặc UI.

CLI

Tạo phiên bash mới trong vùng chứa Vault:

$ docker-compose exec vault bash

Sau đó chạy:

# Init
bash-5.1# vault operator init

# Unseal
bash-5.1# vault operator unseal

# Authenticate
bash-5.1# vault login

# Enable secrets
bash-5.1# vault secrets enable kv

# Add a new static secret
bash-5.1# vault kv put kv/foo bar=precious

# Read it back
bash-5.1# vault kv get kv/foo

Giao diện người dùng

lãnh sự kho tiền

Lưu ý rằng không có tệp hoặc thư mục nào trong "vault / data". Tại sao bạn nghĩ rằng đây là?

Bạn muốn thêm một máy chủ Consul khác vào hỗn hợp? Thêm một dịch vụ mới vào docker-compos.yml :

consul-worker:
  build:
    context: ./consul
    dockerfile: Dockerfile
  command: agent -server -join consul -config-file=/consul/config/config.json
  volumes:
    - ./consul/config/consul-config.json:/consul/config/config.json
  depends_on:
    - consul

Ở đây, chúng tôi đã sử dụng lệnh tham gia để kết nối tác nhân này với một cụm hiện có. Lưu ý rằng chúng tôi chỉ cần tham chiếu đến tên dịch vụ như thế nào consul:.

Sau đó:

  1. Thoát khỏi phiên bash (nếu cần)
  2. Mang các thùng chứa xuống
  3. Xóa thư mục dữ liệu trong "consul / data" (Tại sao?)
  4. Quay các thùng chứa sao lưu và kiểm tra

lãnh sự ui

Sự kết luận

Trong hướng dẫn này, chúng tôi đã giới thiệu cho các bạn cách thiết lập và chạy Vault and Consul bên trong vùng chứa Docker. Bây giờ bạn đã hiểu rõ về cách tương tác với Vault và thực hiện các thao tác cơ bản.

Lấy mã cuối cùng từ repo vault-consul-docker . Kiểm tra bản trình bày là tốt.

Nguồn:  https://testdriven.io

#vault #consul 

Cách Thiết Lập Và Sử Dụng Các Dự Án Vault Và Consul Của Hashicorp

Как настроить и использовать проекты Hashicorp Vault и Consul

В следующем руководстве подробно описано, как настроить и использовать проекты Hashicorp Vault и Consul для безопасного хранения секретов и управления ими.

Мы начнем с запуска одного экземпляра Vault в контейнере Docker, а затем перейдем к управлению как статическими, так и динамическими секретами вместе с функцией Vault «шифрование как услуга». Затем мы добавим Consul и посмотрим, как масштабировать Vault.

Основные зависимости:

  • Докер v20.10.8
  • Докер-композитор v1.29.2
  • Убежище v1.8.2
  • Консул v1.10.2

Цели

К концу этого урока вы должны уметь:

  1. Объясните, что такое Vault и почему вы можете захотеть его использовать.
  2. Описать базовую архитектуру Vault, а также динамические и статические секреты, различные серверные части (хранилище, секрет, аутентификация, аудит) и то, как Vault можно использовать в качестве «шифрования как услуги».
  3. Настройте и запустите Vault и Consul с помощью Docker
  4. Раскрутите Vault с помощью серверной части файловой системы
  5. Инициализировать и распечатать хранилище
  6. Аутентификация в хранилище
  7. Настройте серверную часть аудита для регистрации всех взаимодействий с Vault.
  8. Работайте со статическими и динамическими секретами через интерфейс командной строки, HTTP API и пользовательский интерфейс.
  9. Создайте политику Vault, чтобы ограничить доступ к определенному пути
  10. Используйте серверную часть Transit как «шифрование как услугу».
  11. Настройте Consul для работы с Vault в качестве хранилища для секретов.
  12. Определите пользовательский период аренды для секрета и отзовите секрет до окончания этого периода.

Что такое Сейф?

Vault — это инструмент с открытым исходным кодом, используемый для безопасного хранения секретов и управления ими.

Что такое секрет? Секреты в контексте этого руководства — это конфиденциальная или личная информация, такая как учетные данные базы данных, ключи SSH, имена пользователей и пароли, учетные данные AWS IAM, токены API, номера социального страхования, номера кредитных карт и т. д.

Найдите минутку, чтобы подумать о том, как ваша команда в настоящее время управляет секретами и распределяет их:

  1. Кто имеет к ним доступ?
  2. Кто ими управляет?
  3. Как вы контролируете, кто имеет к ним доступ?
  4. Как ваши приложения получают их?
  5. Как они обновляются?
  6. Как они отзываются?

Vault дает ответы на эти вопросы и помогает решить следующие проблемы, связанные с управлением секретами:

ПроблемыЦели Убежища
Секреты повсюду.Хранилище — единственный источник правды для всех секретов.
Обычно они не зашифрованы.Сейф автоматически управляет шифрованием (во время передачи и хранения).
Их сложно динамически генерировать.Секреты могут генерироваться динамически.
Еще труднее сдать в аренду и отозвать их.Секреты можно сдавать в аренду и отзывать.
Нет никакого контрольного следа.Существует контрольный журнал для создания и использования секретов.

Vault состоит из нескольких движущихся частей, поэтому может потребоваться некоторое время, чтобы привыкнуть к общей архитектуре. Найдите минутку, чтобы просмотреть руководство по архитектуре , приняв во внимание следующие серверные части:

БэкендИспользоватьПримеры
ХранилищеГде хранятся секретыКонсул *, файловая система *, в памяти, PostgreSQL, S3
СекретОбрабатывает статические или динамические секретыAWS *, базы данных, ключ/значение *, RabbitMQ, SSH
АвторизацияОбрабатывает аутентификацию и авторизациюAWS, Azure, Google Cloud, GitHub, токены *, имя пользователя и пароль
АудитРегистрирует все запросы и ответыФайл *, системный журнал, сокет

*используется в этом уроке

Итак, давайте начнем использовать Vault.

Бэкэнд файловой системы

Чтобы быстро приступить к работе, мы будем использовать серверную часть файловой системы для хранения секретов в состоянии покоя.

Серверную часть файловой системы следует использовать только для локальной разработки или развертывания Vault на одном сервере, поскольку она не поддерживает высокую доступность.

Создайте новый каталог проекта:

$ mkdir vault-consul-docker && cd vault-consul-docker

Затем добавьте следующие папки:

└── vault
    ├── config
    ├── data
    ├── logs
    └── policies

Добавьте Dockerfile в каталог «хранилище»:

# base image
FROM alpine:3.14

# set vault version
ENV VAULT_VERSION 1.8.2

# create a new directory
RUN mkdir /vault

# download dependencies
RUN apk --no-cache add \
      bash \
      ca-certificates \
      wget

# download and set up vault
RUN wget --quiet --output-document=/tmp/vault.zip https://releases.hashicorp.com/vault/${VAULT_VERSION}/vault_${VAULT_VERSION}_linux_amd64.zip && \
    unzip /tmp/vault.zip -d /vault && \
    rm -f /tmp/vault.zip && \
    chmod +x /vault

# update PATH
ENV PATH="PATH=$PATH:$PWD/vault"

# add the config file
COPY ./config/vault-config.json /vault/config/vault-config.json

# expose port 8200
EXPOSE 8200

# run vault
ENTRYPOINT ["vault"]

Затем добавьте файл docker-compose.yml в корень проекта:

version: '3.8'

services:

  vault:
    build:
      context: ./vault
      dockerfile: Dockerfile
    ports:
      - 8200:8200
    volumes:
      - ./vault/config:/vault/config
      - ./vault/policies:/vault/policies
      - ./vault/data:/vault/data
      - ./vault/logs:/vault/logs
    environment:
      - VAULT_ADDR=http://127.0.0.1:8200
      - VAULT_API_ADDR=http://127.0.0.1:8200
    command: server -config=/vault/config/vault-config.json
    cap_add:
      - IPC_LOCK

Добавьте файл конфигурации с именем vault-config.json в «vault/config»:

{
  "backend": {
    "file": {
      "path": "vault/data"
    }
  },
  "listener": {
    "tcp":{
      "address": "0.0.0.0:8200",
      "tls_disable": 1
    }
  },
  "ui": true
}

Здесь мы настроили Vault для использования серверной части файловой системы, определили прослушиватель для Vault, отключили TLS и включили пользовательский интерфейс Vault . Ознакомьтесь с документацией для получения дополнительной информации о настройке Vault.

Теперь мы можем создать образ и запустить контейнер:

$ docker-compose up -d --build

Откройте журналы Docker, чтобы убедиться, что в сборке нет ошибок:

$ docker-compose logs

Вы должны увидеть что-то похожее на:

Attaching to vault-consul-docker_vault_1
vault_1  | ==> Vault server configuration:
vault_1  |
vault_1  |              Api Address: http://127.0.0.1:8200
vault_1  | 2021-09-08T14:48:35.014Z [INFO]  proxy environment: http_proxy="" https_proxy="" no_proxy=""
vault_1  |                      Cgo: disabled
vault_1  |          Cluster Address: https://127.0.0.1:8201
vault_1  |               Go Version: go1.16.7
vault_1  |               Listener 1: tcp (addr: "0.0.0.0:8200", cluster address: "0.0.0.0:8201", max_request_duration: "1m30s", max_request_size: "33554432", tls: "disabled")
vault_1  |                Log Level: info
vault_1  |                    Mlock: supported: true, enabled: true
vault_1  |            Recovery Mode: false
vault_1  |                  Storage: file
vault_1  |                  Version: Vault v1.8.2
vault_1  |              Version Sha: aca76f63357041a43b49f3e8c11d67358496959f
vault_1  |
vault_1  | ==> Vault server started! Log data will stream in below:
vault_1  |

Инициализация и распечатка

Запустите сеанс bash в работающем контейнере:

$ docker-compose exec vault bash

В оболочке инициализируйте Vault:

bash-5.1# vault operator init

Обратите внимание на ключи распечатывания и первоначальный корневой токен. Вам нужно будет предоставлять три ключа распечатывания каждый раз, когда сервер Vault повторно запечатывается или перезапускается.

Почему 3 ключа? Просмотрите секретный обмен Шамира .

Теперь вы можете открыть Хранилище с помощью трех ключей:

bash-5.1# vault operator unseal
Unseal Key (will be hidden):

Запустите эту команду еще два раза, каждый раз используя разные клавиши. После этого убедитесь, Sealedчто false:

Key             Value
---             -----
Seal Type       shamir
Initialized     true
Sealed          false
Total Shares    5
Threshold       3
Version         1.8.2
Storage Type    file
Cluster Name    vault-cluster-8fcf9d05
Cluster ID      d86e0274-ad9c-d2c1-d6ec-baeab410797b
HA Enabled      false

Используя корневой токен, теперь вы можете аутентифицироваться:

bash-5.1# vault login
Token (will be hidden):

Вы должны увидеть что-то похожее на:

Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.

Key                  Value
---                  -----
token                s.c0kYHWiOTqQvtR8JuSeTz6sZ
token_accessor       3FQJVxOY5C1brzlHHQSFaCdZ
token_duration       ∞
token_renewable      false
token_policies       ["root"]
identity_policies    []
policies             ["root"]

Имейте в виду, что при этом используется корневая политика. В рабочей среде вам потребуется настроить политики с разными уровнями доступа. Вскоре мы рассмотрим, как это сделать.

инициализация хранилища

Хранилище теперь распечатано и готово к использованию.

Аудиторская проверка

Прежде чем мы проверим функциональность, давайте включим устройство аудита :

bash-5.1# vault audit enable file file_path=/vault/logs/audit.log

Success! Enabled the file audit device at: file/

Теперь вы сможете просматривать журналы локально в «хранилище/журналах». Для проверки выполните следующую команду, чтобы просмотреть все включенные устройства аудита:

bash-5.1# vault audit list

Path     Type    Description
----     ----    -----------
file/    file    n/a

Запрос и последующий ответ должны регистрироваться в vault/logs/audit.log . Взглянем.

Секреты

В Vault есть два типа секретов: статические и динамические .

Статические секреты (например, зашифрованные Redis или Memcached) имеют интервалы обновления, но срок их действия не истекает, если они не отменены явным образом. Они определяются заранее с помощью бэкэнда Key/Value (ранее «общий» бэкэнд), а затем совместно используются.

надежное секретное хранилище

Динамические секреты генерируются по запросу. Они имеют принудительные договоры аренды и обычно истекают через короткий промежуток времени. Поскольку они не существуют до тех пор, пока к ним не будет осуществлен доступ, они менее уязвимы, поэтому динамические секреты гораздо более безопасны. Vault поставляется с несколькими динамическими бэкендами, например, AWS , Databases , Google Cloud , Consul и RabbitMQ .

Ознакомьтесь с записью блога « Почему нам нужны динамические секреты », чтобы получить дополнительную информацию о преимуществах использования динамических секретов.

Статические секреты

Хранилищем можно управлять через интерфейс командной строки , HTTP API или пользовательский интерфейс .

CLI

Все еще в сеансе bash в контейнере мы можем создавать, читать, обновлять и удалять секреты. Мы также рассмотрим, как создавать версии и откатывать секреты.

Включите секреты с помощью следующей команды:

bash-5.1# vault secrets enable kv

Success! Enabled the kv secrets engine at: kv/

Создайте новый секрет с ключом barи значением preciousвнутри kv/fooпути:

bash-5.1# vault kv put kv/foo bar=precious

Success! Data written to: kv/foo

Читать:

bash-5.1# vault kv get kv/foo

=== Data ===
Key    Value
---    -----
bar    precious

Чтобы работать с разными версиями определенного ключа, нам нужно обновить до версии 2 серверную часть Key/Value :

bash-5.1# vault kv enable-versioning kv/

Success! Tuned the secrets engine at: kv/

Добавьте версию 2, изменив значение на copper:

bash-5.1# vault kv put kv/foo bar=copper

Key              Value
---              -----
created_time     2021-09-08T18:23:14.4154928Z
deletion_time    n/a
destroyed        false
version          2

Читать версию 1:

bash-5.1# vault kv get -version=1 kv/foo

====== Metadata ======
Key              Value
---              -----
created_time     2021-09-08T18:22:37.2548824Z
deletion_time    n/a
destroyed        false
version          1

=== Data ===
Key    Value
---    -----
bar    precious

Читать версию 2:

bash-5.1# vault kv get -version=2 kv/foo

====== Metadata ======
Key              Value
---              -----
created_time     2021-09-08T18:23:14.4154928Z
deletion_time    n/a
destroyed        false
version          2

=== Data ===
Key    Value
---    -----
bar    copper

Удалить последнюю версию (например, версию 2):

bash-5.1# vault kv delete kv/foo

Success! Data deleted (if it existed) at: kv/foo

Удалить версию 1:

bash-5.1# vault kv delete -versions=1 kv/foo

Success! Data deleted (if it existed) at: kv/foo

Вы также можете восстановить:

bash-5.1# vault kv undelete -versions=1 kv/foo

Success! Data written to: kv/undelete/foo

Удаление сродни мягкому удалению. Если вы хотите удалить базовые метаданные, вам придется использовать команду destroy :

bash-5.1# vault kv destroy -versions=1 kv/foo

Success! Data written to: kv/destroy/foo

Просмотрите v1 и v2 , чтобы просмотреть все доступные команды.

Обратите внимание на журнал аудита. Каждый из вышеуказанных запросов был зарегистрирован!

API

Вы также можете взаимодействовать с Vault через HTTP API . Мы будем делать запросы к v2 API. Откройте новую вкладку терминала, а затем установите корневой токен в качестве переменной среды:

$ export VAULT_TOKEN=your_token_goes_here

Создайте новый секрет с именем fooсо значением world:

$ curl \
    -H "X-Vault-Token: $VAULT_TOKEN" \
    -H "Content-Type: application/json" \
    -X POST \
    -d '{ "data": { "foo": "world" } }' \
    http://127.0.0.1:8200/v1/kv/data/hello

Прочтите секрет:

$ curl \
    -H "X-Vault-Token: $VAULT_TOKEN" \
    -X GET \
    http://127.0.0.1:8200/v1/kv/data/hello

Ответ JSON должен содержать dataключ со значением, подобным:

"data": {
  "data":{
    "foo": "world"
  },
  "metadata": {
    "created_time": "2021-09-08T18:30:32.5140484Z",
    "deletion_time": "",
    "destroyed": false,
    "version": 1
  }
}

API хранилища

Попробуйте добавлять новые версии, удалять и уничтожать самостоятельно.

интерфейс

Пользовательский интерфейс должен работать по адресу http://localhost:8200/ui/vault . Используйте корневой токен для входа. Затем самостоятельно изучите бэкэнд Key/Value:

интерфейс хранилища

Политики

До сих пор мы использовали корневую политику для взаимодействия с API. Давайте настроим политику, которая имеет доступ только для чтения.

Добавьте новый файл конфигурации с именем app-policy.json в «vault/policies»:

{
  "path": {
    "kv/data/app/*": {
      "policy": "read"
    }
  }
}

Создайте новую политику в сеансе bash:

bash-5.1# vault policy write app /vault/policies/app-policy.json

Success! Uploaded policy: app

Затем создайте новый токен:

bash-5.1# vault token create -policy=app

Key                  Value
---                  -----
token                s.ZOUMx3RIhVRhI4ijlZg8KXRQ
token_accessor       TT53xOxbIfGjI7l4392gjXcg
token_duration       768h
token_renewable      true
token_policies       ["app" "default"]
identity_policies    []
policies             ["app" "default"]

На другой новой вкладке терминала (теперь их должно быть три) добавьте VAULT_TOKENпеременную среды с новым токеном:

$ export VAULT_TOKEN=your_token_goes_here

Попробуйте прочитать fooсекрет, который мы установили ранее:

$ curl \
    -H "X-Vault-Token: $VAULT_TOKEN" \
    -X GET \
    http://127.0.0.1:8200/v1/kv/data/hello

У вас не должно быть правильных разрешений для просмотра этого секрета:

{
  "errors":[
    "1 error occurred:\n\t* permission denied\n\n"
  ]
}

Почему мы даже не можем это прочитать? Вернитесь к конфигурации политики в vault-config.json . kv/data/app/*указывает, что политика может читать только из appпути.

Как вы, наверное, уже заметили, почти все в Vault основано на путях.

Вернувшись в сеанс bash в контейнере, добавьте новый секрет к app/testпути:

bash-5.1# vault kv put kv/app/test ping=pong

Key              Value
---              -----
created_time     2021-09-08T18:40:35.2694047Z
deletion_time    n/a
destroyed        false
version          1

Вы должны иметь возможность просматривать секрет, используя токен, связанный с appполитикой:

$ curl \
    -H "X-Vault-Token: $VAULT_TOKEN" \
    -X GET \
    http://127.0.0.1:8200/v1/kv/data/app/test

Политиками также можно управлять из пользовательского интерфейса:

интерфейс хранилища

Шифрование как услуга

Прежде чем мы рассмотрим динамические секреты, давайте быстро рассмотрим серверную часть Transit , которую можно использовать как «шифрование как услугу» для:

  • Шифрование и дешифрование данных «в пути» без сохранения их в хранилище
  • Простая интеграция шифрования в рабочий процесс вашего приложения

Вернувшись в сеанс bash в контейнере, включите Transit:

bash-5.1# vault secrets enable transit

Success! Enabled the transit secrets engine at: transit/

Настройте именованный ключ шифрования:

bash-5.1# vault write -f transit/keys/foo

Success! Data written to: transit/keys/foo

Зашифровать:

bash-5.1# vault write transit/encrypt/foo plaintext=$(base64 <<< "my precious")

Key           Value
---           -----
ciphertext    vault:v1:cFnk5AQLE9Mg+mZ7Ej17vRmYT5aqheikdZQ1FC4vre5jAod0L/uHDA==

Расшифровать:

bash-5.1# vault write transit/decrypt/foo ciphertext=vault:v1:cFnk5AQLE9Mg+mZ7Ej17vRmYT5aqheikdZQ1FC4vre5jAod0L/uHDA==

Key          Value
---          -----
plaintext    bXkgcHJlY2lvdXMK

Расшифровать:

bash-5.1# base64 -d <<< "bXkgcHJlY2lvdXMK"

my precious

Проверьте это и в пользовательском интерфейсе:

интерфейс хранилища

Динамические секреты

Как уже упоминалось, Vault поддерживает ряд динамических секретных бэкэндов для динамического создания секретов при необходимости. Например, с помощью серверных частей AWS и Google Cloud вы можете создавать учетные данные для доступа на основе политик IAM. Тем временем серверная часть баз данных генерирует учетные данные базы данных на основе настроенных ролей .

Динамические секреты:

  • генерируются по запросу
  • иметь ограниченный доступ в зависимости от роли
  • сдаются в аренду на срок
  • может быть отозван
  • прийти с аудиторским следом

Давайте посмотрим, как сгенерировать учетные данные AWS с помощью серверной части AWS.

Учетные данные AWS

Включите серверную часть секретов AWS:

bash-5.1# vault secrets enable -path=aws aws

Success! Enabled the aws secrets engine at: aws/

Аутентификация:

bash-5.1# vault write aws/config/root access_key=foo secret_key=bar

Success! Data written to: aws/config/root

Обязательно замените fooи barна идентификатор ключа доступа AWS и секретный ключ соответственно.

Создать роль:

bash-5.1# vault write aws/roles/ec2-read credential_type=iam_user policy_document=-<<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "Stmt1426528957000",
      "Effect": "Allow",
      "Action": [
        "ec2:*"
      ],
      "Resource": [
        "*"
      ]
    }
  ]
}
EOF

Success! Data written to: aws/roles/ec2-read

Здесь мы создали новую роль на основе политикиAmazonEC2ReadOnlyAccess , управляемой AWS . Как следует из названия, он предоставляет пользователям доступ только для чтения к консоли EC2; они не могут выполнять какие-либо действия или создавать новые ресурсы. Вы также можете использовать встроенную политику для создания пользовательской роли в соответствии с вашими индивидуальными потребностями. Вскоре мы рассмотрим это на примере. Дополнительную информацию см . в документации AWS Secrets Engine .

Помните : динамические секреты генерируются только тогда, когда они запрашиваются (например, веб-приложение запрашивает доступ к S3). До этого в магазине их нет.

Создайте новый набор учетных данных:

bash-5.1# vault read aws/creds/ec2-read

Key                Value
---                -----
lease_id           aws/creds/ec2-read/9KdO6J7KVBiSwOPEvwrqqALG
lease_duration     768h
lease_renewable    true
access_key         AKIAZ4DZAKZKEULSDW5A
secret_key         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
security_token     <nil>

Теперь вы должны увидеть пользователя в разделе «Пользователи» на консоли IAM на AWS:

Мне жаль

Аренда и отзыв

В этом разделе мы кратко рассмотрим, как определить пользовательский период аренды и отозвать секрет до окончания этого периода.

Создайте новую роль AWS:

bash-5.1# vault write aws/roles/foo credential_type=iam_user policy_document=-<<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "Stmt1426528957000",
      "Effect": "Allow",
      "Action": [
        "ec2:*"
      ],
      "Resource": [
        "*"
      ]
    }
  ]
}
EOF

Success! Data written to: aws/roles/foo

Обратите внимание на то, что lease_durationпри создании новых учетных данных AWS:

bash-5.1# vault read aws/creds/foo

Key                Value
---                -----
lease_id           aws/creds/foo/F0oBbnBIHEoz0ywVVtbuJB7r
lease_duration     768h
lease_renewable    true
access_key         AKIAZ4DZAKZKLJKB7CPX
secret_key         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
security_token     <nil>

Что, если вы хотите, чтобы период аренды для всех динамических секретов AWS IAM составлял 30 минут?

bash-5.1# vault write aws/config/lease lease=1800s lease_max=1800s

В этом примере, поскольку lease_maxэто то же самое lease, что и , вы не сможете обновить токен. Если вы установите значение lease_max, 3600sвы сможете продлить аренду один раз. Для получения дополнительной информации ознакомьтесь с руководством по токенам и аренде .

Создайте новые учетные данные:

bash-5.1# vault read aws/creds/foo

Key                Value
---                -----
lease_id           aws/creds/foo/xQlJpKDS1ljE9Awz0aywXgbB
lease_duration     30m
lease_renewable    true
access_key         AKIAZ4DZAKZKJPL5OM5W
secret_key         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
security_token     <nil>

Хотите быстро отозвать эти учетные данные? Возьмите, lease_idа затем запустите:

bash-5.1# vault lease revoke aws/creds/foo/xQlJpKDS1ljE9Awz0aywXgbB

Хотите отозвать все кредиты AWS?

bash-5.1# vault lease revoke -prefix aws/

Дополнительные сведения об этих концепциях см. в руководстве по аренде, продлению и отзыву .

Консул Бэкэнд

До сих пор мы использовали серверную часть файловой системы . Он не будет масштабироваться за пределы одного сервера, поэтому он не использует преимущества высокой доступности Vault. К счастью, существует ряд других бэкендов Storage , таких как бэкенд Consul , предназначенных для распределенных систем.

Чтобы настроить Consul , начните с обновления файла docker-compose.yml :

version: '3.8'

services:

  vault:
    build:
      context: ./vault
      dockerfile: Dockerfile
    ports:
      - 8200:8200
    volumes:
      - ./vault/config:/vault/config
      - ./vault/policies:/vault/policies
      - ./vault/data:/vault/data
      - ./vault/logs:/vault/logs
    environment:
      - VAULT_ADDR=http://127.0.0.1:8200
      - VAULT_API_ADDR=http://127.0.0.1:8200
    command: server -config=/vault/config/vault-config.json
    cap_add:
      - IPC_LOCK
    depends_on:
      - consul

  consul:
    build:
      context: ./consul
      dockerfile: Dockerfile
    ports:
      - 8500:8500
    command: agent -server -bind 0.0.0.0 -client 0.0.0.0 -bootstrap-expect 1 -config-file=/consul/config/config.json
    volumes:
      - ./consul/config/consul-config.json:/consul/config/config.json
      - ./consul/data:/consul/data

Добавьте новый каталог в корень проекта с именем «consul», а затем добавьте новый Dockerfile в этот вновь созданный каталог:

# base image
FROM alpine:3.14

# set consul version
ENV CONSUL_VERSION 1.10.2

# create a new directory
RUN mkdir /consul

# download dependencies
RUN apk --no-cache add \
      bash \
      ca-certificates \
      wget

# download and set up consul
RUN wget --quiet --output-document=/tmp/consul.zip https://releases.hashicorp.com/consul/${CONSUL_VERSION}/consul_${CONSUL_VERSION}_linux_amd64.zip && \
    unzip /tmp/consul.zip -d /consul && \
    rm -f /tmp/consul.zip && \
    chmod +x /consul/consul

# update PATH
ENV PATH="PATH=$PATH:$PWD/consul"

# add the config file
COPY ./config/consul-config.json /consul/config/config.json

# expose ports
EXPOSE 8300 8400 8500 8600

# run consul
ENTRYPOINT ["consul"]

Затем в каталоге «consul» добавьте два новых каталога: «config» и «data». Затем в «config» добавьте файл конфигурации с именем consul-config.json :

{
  "datacenter": "localhost",
  "data_dir": "/consul/data",
  "log_level": "DEBUG",
  "server": true,
  "ui": true,
  "ports": {
    "dns": 53
  }
}

Обязательно ознакомьтесь с параметрами конфигурации из документов Consul для получения дополнительной информации о вышеуказанных параметрах.

Теперь каталог «consul» должен выглядеть так:

├── Dockerfile
├── config
│   └── consul-config.json
└── data

Выйдите из сеанса bash. Перенесите контейнер, а затем обновите файл конфигурации Vault:

{
  "backend": {
    "consul": {
      "address": "consul:8500",
      "path": "vault/"
    }
  },
  "listener": {
    "tcp":{
      "address": "0.0.0.0:8200",
      "tls_disable": 1
    }
  },
  "ui": true
}

Итак, теперь мы используем бэкенд Consul вместо файловой системы. Мы использовали название службы consulкак часть адреса. Ключ определяет путь pathв хранилище ключей/значений Consul, где будут храниться данные Vault.

Очистите все файлы и папки в каталоге «vault/data», чтобы удалить серверную часть файловой системы. Создайте новые образы и разверните контейнеры:

$ docker-compose down
$ docker-compose up -d --build

Убедитесь, что все в порядке, перейдя в браузере по адресу http://localhost:8500/ui :

консул пользовательского интерфейса

Проверьте это из интерфейса командной строки или пользовательского интерфейса.

CLI

Создайте новую сессию bash в контейнере Vault:

$ docker-compose exec vault bash

Затем запустите:

# Init
bash-5.1# vault operator init

# Unseal
bash-5.1# vault operator unseal

# Authenticate
bash-5.1# vault login

# Enable secrets
bash-5.1# vault secrets enable kv

# Add a new static secret
bash-5.1# vault kv put kv/foo bar=precious

# Read it back
bash-5.1# vault kv get kv/foo

интерфейс

консул хранилища

Обратите внимание, что в «хранилище/данных» нет файлов или папок. Как вы думаете, почему это так?

Хотите добавить еще один сервер Consul? Добавьте новый сервис в docker-compose.yml :

consul-worker:
  build:
    context: ./consul
    dockerfile: Dockerfile
  command: agent -server -join consul -config-file=/consul/config/config.json
  volumes:
    - ./consul/config/consul-config.json:/consul/config/config.json
  depends_on:
    - consul

Здесь мы использовали команду join для подключения этого агента к существующему кластеру. Обратите внимание, что нам просто нужно было сослаться на имя службы: consul.

Затем:

  1. Выход из bash-сессии (при необходимости)
  2. Снести контейнеры
  3. Очистите каталог данных в «consul/data» (почему?)
  4. Вращайте контейнеры обратно и проверяйте

консул пользовательского интерфейса

Вывод

В этом руководстве мы рассмотрели, как настроить и запустить Vault и Consul внутри контейнера Docker. Теперь у вас должно быть четкое представление о том, как взаимодействовать с Vault и выполнять основные операции.

Возьмите окончательный код из репозитория vault-consul-docker . Ознакомьтесь также с презентацией .

Источник:  https://testdriven.io

#vault #consul 

Как настроить и использовать проекты Hashicorp Vault и Consul
工藤  晃

工藤 晃

1660625880

如何設置和使用 Hashicorp 的 Vault 和 Consul 項目

以下教程詳細介紹瞭如何設置和使用 Hashicorp 的 Vault 和 Consul 項目來安全地存儲和管理機密。

我們將從在 Docker 容器中啟動單個 Vault 實例開始,然後開始管理靜態和動態機密以及 Vault 的“加密即服務”功能。然後,我們將 Consul 添加到組合中,並研究如何擴展 Vault。

主要依賴:

  • 碼頭工人 v20.10.8
  • Docker-Compose v1.29.2
  • 保險櫃 v1.8.2
  • 領事 v1.10.2

目標

在本教程結束時,您應該能夠:

  1. 解釋 Vault 是什麼以及為什麼要使用它
  2. 描述基本的 Vault 架構以及動態和靜態機密、各種後端(存儲、機密、身份驗證、審計),以及如何將 Vault 用作“加密即服務”
  3. 使用 Docker 配置和運行 Vault 和 Consul
  4. 使用文件系統後端啟動 Vault
  5. 初始化和解封 Vault
  6. 針對 Vault 進行身份驗證
  7. 配置審核後端以記錄與 Vault 的所有交互
  8. 通過 CLI、HTTP API 和 UI 處理靜態和動態機密
  9. 創建 Vault 策略以限制對特定路徑的訪問
  10. 將 Transit 後端用作“加密即服務”
  11. 設置 Consul 以使用 Vault 作為機密的存儲後端
  12. 為密鑰定義自定義租期並在該期限結束前撤銷密鑰

什麼是保險櫃?

Vault是一種用於安全存儲和管理機密的開源工具。

What is a secret? Secrets, in the context of this tutorial, are securely-sensitive or personally identifiable info like database credentials, SSH keys, usernames and passwords, AWS IAM credentials, API tokens, Social Security Numbers, credit card numbers, just to name a few.

Take a moment to think about how your team currently manages and distributes secrets:

  1. Who has access to them?
  2. Who manages them?
  3. How do you control who has access to them?
  4. How do your apps get them?
  5. How are they updated?
  6. How are they revoked?

Vault provides answers to those questions and helps to solve the following problems with regard to secret management:

ProblemsVault's Goals
Secrets are everywhere.Vault is the single source of truth for all secrets.
They are generally unencrypted.Vault manages encryption (during transit and at rest) out of the box.
It's difficult to dynamically generate them.Secrets can be dynamically generated.
It's even more difficult to lease and revoke them.Secrets can be leased and revoked.
There's no audit trail.There's an audit trail for generating and using secrets.

Vault has a number of moving pieces so it can take some time to get up to speed with the overall architecture. Take a moment to review the Architecture guide, taking note of the following backends:

BackendUseExamples
StorageWhere secrets are storedConsul*, Filesystem*, In-Memory, PostgreSQL, S3
SecretHandles static or dynamic secretsAWS*, Databases, Key/Value*, RabbitMQ, SSH
AuthHandles authentication and authorizationAWS, Azure, Google Cloud, GitHub, Tokens*, Username & Password
AuditLogs all requests and responsesFile*, Syslog, Socket

* used in this tutorial

With that, let's start using Vault.

Filesystem Backend

To get up and running quickly, we'll use the Filesystem backend to store secrets at rest.

The filesystem backend should only be used for local development or a single-server Vault deployment since it does not support high availability.

Create a new project directory:

$ mkdir vault-consul-docker && cd vault-consul-docker

Then add the following folders:

└── vault
    ├── config
    ├── data
    ├── logs
    └── policies

Add a Dockerfile to the "vault" directory:

# base image
FROM alpine:3.14

# set vault version
ENV VAULT_VERSION 1.8.2

# create a new directory
RUN mkdir /vault

# download dependencies
RUN apk --no-cache add \
      bash \
      ca-certificates \
      wget

# download and set up vault
RUN wget --quiet --output-document=/tmp/vault.zip https://releases.hashicorp.com/vault/${VAULT_VERSION}/vault_${VAULT_VERSION}_linux_amd64.zip && \
    unzip /tmp/vault.zip -d /vault && \
    rm -f /tmp/vault.zip && \
    chmod +x /vault

# update PATH
ENV PATH="PATH=$PATH:$PWD/vault"

# add the config file
COPY ./config/vault-config.json /vault/config/vault-config.json

# expose port 8200
EXPOSE 8200

# run vault
ENTRYPOINT ["vault"]

Next, add a docker-compose.yml file to the project root:

version: '3.8'

services:

  vault:
    build:
      context: ./vault
      dockerfile: Dockerfile
    ports:
      - 8200:8200
    volumes:
      - ./vault/config:/vault/config
      - ./vault/policies:/vault/policies
      - ./vault/data:/vault/data
      - ./vault/logs:/vault/logs
    environment:
      - VAULT_ADDR=http://127.0.0.1:8200
      - VAULT_API_ADDR=http://127.0.0.1:8200
    command: server -config=/vault/config/vault-config.json
    cap_add:
      - IPC_LOCK

Add a config file called vault-config.json to "vault/config":

{
  "backend": {
    "file": {
      "path": "vault/data"
    }
  },
  "listener": {
    "tcp":{
      "address": "0.0.0.0:8200",
      "tls_disable": 1
    }
  },
  "ui": true
}

Here, we configured Vault to use the Filesystem backend, defined the listener for Vault, disabled TLS, and enabled the Vault UI. Review the docs for more info on configuring Vault.

Now we can build the image and spin up the container:

$ docker-compose up -d --build

Pull up the Docker logs to make sure there were no errors in the build:

$ docker-compose logs

You should see something similar to:

Attaching to vault-consul-docker_vault_1
vault_1  | ==> Vault server configuration:
vault_1  |
vault_1  |              Api Address: http://127.0.0.1:8200
vault_1  | 2021-09-08T14:48:35.014Z [INFO]  proxy environment: http_proxy="" https_proxy="" no_proxy=""
vault_1  |                      Cgo: disabled
vault_1  |          Cluster Address: https://127.0.0.1:8201
vault_1  |               Go Version: go1.16.7
vault_1  |               Listener 1: tcp (addr: "0.0.0.0:8200", cluster address: "0.0.0.0:8201", max_request_duration: "1m30s", max_request_size: "33554432", tls: "disabled")
vault_1  |                Log Level: info
vault_1  |                    Mlock: supported: true, enabled: true
vault_1  |            Recovery Mode: false
vault_1  |                  Storage: file
vault_1  |                  Version: Vault v1.8.2
vault_1  |              Version Sha: aca76f63357041a43b49f3e8c11d67358496959f
vault_1  |
vault_1  | ==> Vault server started! Log data will stream in below:
vault_1  |

Initializing and Unsealing

Start a bash session within the running container:

$ docker-compose exec vault bash

Within the shell, initialize Vault:

bash-5.1# vault operator init

Take note of the unseal keys and the initial root token. You will need to provide three of the unseal keys every time the Vault server is resealed or restarted.

Why 3 keys? Review Shamir's Secret Sharing.

Now you can unseal Vault using three of the keys:

bash-5.1# vault operator unseal
Unseal Key (will be hidden):

Run this command two more times, using different keys each time. Once done, make sure Sealed is false:

Key             Value
---             -----
Seal Type       shamir
Initialized     true
Sealed          false
Total Shares    5
Threshold       3
Version         1.8.2
Storage Type    file
Cluster Name    vault-cluster-8fcf9d05
Cluster ID      d86e0274-ad9c-d2c1-d6ec-baeab410797b
HA Enabled      false

Using the root token, you can now authenticate:

bash-5.1# vault login
Token (will be hidden):

You should see something similar to:

Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.

Key                  Value
---                  -----
token                s.c0kYHWiOTqQvtR8JuSeTz6sZ
token_accessor       3FQJVxOY5C1brzlHHQSFaCdZ
token_duration       ∞
token_renewable      false
token_policies       ["root"]
identity_policies    []
policies             ["root"]

Keep in mind that this uses the root policy. In production you'll want to set up policies with different levels of access. We'll look at how to do this shortly.

保險庫初始化

Vault is now unsealed and ready for use.

Auditing

Before we test out the functionality, let's enable an Audit Device:

bash-5.1# vault audit enable file file_path=/vault/logs/audit.log

Success! Enabled the file audit device at: file/

You should now be able to view the logs locally in "vault/logs". To test, run the following command to view all enabled Audit Devices:

bash-5.1# vault audit list

Path     Type    Description
----     ----    -----------
file/    file    n/a

The request and subsequent response should be logged in vault/logs/audit.log. Take a look.

Secrets

There are two types of secrets in Vault: static and dynamic.

Static secrets (think encrypted Redis or Memcached) have refresh intervals but they do not expire unless explicitly revoked. They are defined ahead of time with the Key/Value backend (formerly the "generic" backend) and then shared.

安全的秘密存儲

Dynamic secrets are generated on demand. They have enforced leases and generally expire after a short period of time. Since they do not exist until they are accessed, there's less exposure -- so dynamic secrets are much more secure. Vault ships with a number of dynamic backends -- i.e., AWS, Databases, Google Cloud, Consul, and RabbitMQ.

Review the Why We Need Dynamic Secrets blog post for more info on the advantages of using dynamic secrets.

Static Secrets

Vault can be managed through the CLI, HTTP API, or UI.

CLI

Still within the bash session in the container, we can create, read, update, and delete secrets. We'll also look at how to version and roll back secrets.

Enable secrets with following command:

bash-5.1# vault secrets enable kv

Success! Enabled the kv secrets engine at: kv/

Create a new secret with a key of bar and value of precious within the kv/foo path:

bash-5.1# vault kv put kv/foo bar=precious

Success! Data written to: kv/foo

Read:

bash-5.1# vault kv get kv/foo

=== Data ===
Key    Value
---    -----
bar    precious

To work with different versions of a specific key, we'll need to upgrade to v2 of the Key/Value backend:

bash-5.1# vault kv enable-versioning kv/

Success! Tuned the secrets engine at: kv/

Add version 2 by updating the value to copper:

bash-5.1# vault kv put kv/foo bar=copper

Key              Value
---              -----
created_time     2021-09-08T18:23:14.4154928Z
deletion_time    n/a
destroyed        false
version          2

Read version 1:

bash-5.1# vault kv get -version=1 kv/foo

====== Metadata ======
Key              Value
---              -----
created_time     2021-09-08T18:22:37.2548824Z
deletion_time    n/a
destroyed        false
version          1

=== Data ===
Key    Value
---    -----
bar    precious

Read version 2:

bash-5.1# vault kv get -version=2 kv/foo

====== Metadata ======
Key              Value
---              -----
created_time     2021-09-08T18:23:14.4154928Z
deletion_time    n/a
destroyed        false
version          2

=== Data ===
Key    Value
---    -----
bar    copper

Delete the latest version (e.g., version 2):

bash-5.1# vault kv delete kv/foo

Success! Data deleted (if it existed) at: kv/foo

Delete version 1:

bash-5.1# vault kv delete -versions=1 kv/foo

Success! Data deleted (if it existed) at: kv/foo

You can undelete as well:

bash-5.1# vault kv undelete -versions=1 kv/foo

Success! Data written to: kv/undelete/foo

Delete is akin to a soft delete. If you want to remove the underlying metadata, you'll have to use the destroy command:

bash-5.1# vault kv destroy -versions=1 kv/foo

Success! Data written to: kv/destroy/foo

Review v1 and v2 to view all the available commands.

Take note of the audit log. Each of the above requests were logged!

API

You can also interact with Vault via the HTTP API. We'll make requests against v2 of the API. Open a new terminal tab, and then set the root token as an environment variable:

$ export VAULT_TOKEN=your_token_goes_here

Create a new secret called foo with a value of world:

$ curl \
    -H "X-Vault-Token: $VAULT_TOKEN" \
    -H "Content-Type: application/json" \
    -X POST \
    -d '{ "data": { "foo": "world" } }' \
    http://127.0.0.1:8200/v1/kv/data/hello

Read the secret:

$ curl \
    -H "X-Vault-Token: $VAULT_TOKEN" \
    -X GET \
    http://127.0.0.1:8200/v1/kv/data/hello

The JSON response should contain a data key with a value similar to:

"data": {
  "data":{
    "foo": "world"
  },
  "metadata": {
    "created_time": "2021-09-08T18:30:32.5140484Z",
    "deletion_time": "",
    "destroyed": false,
    "version": 1
  }
}

保險庫 API

Try adding new versions, deleting, and destroying on your own.

UI

The UI should be up at running at http://localhost:8200/ui/vault. Use the root token to login. Then, explore the Key/Value backend on your own:

保險庫用戶界面

Policies

Thus far we've been using the root policy to interact with the API. Let's set up a policy that only has read access.

Add a new config file called app-policy.json to "vault/policies":

{
  "path": {
    "kv/data/app/*": {
      "policy": "read"
    }
  }
}

Create a new policy back in the bash session:

bash-5.1# vault policy write app /vault/policies/app-policy.json

Success! Uploaded policy: app

Then, create a new token:

bash-5.1# vault token create -policy=app

Key                  Value
---                  -----
token                s.ZOUMx3RIhVRhI4ijlZg8KXRQ
token_accessor       TT53xOxbIfGjI7l4392gjXcg
token_duration       768h
token_renewable      true
token_policies       ["app" "default"]
identity_policies    []
policies             ["app" "default"]

Within another new terminal tab (you should now have three), add the VAULT_TOKEN environment variable with the new token:

$ export VAULT_TOKEN=your_token_goes_here

Try to read the foo secret that we previously set:

$ curl \
    -H "X-Vault-Token: $VAULT_TOKEN" \
    -X GET \
    http://127.0.0.1:8200/v1/kv/data/hello

You should not have the correct permissions to view that secret:

{
  "errors":[
    "1 error occurred:\n\t* permission denied\n\n"
  ]
}

Why can't we even read it? Jump back to the policy config in vault-config.json. kv/data/app/* indicates that the policy can only read from the app path.

As you've probably already noticed, nearly everything in Vault is path-based.

Back within the bash session in the container, add a new secret to the app/test path:

bash-5.1# vault kv put kv/app/test ping=pong

Key              Value
---              -----
created_time     2021-09-08T18:40:35.2694047Z
deletion_time    n/a
destroyed        false
version          1

You should be able to view the secret using the token associated with the app policy:

$ curl \
    -H "X-Vault-Token: $VAULT_TOKEN" \
    -X GET \
    http://127.0.0.1:8200/v1/kv/data/app/test

Policies can be managed from the UI as well:

保險庫用戶界面

Encryption as a Service

Before we look at dynamic secrets, let's quickly review the Transit backend, which can be used as an "encryption as a service" for:

  • Encrypting and decrypting data "in-transit" without storing it inside Vault
  • Easily integrating encryption into your application workflow

Back within the bash session in the container, enable Transit:

bash-5.1# vault secrets enable transit

Success! Enabled the transit secrets engine at: transit/

Configure a named encryption key:

bash-5.1# vault write -f transit/keys/foo

Success! Data written to: transit/keys/foo

Encrypt:

bash-5.1# vault write transit/encrypt/foo plaintext=$(base64 <<< "my precious")

Key           Value
---           -----
ciphertext    vault:v1:cFnk5AQLE9Mg+mZ7Ej17vRmYT5aqheikdZQ1FC4vre5jAod0L/uHDA==

Decrypt:

bash-5.1# vault write transit/decrypt/foo ciphertext=vault:v1:cFnk5AQLE9Mg+mZ7Ej17vRmYT5aqheikdZQ1FC4vre5jAod0L/uHDA==

Key          Value
---          -----
plaintext    bXkgcHJlY2lvdXMK

Decode:

bash-5.1# base64 -d <<< "bXkgcHJlY2lvdXMK"

my precious

Test it out in the UI as well:

保險庫用戶界面

Dynamic Secrets

As mentioned, Vault supports a number of dynamic secret backends for generating secrets dynamically when needed. For example, with the AWS and Google Cloud backends, you can create access credentials based on IAM policies. The Databases backend, meanwhile, generates database credentials based on configured roles.

Dynamic Secrets:

  • are generated on demand
  • have limited access based on role
  • are leased for a period of time
  • can be revoked
  • come with an audit trail

Let's look at how to generate AWS credentials using the AWS backend.

AWS Credentials

Enable the AWS secrets backend:

bash-5.1# vault secrets enable -path=aws aws

Success! Enabled the aws secrets engine at: aws/

Authenticate:

bash-5.1# vault write aws/config/root access_key=foo secret_key=bar

Success! Data written to: aws/config/root

Make sure to replace foo and bar with your AWS access key id and secret key, respectively.

Create role:

bash-5.1# vault write aws/roles/ec2-read credential_type=iam_user policy_document=-<<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "Stmt1426528957000",
      "Effect": "Allow",
      "Action": [
        "ec2:*"
      ],
      "Resource": [
        "*"
      ]
    }
  ]
}
EOF

Success! Data written to: aws/roles/ec2-read

Here, we created a new role based on AmazonEC2ReadOnlyAccess, which is an AWS-managed policy. As the name suggests, it give users read-only access to the EC2 console; they cannot perform any actions or create new resources. You can also use an inline policy to create a custom role based on your individual needs. We'll look at an example of this shortly. Refer to the AWS Secrets Engine docs for more info.

Remember: Dynamic Secrets are generated only when they are requested (i.e., a web app requests access to S3). They are not available in the store before this.

Create a new set of credentials:

bash-5.1# vault read aws/creds/ec2-read

Key                Value
---                -----
lease_id           aws/creds/ec2-read/9KdO6J7KVBiSwOPEvwrqqALG
lease_duration     768h
lease_renewable    true
access_key         AKIAZ4DZAKZKEULSDW5A
secret_key         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
security_token     <nil>

You should now be able to see the user within the "Users" section on the IAM console on AWS:

對不起

Leases and Revocation

In this section, we'll take a quick look at how to define a custom lease period and revoke a secret before the end of that period.

Create a new AWS role:

bash-5.1# vault write aws/roles/foo credential_type=iam_user policy_document=-<<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "Stmt1426528957000",
      "Effect": "Allow",
      "Action": [
        "ec2:*"
      ],
      "Resource": [
        "*"
      ]
    }
  ]
}
EOF

Success! Data written to: aws/roles/foo

Take note of the lease_duration when you create a new AWS credential:

bash-5.1# vault read aws/creds/foo

Key                Value
---                -----
lease_id           aws/creds/foo/F0oBbnBIHEoz0ywVVtbuJB7r
lease_duration     768h
lease_renewable    true
access_key         AKIAZ4DZAKZKLJKB7CPX
secret_key         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
security_token     <nil>

What if you only wanted the lease period for all AWS IAM dynamic secrets to be 30 minutes?

bash-5.1# vault write aws/config/lease lease=1800s lease_max=1800s

In this example, since lease_max is the same as lease, you won't be able to renew the token. If you set the lease_max to 3600s, you'd be able to renew the lease once. For more, review the Tokens and Leases guide.

Create a new credential:

bash-5.1# vault read aws/creds/foo

Key                Value
---                -----
lease_id           aws/creds/foo/xQlJpKDS1ljE9Awz0aywXgbB
lease_duration     30m
lease_renewable    true
access_key         AKIAZ4DZAKZKJPL5OM5W
secret_key         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
security_token     <nil>

Want to quickly revoke this credential? Grab the lease_id and then run:

bash-5.1# vault lease revoke aws/creds/foo/xQlJpKDS1ljE9Awz0aywXgbB

Want to revoke all AWS creds?

bash-5.1# vault lease revoke -prefix aws/

Refer to the Lease, Renew, and Revoke guide for more info these concepts.

Consul Backend

Thus far, we've been using the Filesystem backend. This will not scale beyond a single server, so it does not take advantage of Vault's high availability. Fortunately, there are a number of other Storage backends, like the Consul backend, designed for distributed systems.

To set up Consul, start by updating the docker-compose.yml file:

version: '3.8'

services:

  vault:
    build:
      context: ./vault
      dockerfile: Dockerfile
    ports:
      - 8200:8200
    volumes:
      - ./vault/config:/vault/config
      - ./vault/policies:/vault/policies
      - ./vault/data:/vault/data
      - ./vault/logs:/vault/logs
    environment:
      - VAULT_ADDR=http://127.0.0.1:8200
      - VAULT_API_ADDR=http://127.0.0.1:8200
    command: server -config=/vault/config/vault-config.json
    cap_add:
      - IPC_LOCK
    depends_on:
      - consul

  consul:
    build:
      context: ./consul
      dockerfile: Dockerfile
    ports:
      - 8500:8500
    command: agent -server -bind 0.0.0.0 -client 0.0.0.0 -bootstrap-expect 1 -config-file=/consul/config/config.json
    volumes:
      - ./consul/config/consul-config.json:/consul/config/config.json
      - ./consul/data:/consul/data

Add a new directory in the project root called "consul", and then add a new Dockerfile to that newly created directory:

# base image
FROM alpine:3.14

# set consul version
ENV CONSUL_VERSION 1.10.2

# create a new directory
RUN mkdir /consul

# download dependencies
RUN apk --no-cache add \
      bash \
      ca-certificates \
      wget

# download and set up consul
RUN wget --quiet --output-document=/tmp/consul.zip https://releases.hashicorp.com/consul/${CONSUL_VERSION}/consul_${CONSUL_VERSION}_linux_amd64.zip && \
    unzip /tmp/consul.zip -d /consul && \
    rm -f /tmp/consul.zip && \
    chmod +x /consul/consul

# update PATH
ENV PATH="PATH=$PATH:$PWD/consul"

# add the config file
COPY ./config/consul-config.json /consul/config/config.json

# expose ports
EXPOSE 8300 8400 8500 8600

# run consul
ENTRYPOINT ["consul"]

Next, within the "consul" directory add two new directories: "config" and "data". Then, within "config", add a config file called consul-config.json:

{
  "datacenter": "localhost",
  "data_dir": "/consul/data",
  "log_level": "DEBUG",
  "server": true,
  "ui": true,
  "ports": {
    "dns": 53
  }
}

請務必查看 Consul 文檔中的配置選項,以獲取有關上述選項的更多信息。

“consul”目錄現在應該如下所示:

├── Dockerfile
├── config
│   └── consul-config.json
└── data

退出 bash 會話。關閉容器,然後更新 Vault 配置文件:

{
  "backend": {
    "consul": {
      "address": "consul:8500",
      "path": "vault/"
    }
  },
  "listener": {
    "tcp":{
      "address": "0.0.0.0:8200",
      "tls_disable": 1
    }
  },
  "ui": true
}

所以,現在我們使用Consul後端而不是文件系統。我們使用了服務的名稱consul,作為地址的一部分。path鍵定義了在 Consul 的鍵/值存儲中存儲 Vault 數據的路徑。

清除“vault/data”目錄中的所有文件和文件夾以刪除文件系統後端。構建新圖像並啟動容器:

$ docker-compose down
$ docker-compose up -d --build

通過在瀏覽器中導航到http://localhost:8500/ui來確保一切順利:

領事

從 CLI 或 UI 中對此進行測試。

命令行界面

在 Vault 容器中創建一個新的 bash 會話:

$ docker-compose exec vault bash

然後,運行:

# Init
bash-5.1# vault operator init

# Unseal
bash-5.1# vault operator unseal

# Authenticate
bash-5.1# vault login

# Enable secrets
bash-5.1# vault secrets enable kv

# Add a new static secret
bash-5.1# vault kv put kv/foo bar=precious

# Read it back
bash-5.1# vault kv get kv/foo

用戶界面

保險庫領事

請注意“保管庫/數據”中沒有文件或文件夾。你為什麼認為這是?

想要添加另一個 Consul 服務器嗎?向docker-compose.yml添加新服務:

consul-worker:
  build:
    context: ./consul
    dockerfile: Dockerfile
  command: agent -server -join consul -config-file=/consul/config/config.json
  volumes:
    - ./consul/config/consul-config.json:/consul/config/config.json
  depends_on:
    - consul

在這裡,我們使用join命令將此代理連接到現有集群。請注意我們如何只需要引用服務名稱:consul

然後:

  1. 退出 bash 會話(如有必要)
  2. 放下容器
  3. 清除“consul/data”中的數據目錄(為什麼?)
  4. 旋轉容器備份並測試

領事

結論

在本教程中,我們介紹瞭如何在 Docker 容器中設置和運行 Vault 和 Consul。您現在應該清楚地了解如何與 Vault 交互並執行基本操作。

vault-consul-docker存儲庫中獲取最終代碼。也請查看演示文稿

來源:  https ://testdriven.io

#vault #consul 

如何設置和使用 Hashicorp 的 Vault 和 Consul 項目
Noelia  Graham

Noelia Graham

1660618620

Gérer Les Secrets Avec Vault Et Consul

Le didacticiel suivant explique comment configurer et utiliser les projets Vault et Consul de Hashicorp pour stocker et gérer en toute sécurité des secrets.

Nous commencerons par lancer une seule instance de Vault dans un conteneur Docker, puis passerons à la gestion des secrets statiques et dynamiques avec la fonctionnalité de "chiffrement en tant que service" de Vault. Ensuite, nous ajouterons Consul au mélange et verrons comment mettre à l'échelle Vault.

Principales dépendances :

  • Docker v20.10.8
  • Docker Compose v1.29.2
  • Coffre-fort v1.8.2
  • Consul v1.10.2

Objectifs

À la fin de ce didacticiel, vous devriez être en mesure de :

  1. Expliquez ce qu'est Vault et pourquoi vous voudrez peut-être l'utiliser
  2. Décrire l'architecture de base de Vault ainsi que les secrets dynamiques et statiques, les différents backends (stockage, secret, authentification, audit) et comment Vault peut être utilisé comme "chiffrement en tant que service"
  3. Configurer et exécuter Vault et Consul avec Docker
  4. Faites tourner Vault avec le backend du système de fichiers
  5. Initialiser et desceller Vault
  6. S'authentifier auprès de Vault
  7. Configurer un backend d'audit pour consigner toutes les interactions avec Vault
  8. Travailler avec des secrets statiques et dynamiques via la CLI, l'API HTTP et l'interface utilisateur
  9. Créer une stratégie Vault pour limiter l'accès à un chemin spécifique
  10. Utilisez le backend Transit comme "chiffrement en tant que service"
  11. Configurer Consul pour qu'il fonctionne avec Vault en tant que backend de stockage pour les secrets
  12. Définir une période de bail personnalisée pour un secret et révoquer un secret avant la fin de cette période

Qu'est-ce que le coffre-fort ?

Vault est un outil open source utilisé pour stocker et gérer en toute sécurité des secrets.

Qu'est-ce qu'un secret ? Les secrets, dans le contexte de ce didacticiel, sont des informations sécurisées ou personnellement identifiables telles que les informations d'identification de la base de données, les clés SSH, les noms d'utilisateur et les mots de passe, les informations d'identification AWS IAM, les jetons API, les numéros de sécurité sociale, les numéros de carte de crédit, pour n'en nommer que quelques-uns.

Prenez un moment pour réfléchir à la manière dont votre équipe gère et distribue actuellement les secrets :

  1. Qui y a accès ?
  2. Qui les gère ?
  3. Comment contrôlez-vous qui y a accès ?
  4. Comment vos applications les obtiennent-elles ?
  5. Comment sont-ils mis à jour ?
  6. Comment sont-ils révoqués ?

Vault apporte des réponses à ces questions et aide à résoudre les problèmes suivants concernant la gestion des secrets :

ProblèmesObjectifs du coffre-fort
Les secrets sont partout.Vault est la seule source de vérité pour tous les secrets.
Ils sont généralement non cryptés.Vault gère le chiffrement (pendant le transit et au repos) prêt à l'emploi.
Il est difficile de les générer dynamiquement.Les secrets peuvent être générés dynamiquement.
Il est encore plus difficile de les louer et de les révoquer.Les secrets peuvent être loués et révoqués.
Il n'y a pas de piste d'audit.Il existe une piste d'audit pour générer et utiliser des secrets.

Vault a un certain nombre de pièces mobiles, il peut donc prendre un certain temps pour se familiariser avec l'architecture globale. Prenez un moment pour consulter le guide d' architecture , en prenant note des backends suivants :

BackendUtilisationExemples
StockageOù sont stockés les secretsConsul *, Système de fichiers *, In-Memory, PostgreSQL, S3
SecretGère les secrets statiques ou dynamiquesAWS *, Bases de données, Clé/Valeur *, RabbitMQ, SSH
AuthentificationGère l'authentification et l'autorisationAWS, Azure, Google Cloud, GitHub, jetons *, nom d'utilisateur et mot de passe
AuditEnregistre toutes les demandes et réponsesFichier *, Syslog, Socket

*utilisé dans ce tutoriel

Sur ce, commençons à utiliser Vault.

Backend du système de fichiers

Pour être opérationnel rapidement, nous utiliserons le backend du système de fichiers pour stocker les secrets au repos.

Le backend du système de fichiers ne doit être utilisé que pour le développement local ou un déploiement Vault sur un seul serveur, car il ne prend pas en charge la haute disponibilité.

Créez un nouveau répertoire de projet :

$ mkdir vault-consul-docker && cd vault-consul-docker

Ajoutez ensuite les dossiers suivants :

└── vault
    ├── config
    ├── data
    ├── logs
    └── policies

Ajoutez un Dockerfile au répertoire "vault":

# base image
FROM alpine:3.14

# set vault version
ENV VAULT_VERSION 1.8.2

# create a new directory
RUN mkdir /vault

# download dependencies
RUN apk --no-cache add \
      bash \
      ca-certificates \
      wget

# download and set up vault
RUN wget --quiet --output-document=/tmp/vault.zip https://releases.hashicorp.com/vault/${VAULT_VERSION}/vault_${VAULT_VERSION}_linux_amd64.zip && \
    unzip /tmp/vault.zip -d /vault && \
    rm -f /tmp/vault.zip && \
    chmod +x /vault

# update PATH
ENV PATH="PATH=$PATH:$PWD/vault"

# add the config file
COPY ./config/vault-config.json /vault/config/vault-config.json

# expose port 8200
EXPOSE 8200

# run vault
ENTRYPOINT ["vault"]

Ensuite, ajoutez un fichier docker-compose.yml à la racine du projet :

version: '3.8'

services:

  vault:
    build:
      context: ./vault
      dockerfile: Dockerfile
    ports:
      - 8200:8200
    volumes:
      - ./vault/config:/vault/config
      - ./vault/policies:/vault/policies
      - ./vault/data:/vault/data
      - ./vault/logs:/vault/logs
    environment:
      - VAULT_ADDR=http://127.0.0.1:8200
      - VAULT_API_ADDR=http://127.0.0.1:8200
    command: server -config=/vault/config/vault-config.json
    cap_add:
      - IPC_LOCK

Ajoutez un fichier de configuration appelé vault-config.json à "vault/config":

{
  "backend": {
    "file": {
      "path": "vault/data"
    }
  },
  "listener": {
    "tcp":{
      "address": "0.0.0.0:8200",
      "tls_disable": 1
    }
  },
  "ui": true
}

Ici, nous avons configuré Vault pour utiliser le backend du système de fichiers, défini l' écouteur pour Vault, désactivé TLS et activé l' interface utilisateur Vault . Consultez la documentation pour plus d'informations sur la configuration de Vault.

Nous pouvons maintenant créer l'image et faire tourner le conteneur :

$ docker-compose up -d --build

Extrayez les journaux Docker pour vous assurer qu'il n'y a pas d'erreurs dans la compilation :

$ docker-compose logs

Vous devriez voir quelque chose de similaire à :

Attaching to vault-consul-docker_vault_1
vault_1  | ==> Vault server configuration:
vault_1  |
vault_1  |              Api Address: http://127.0.0.1:8200
vault_1  | 2021-09-08T14:48:35.014Z [INFO]  proxy environment: http_proxy="" https_proxy="" no_proxy=""
vault_1  |                      Cgo: disabled
vault_1  |          Cluster Address: https://127.0.0.1:8201
vault_1  |               Go Version: go1.16.7
vault_1  |               Listener 1: tcp (addr: "0.0.0.0:8200", cluster address: "0.0.0.0:8201", max_request_duration: "1m30s", max_request_size: "33554432", tls: "disabled")
vault_1  |                Log Level: info
vault_1  |                    Mlock: supported: true, enabled: true
vault_1  |            Recovery Mode: false
vault_1  |                  Storage: file
vault_1  |                  Version: Vault v1.8.2
vault_1  |              Version Sha: aca76f63357041a43b49f3e8c11d67358496959f
vault_1  |
vault_1  | ==> Vault server started! Log data will stream in below:
vault_1  |

Initialisation et descellement

Démarrez une session bash dans le conteneur en cours d'exécution :

$ docker-compose exec vault bash

Dans le shell, initialisez Vault :

bash-5.1# vault operator init

Prenez note des clés de descellement et du jeton racine initial. Vous devrez fournir trois des clés de descellement chaque fois que le serveur Vault est rescellé ou redémarré.

Pourquoi 3 clés ? Passez en revue le partage secret de Shamir .

Vous pouvez désormais desceller Vault à l'aide de trois des clés :

bash-5.1# vault operator unseal
Unseal Key (will be hidden):

Exécutez cette commande deux fois de plus, en utilisant des clés différentes à chaque fois. Une fois fait, assurez Sealed- vous que false:

Key             Value
---             -----
Seal Type       shamir
Initialized     true
Sealed          false
Total Shares    5
Threshold       3
Version         1.8.2
Storage Type    file
Cluster Name    vault-cluster-8fcf9d05
Cluster ID      d86e0274-ad9c-d2c1-d6ec-baeab410797b
HA Enabled      false

À l'aide du jeton racine, vous pouvez désormais vous authentifier :

bash-5.1# vault login
Token (will be hidden):

Vous devriez voir quelque chose de similaire à :

Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.

Key                  Value
---                  -----
token                s.c0kYHWiOTqQvtR8JuSeTz6sZ
token_accessor       3FQJVxOY5C1brzlHHQSFaCdZ
token_duration       ∞
token_renewable      false
token_policies       ["root"]
identity_policies    []
policies             ["root"]

Gardez à l'esprit que cela utilise la politique racine. En production, vous souhaiterez configurer des stratégies avec différents niveaux d'accès. Nous verrons comment procéder sous peu.

initialisation du coffre-fort

Vault est maintenant descellé et prêt à l'emploi.

Audit

Avant de tester la fonctionnalité, activons un périphérique d'audit :

bash-5.1# vault audit enable file file_path=/vault/logs/audit.log

Success! Enabled the file audit device at: file/

Vous devriez maintenant pouvoir afficher les journaux localement dans "vault/logs". Pour tester, exécutez la commande suivante pour afficher tous les appareils d'audit activés :

bash-5.1# vault audit list

Path     Type    Description
----     ----    -----------
file/    file    n/a

La demande et la réponse qui s'ensuit doivent être consignées dans le coffre-fort/logs/audit.log . Regarde.

secrets

There are two types of secrets in Vault: static and dynamic.

Static secrets (think encrypted Redis or Memcached) have refresh intervals but they do not expire unless explicitly revoked. They are defined ahead of time with the Key/Value backend (formerly the "generic" backend) and then shared.

stockage secret sécurisé

Les secrets dynamiques sont générés à la demande. Ils ont fait respecter les baux et expirent généralement après une courte période de temps. Puisqu'ils n'existent pas jusqu'à ce qu'ils soient accessibles, il y a moins d'exposition - donc les secrets dynamiques sont beaucoup plus sécurisés. Vault est livré avec un certain nombre de backends dynamiques, à savoir AWS , Databases , Google Cloud , Consul et RabbitMQ .

Consultez l' article de blog Why We Need Dynamic Secrets pour plus d'informations sur les avantages de l'utilisation des secrets dynamiques.

Secrets statiques

Vault peut être géré via l' interface de ligne de commande , l' API HTTP ou l' interface utilisateur .

CLI

Toujours dans la session bash du conteneur, nous pouvons créer, lire, mettre à jour et supprimer des secrets. Nous verrons également comment versionner et restaurer les secrets.

Activez les secrets avec la commande suivante :

bash-5.1# vault secrets enable kv

Success! Enabled the kv secrets engine at: kv/

Créez un nouveau secret avec une clé baret une valeur preciousdans le kv/foochemin :

bash-5.1# vault kv put kv/foo bar=precious

Success! Data written to: kv/foo

Lis:

bash-5.1# vault kv get kv/foo

=== Data ===
Key    Value
---    -----
bar    precious

Pour travailler avec différentes versions d'une clé spécifique, nous devrons effectuer une mise à niveau vers la v2 du backend clé/valeur :

bash-5.1# vault kv enable-versioning kv/

Success! Tuned the secrets engine at: kv/

Ajoutez la version 2 en mettant à jour la valeurcopper :

bash-5.1# vault kv put kv/foo bar=copper

Key              Value
---              -----
created_time     2021-09-08T18:23:14.4154928Z
deletion_time    n/a
destroyed        false
version          2

Lire la version 1 :

bash-5.1# vault kv get -version=1 kv/foo

====== Metadata ======
Key              Value
---              -----
created_time     2021-09-08T18:22:37.2548824Z
deletion_time    n/a
destroyed        false
version          1

=== Data ===
Key    Value
---    -----
bar    precious

Lire la version 2 :

bash-5.1# vault kv get -version=2 kv/foo

====== Metadata ======
Key              Value
---              -----
created_time     2021-09-08T18:23:14.4154928Z
deletion_time    n/a
destroyed        false
version          2

=== Data ===
Key    Value
---    -----
bar    copper

Supprimez la dernière version (par exemple, la version 2) :

bash-5.1# vault kv delete kv/foo

Success! Data deleted (if it existed) at: kv/foo

Supprimer la version 1 :

bash-5.1# vault kv delete -versions=1 kv/foo

Success! Data deleted (if it existed) at: kv/foo

Vous pouvez également restaurer :

bash-5.1# vault kv undelete -versions=1 kv/foo

Success! Data written to: kv/undelete/foo

La suppression s'apparente à une suppression douce. Si vous souhaitez supprimer les métadonnées sous-jacentes, vous devrez utiliser la commande destroy :

bash-5.1# vault kv destroy -versions=1 kv/foo

Success! Data written to: kv/destroy/foo

Passez en revue v1 et v2 pour afficher toutes les commandes disponibles.

Prenez note du journal d'audit. Chacune des demandes ci-dessus a été enregistrée !

API

Vous pouvez également interagir avec Vault via l' API HTTP . Nous ferons des requêtes contre la v2 de l'API. Ouvrez un nouvel onglet de terminal, puis définissez le jeton racine en tant que variable d'environnement :

$ export VAULT_TOKEN=your_token_goes_here

Créez un nouveau secret appelé fooavec une valeur deworld :

$ curl \
    -H "X-Vault-Token: $VAULT_TOKEN" \
    -H "Content-Type: application/json" \
    -X POST \
    -d '{ "data": { "foo": "world" } }' \
    http://127.0.0.1:8200/v1/kv/data/hello

Lisez le secret :

$ curl \
    -H "X-Vault-Token: $VAULT_TOKEN" \
    -X GET \
    http://127.0.0.1:8200/v1/kv/data/hello

La réponse JSON doit contenir une dataclé avec une valeur similaire à :

"data": {
  "data":{
    "foo": "world"
  },
  "metadata": {
    "created_time": "2021-09-08T18:30:32.5140484Z",
    "deletion_time": "",
    "destroyed": false,
    "version": 1
  }
}

API de coffre-fort

Essayez d'ajouter de nouvelles versions, de supprimer et de détruire vous-même.

interface utilisateur

L' interface utilisateur doit être opérationnelle sur http://localhost:8200/ui/vault . Utilisez le jeton racine pour vous connecter. Ensuite, explorez le backend clé/valeur par vous-même :

interface utilisateur du coffre-fort

Stratégies

Jusqu'à présent, nous avons utilisé la politique racine pour interagir avec l'API. Configurons une stratégie qui n'a qu'un accès en lecture.

Ajoutez un nouveau fichier de configuration appelé app-policy.json à "vault/policies":

{
  "path": {
    "kv/data/app/*": {
      "policy": "read"
    }
  }
}

Créez une nouvelle stratégie dans la session bash :

bash-5.1# vault policy write app /vault/policies/app-policy.json

Success! Uploaded policy: app

Ensuite, créez un nouveau jeton :

bash-5.1# vault token create -policy=app

Key                  Value
---                  -----
token                s.ZOUMx3RIhVRhI4ijlZg8KXRQ
token_accessor       TT53xOxbIfGjI7l4392gjXcg
token_duration       768h
token_renewable      true
token_policies       ["app" "default"]
identity_policies    []
policies             ["app" "default"]

Dans un autre nouvel onglet de terminal (vous devriez maintenant en avoir trois), ajoutez la VAULT_TOKENvariable d'environnement avec le nouveau jeton :

$ export VAULT_TOKEN=your_token_goes_here

Essayez de lire le foosecret que nous avons défini précédemment :

$ curl \
    -H "X-Vault-Token: $VAULT_TOKEN" \
    -X GET \
    http://127.0.0.1:8200/v1/kv/data/hello

Vous ne devez pas disposer des autorisations appropriées pour afficher ce secret :

{
  "errors":[
    "1 error occurred:\n\t* permission denied\n\n"
  ]
}

Pourquoi ne pouvons-nous même pas le lire ? Revenez à la configuration de la stratégie dans le coffre-fort-config.json . kv/data/app/*indique que la stratégie ne peut lire qu'à partir du appchemin.

Comme vous l'avez probablement déjà remarqué, presque tout dans Vault est basé sur un chemin.

De retour dans la session bash du conteneur, ajoutez un nouveau secret au app/testchemin :

bash-5.1# vault kv put kv/app/test ping=pong

Key              Value
---              -----
created_time     2021-09-08T18:40:35.2694047Z
deletion_time    n/a
destroyed        false
version          1

Vous devriez pouvoir afficher le secret à l'aide du jeton associé à la appstratégie :

$ curl \
    -H "X-Vault-Token: $VAULT_TOKEN" \
    -X GET \
    http://127.0.0.1:8200/v1/kv/data/app/test

Les stratégies peuvent également être gérées à partir de l'interface utilisateur :

interface utilisateur du coffre-fort

Chiffrement en tant que service

Avant d'aborder les secrets dynamiques, passons rapidement en revue le backend Transit , qui peut être utilisé comme un "chiffrement en tant que service" pour :

  • Chiffrement et déchiffrement des données "en transit" sans les stocker dans Vault
  • Intégrez facilement le chiffrement dans le flux de travail de votre application

De retour dans la session bash du conteneur, activez Transit :

bash-5.1# vault secrets enable transit

Success! Enabled the transit secrets engine at: transit/

Configurez une clé de chiffrement nommée :

bash-5.1# vault write -f transit/keys/foo

Success! Data written to: transit/keys/foo

Crypter:

bash-5.1# vault write transit/encrypt/foo plaintext=$(base64 <<< "my precious")

Key           Value
---           -----
ciphertext    vault:v1:cFnk5AQLE9Mg+mZ7Ej17vRmYT5aqheikdZQ1FC4vre5jAod0L/uHDA==

Décrypter :

bash-5.1# vault write transit/decrypt/foo ciphertext=vault:v1:cFnk5AQLE9Mg+mZ7Ej17vRmYT5aqheikdZQ1FC4vre5jAod0L/uHDA==

Key          Value
---          -----
plaintext    bXkgcHJlY2lvdXMK

Décoder:

bash-5.1# base64 -d <<< "bXkgcHJlY2lvdXMK"

my precious

Testez-le également dans l'interface utilisateur :

interface utilisateur du coffre-fort

Secrets dynamiques

Comme mentionné, Vault prend en charge un certain nombre de backends secrets dynamiques pour générer des secrets de manière dynamique en cas de besoin. Par exemple, avec les backends AWS et Google Cloud , vous pouvez créer des identifiants d'accès basés sur des stratégies IAM. Le backend des bases de données , quant à lui, génère des informations d'identification de base de données en fonction des rôles configurés.

Secrets dynamiques :

  • sont générés à la demande
  • avoir un accès limité en fonction du rôle
  • sont loués pour une durée
  • peut être révoqué
  • venir avec une piste d'audit

Voyons comment générer des informations d'identification AWS à l'aide du backend AWS.

Identifiants AWS

Activez le backend des secrets AWS :

bash-5.1# vault secrets enable -path=aws aws

Success! Enabled the aws secrets engine at: aws/

Authentifier:

bash-5.1# vault write aws/config/root access_key=foo secret_key=bar

Success! Data written to: aws/config/root

Assurez-vous de remplacer fooet barpar votre ID de clé d'accès AWS et votre clé secrète, respectivement.

Créer un rôle :

bash-5.1# vault write aws/roles/ec2-read credential_type=iam_user policy_document=-<<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "Stmt1426528957000",
      "Effect": "Allow",
      "Action": [
        "ec2:*"
      ],
      "Resource": [
        "*"
      ]
    }
  ]
}
EOF

Success! Data written to: aws/roles/ec2-read

Ici, nous avons créé un nouveau rôle basé sur , qui est une stratégieAmazonEC2ReadOnlyAccess gérée par AWS . Comme son nom l'indique, il donne aux utilisateurs un accès en lecture seule à la console EC2 ; ils ne peuvent effectuer aucune action ou créer de nouvelles ressources. Vous pouvez également utiliser une stratégie en ligne pour créer un rôle personnalisé en fonction de vos besoins individuels. Nous en verrons un exemple sous peu. Reportez-vous à la documentation AWS Secrets Engine pour plus d'informations.

N'oubliez pas : les secrets dynamiques sont générés uniquement lorsqu'ils sont demandés (par exemple, une application Web demande l'accès à S3). Ils ne sont pas disponibles dans le magasin avant cela.

Créez un nouvel ensemble d'identifiants :

bash-5.1# vault read aws/creds/ec2-read

Key                Value
---                -----
lease_id           aws/creds/ec2-read/9KdO6J7KVBiSwOPEvwrqqALG
lease_duration     768h
lease_renewable    true
access_key         AKIAZ4DZAKZKEULSDW5A
secret_key         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
security_token     <nil>

Vous devriez maintenant pouvoir voir l'utilisateur dans la section « Utilisateurs » de la console IAM sur AWS :

Je suis désolé

Baux et révocation

Dans cette section, nous verrons rapidement comment définir une période de bail personnalisée et révoquer un secret avant la fin de cette période.

Créez un nouveau rôle AWS :

bash-5.1# vault write aws/roles/foo credential_type=iam_user policy_document=-<<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "Stmt1426528957000",
      "Effect": "Allow",
      "Action": [
        "ec2:*"
      ],
      "Resource": [
        "*"
      ]
    }
  ]
}
EOF

Success! Data written to: aws/roles/foo

Prenez note du lease_durationlorsque vous créez un nouvel identifiant AWS :

bash-5.1# vault read aws/creds/foo

Key                Value
---                -----
lease_id           aws/creds/foo/F0oBbnBIHEoz0ywVVtbuJB7r
lease_duration     768h
lease_renewable    true
access_key         AKIAZ4DZAKZKLJKB7CPX
secret_key         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
security_token     <nil>

Et si vous vouliez que la période de bail pour tous les secrets dynamiques AWS IAM soit de 30 minutes ?

bash-5.1# vault write aws/config/lease lease=1800s lease_max=1800s

Dans cet exemple, étant donné que lease_maxest identique à lease, vous ne pourrez pas renouveler le jeton. Si vous définissez le lease_maxsur 3600s, vous pourrez renouveler le bail une fois. Pour en savoir plus, consultez le guide Tokens and Leases .

Créez un nouvel identifiant :

bash-5.1# vault read aws/creds/foo

Key                Value
---                -----
lease_id           aws/creds/foo/xQlJpKDS1ljE9Awz0aywXgbB
lease_duration     30m
lease_renewable    true
access_key         AKIAZ4DZAKZKJPL5OM5W
secret_key         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
security_token     <nil>

Vous souhaitez révoquer rapidement cet identifiant ? Saisissez le lease_idpuis exécutez :

bash-5.1# vault lease revoke aws/creds/foo/xQlJpKDS1ljE9Awz0aywXgbB

Vous souhaitez révoquer tous les identifiants AWS ?

bash-5.1# vault lease revoke -prefix aws/

Reportez-vous au guide Bail, renouvellement et révocation pour plus d'informations sur ces concepts.

Back-end du consul

Jusqu'à présent, nous avons utilisé le backend du système de fichiers . Cela ne s'étendra pas au-delà d'un seul serveur, il ne tirera donc pas parti de la haute disponibilité de Vault. Heureusement, il existe un certain nombre d'autres backends de stockage , comme le backend Consul , conçu pour les systèmes distribués.

Pour configurer Consul , commencez par mettre à jour le fichier docker-compose.yml :

version: '3.8'

services:

  vault:
    build:
      context: ./vault
      dockerfile: Dockerfile
    ports:
      - 8200:8200
    volumes:
      - ./vault/config:/vault/config
      - ./vault/policies:/vault/policies
      - ./vault/data:/vault/data
      - ./vault/logs:/vault/logs
    environment:
      - VAULT_ADDR=http://127.0.0.1:8200
      - VAULT_API_ADDR=http://127.0.0.1:8200
    command: server -config=/vault/config/vault-config.json
    cap_add:
      - IPC_LOCK
    depends_on:
      - consul

  consul:
    build:
      context: ./consul
      dockerfile: Dockerfile
    ports:
      - 8500:8500
    command: agent -server -bind 0.0.0.0 -client 0.0.0.0 -bootstrap-expect 1 -config-file=/consul/config/config.json
    volumes:
      - ./consul/config/consul-config.json:/consul/config/config.json
      - ./consul/data:/consul/data

Ajoutez un nouveau répertoire à la racine du projet appelé "consul", puis ajoutez un nouveau Dockerfile à ce répertoire nouvellement créé :

# base image
FROM alpine:3.14

# set consul version
ENV CONSUL_VERSION 1.10.2

# create a new directory
RUN mkdir /consul

# download dependencies
RUN apk --no-cache add \
      bash \
      ca-certificates \
      wget

# download and set up consul
RUN wget --quiet --output-document=/tmp/consul.zip https://releases.hashicorp.com/consul/${CONSUL_VERSION}/consul_${CONSUL_VERSION}_linux_amd64.zip && \
    unzip /tmp/consul.zip -d /consul && \
    rm -f /tmp/consul.zip && \
    chmod +x /consul/consul

# update PATH
ENV PATH="PATH=$PATH:$PWD/consul"

# add the config file
COPY ./config/consul-config.json /consul/config/config.json

# expose ports
EXPOSE 8300 8400 8500 8600

# run consul
ENTRYPOINT ["consul"]

Ensuite, dans le répertoire "consul", ajoutez deux nouveaux répertoires : "config" et "data". Ensuite, dans "config", ajoutez un fichier de configuration appelé consul-config.json :

{
  "datacenter": "localhost",
  "data_dir": "/consul/data",
  "log_level": "DEBUG",
  "server": true,
  "ui": true,
  "ports": {
    "dns": 53
  }
}

Assurez-vous de consulter les options de configuration de la documentation Consul pour plus d'informations sur les options ci-dessus.

Le répertoire "consul" devrait maintenant ressembler à :

├── Dockerfile
├── config
│   └── consul-config.json
└── data

Quittez la session bash. Arrêtez le conteneur, puis mettez à jour le fichier de configuration Vault :

{
  "backend": {
    "consul": {
      "address": "consul:8500",
      "path": "vault/"
    }
  },
  "listener": {
    "tcp":{
      "address": "0.0.0.0:8200",
      "tls_disable": 1
    }
  },
  "ui": true
}

Donc, nous utilisons maintenant le backend Consul au lieu du système de fichiers. Nous avons utilisé le nom du service, consul, dans le cadre de l'adresse. La pathclé définit le chemin dans le magasin clé/valeur de Consul où les données du coffre-fort seront stockées.

Effacez tous les fichiers et dossiers du répertoire "vault/data" pour supprimer le backend du système de fichiers. Créez les nouvelles images et faites tourner les conteneurs :

$ docker-compose down
$ docker-compose up -d --build

Assurez-vous que tout va bien en naviguant dans votre navigateur vers http://localhost:8500/ui :

interface utilisateur du consul

Testez ceci à partir de la CLI ou de l'interface utilisateur.

CLI

Créez une session bash dans le conteneur Vault :

$ docker-compose exec vault bash

Ensuite, lancez :

# Init
bash-5.1# vault operator init

# Unseal
bash-5.1# vault operator unseal

# Authenticate
bash-5.1# vault login

# Enable secrets
bash-5.1# vault secrets enable kv

# Add a new static secret
bash-5.1# vault kv put kv/foo bar=precious

# Read it back
bash-5.1# vault kv get kv/foo

interface utilisateur

consul du coffre-fort

Remarquez qu'il n'y a pas de fichiers ou de dossiers dans "vault/data". Pourquoi pensez-vous cela est?

Vous voulez ajouter un autre serveur Consul dans le mix ? Ajoutez un nouveau service à docker-compose.yml :

consul-worker:
  build:
    context: ./consul
    dockerfile: Dockerfile
  command: agent -server -join consul -config-file=/consul/config/config.json
  volumes:
    - ./consul/config/consul-config.json:/consul/config/config.json
  depends_on:
    - consul

Ici, nous avons utilisé la commande join pour connecter cet agent à un cluster existant. Remarquez comment nous devions simplement référencer le nom du service : consul.

Alors:

  1. Quitter la session bash (si nécessaire)
  2. Descendre les conteneurs
  3. Vider le répertoire data dans "consul/data" (Pourquoi ?)
  4. Faites tourner les conteneurs et testez

interface utilisateur du consul

Conclusion

Dans ce didacticiel, nous avons expliqué comment configurer et exécuter Vault et Consul dans un conteneur Docker. Vous devriez maintenant avoir une compréhension claire de la façon d'interagir avec Vault et d'effectuer des opérations de base.

Récupérez le code final dans le référentiel de coffre-fort-consul-docker . Découvrez également la présentation .

Source :  https://testdrive.io

#vault #consul 

Gérer Les Secrets Avec Vault Et Consul

Cómo Configurar Y Utilizar Los Proyectos Vault Y Consul De Hashicorp

El siguiente tutorial detalla cómo configurar y usar los proyectos Vault y Consul de Hashicorp para almacenar y administrar secretos de forma segura.

Comenzaremos activando una sola instancia de Vault dentro de un contenedor Docker y luego pasaremos a administrar secretos estáticos y dinámicos junto con la función de "cifrado como servicio" de Vault. Luego, agregaremos Consul a la mezcla y veremos cómo escalar Vault.

Dependencias principales:

  • Ventana acoplable v20.10.8
  • Docker-Componer v1.29.2
  • Bóveda v1.8.2
  • Cónsul v1.10.2

Objetivos

Al final de este tutorial, debería ser capaz de:

  1. Explicar qué es Vault y por qué es posible que desee usarlo
  2. Describir la arquitectura básica de Vault junto con los secretos dinámicos y estáticos, los diversos backends (almacenamiento, secreto, autenticación, auditoría) y cómo se puede usar Vault como un "cifrado como servicio".
  3. Configurar y ejecutar Vault y Consul con Docker
  4. Haga girar Vault con el backend del sistema de archivos
  5. Iniciar y desbloquear Vault
  6. Autenticar contra Vault
  7. Configure un backend de auditoría para registrar todas las interacciones con Vault
  8. Trabaje con secretos estáticos y dinámicos a través de la CLI, la API HTTP y la interfaz de usuario
  9. Cree una política de Vault para limitar el acceso a una ruta específica
  10. Use el backend de Transit como un "cifrado como servicio"
  11. Configure Consul para trabajar con Vault como backend de almacenamiento para secretos
  12. Definir un período de concesión personalizado para un secreto y revocar un secreto antes del final de ese período

¿Qué es la bóveda?

Vault es una herramienta de código abierto que se utiliza para almacenar y administrar secretos de forma segura.

What is a secret? Secrets, in the context of this tutorial, are securely-sensitive or personally identifiable info like database credentials, SSH keys, usernames and passwords, AWS IAM credentials, API tokens, Social Security Numbers, credit card numbers, just to name a few.

Take a moment to think about how your team currently manages and distributes secrets:

  1. Who has access to them?
  2. Who manages them?
  3. How do you control who has access to them?
  4. How do your apps get them?
  5. How are they updated?
  6. How are they revoked?

Vault provides answers to those questions and helps to solve the following problems with regard to secret management:

ProblemsVault's Goals
Secrets are everywhere.Vault is the single source of truth for all secrets.
They are generally unencrypted.Vault manages encryption (during transit and at rest) out of the box.
It's difficult to dynamically generate them.Secrets can be dynamically generated.
It's even more difficult to lease and revoke them.Secrets can be leased and revoked.
There's no audit trail.There's an audit trail for generating and using secrets.

Vault has a number of moving pieces so it can take some time to get up to speed with the overall architecture. Take a moment to review the Architecture guide, taking note of the following backends:

BackendUseExamples
StorageWhere secrets are storedConsul*, Filesystem*, In-Memory, PostgreSQL, S3
SecretHandles static or dynamic secretsAWS*, Databases, Key/Value*, RabbitMQ, SSH
AuthHandles authentication and authorizationAWS, Azure, Google Cloud, GitHub, Tokens*, Username & Password
AuditLogs all requests and responsesFile*, Syslog, Socket

* used in this tutorial

With that, let's start using Vault.

Filesystem Backend

To get up and running quickly, we'll use the Filesystem backend to store secrets at rest.

The filesystem backend should only be used for local development or a single-server Vault deployment since it does not support high availability.

Create a new project directory:

$ mkdir vault-consul-docker && cd vault-consul-docker

Then add the following folders:

└── vault
    ├── config
    ├── data
    ├── logs
    └── policies

Add a Dockerfile to the "vault" directory:

# base image
FROM alpine:3.14

# set vault version
ENV VAULT_VERSION 1.8.2

# create a new directory
RUN mkdir /vault

# download dependencies
RUN apk --no-cache add \
      bash \
      ca-certificates \
      wget

# download and set up vault
RUN wget --quiet --output-document=/tmp/vault.zip https://releases.hashicorp.com/vault/${VAULT_VERSION}/vault_${VAULT_VERSION}_linux_amd64.zip && \
    unzip /tmp/vault.zip -d /vault && \
    rm -f /tmp/vault.zip && \
    chmod +x /vault

# update PATH
ENV PATH="PATH=$PATH:$PWD/vault"

# add the config file
COPY ./config/vault-config.json /vault/config/vault-config.json

# expose port 8200
EXPOSE 8200

# run vault
ENTRYPOINT ["vault"]

Next, add a docker-compose.yml file to the project root:

version: '3.8'

services:

  vault:
    build:
      context: ./vault
      dockerfile: Dockerfile
    ports:
      - 8200:8200
    volumes:
      - ./vault/config:/vault/config
      - ./vault/policies:/vault/policies
      - ./vault/data:/vault/data
      - ./vault/logs:/vault/logs
    environment:
      - VAULT_ADDR=http://127.0.0.1:8200
      - VAULT_API_ADDR=http://127.0.0.1:8200
    command: server -config=/vault/config/vault-config.json
    cap_add:
      - IPC_LOCK

Add a config file called vault-config.json to "vault/config":

{
  "backend": {
    "file": {
      "path": "vault/data"
    }
  },
  "listener": {
    "tcp":{
      "address": "0.0.0.0:8200",
      "tls_disable": 1
    }
  },
  "ui": true
}

Here, we configured Vault to use the Filesystem backend, defined the listener for Vault, disabled TLS, and enabled the Vault UI. Review the docs for more info on configuring Vault.

Now we can build the image and spin up the container:

$ docker-compose up -d --build

Pull up the Docker logs to make sure there were no errors in the build:

$ docker-compose logs

You should see something similar to:

Attaching to vault-consul-docker_vault_1
vault_1  | ==> Vault server configuration:
vault_1  |
vault_1  |              Api Address: http://127.0.0.1:8200
vault_1  | 2021-09-08T14:48:35.014Z [INFO]  proxy environment: http_proxy="" https_proxy="" no_proxy=""
vault_1  |                      Cgo: disabled
vault_1  |          Cluster Address: https://127.0.0.1:8201
vault_1  |               Go Version: go1.16.7
vault_1  |               Listener 1: tcp (addr: "0.0.0.0:8200", cluster address: "0.0.0.0:8201", max_request_duration: "1m30s", max_request_size: "33554432", tls: "disabled")
vault_1  |                Log Level: info
vault_1  |                    Mlock: supported: true, enabled: true
vault_1  |            Recovery Mode: false
vault_1  |                  Storage: file
vault_1  |                  Version: Vault v1.8.2
vault_1  |              Version Sha: aca76f63357041a43b49f3e8c11d67358496959f
vault_1  |
vault_1  | ==> Vault server started! Log data will stream in below:
vault_1  |

Initializing and Unsealing

Start a bash session within the running container:

$ docker-compose exec vault bash

Within the shell, initialize Vault:

bash-5.1# vault operator init

Take note of the unseal keys and the initial root token. You will need to provide three of the unseal keys every time the Vault server is resealed or restarted.

Why 3 keys? Review Shamir's Secret Sharing.

Now you can unseal Vault using three of the keys:

bash-5.1# vault operator unseal
Unseal Key (will be hidden):

Run this command two more times, using different keys each time. Once done, make sure Sealed is false:

Key             Value
---             -----
Seal Type       shamir
Initialized     true
Sealed          false
Total Shares    5
Threshold       3
Version         1.8.2
Storage Type    file
Cluster Name    vault-cluster-8fcf9d05
Cluster ID      d86e0274-ad9c-d2c1-d6ec-baeab410797b
HA Enabled      false

Using the root token, you can now authenticate:

bash-5.1# vault login
Token (will be hidden):

You should see something similar to:

Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.

Key                  Value
---                  -----
token                s.c0kYHWiOTqQvtR8JuSeTz6sZ
token_accessor       3FQJVxOY5C1brzlHHQSFaCdZ
token_duration       ∞
token_renewable      false
token_policies       ["root"]
identity_policies    []
policies             ["root"]

Keep in mind that this uses the root policy. In production you'll want to set up policies with different levels of access. We'll look at how to do this shortly.

inicio de bóveda

Vault is now unsealed and ready for use.

Auditing

Before we test out the functionality, let's enable an Audit Device:

bash-5.1# vault audit enable file file_path=/vault/logs/audit.log

Success! Enabled the file audit device at: file/

You should now be able to view the logs locally in "vault/logs". To test, run the following command to view all enabled Audit Devices:

bash-5.1# vault audit list

Path     Type    Description
----     ----    -----------
file/    file    n/a

The request and subsequent response should be logged in vault/logs/audit.log. Take a look.

Secrets

There are two types of secrets in Vault: static and dynamic.

Static secrets (think encrypted Redis or Memcached) have refresh intervals but they do not expire unless explicitly revoked. They are defined ahead of time with the Key/Value backend (formerly the "generic" backend) and then shared.

almacenamiento secreto seguro

Dynamic secrets are generated on demand. They have enforced leases and generally expire after a short period of time. Since they do not exist until they are accessed, there's less exposure -- so dynamic secrets are much more secure. Vault ships with a number of dynamic backends -- i.e., AWS, Databases, Google Cloud, Consul, and RabbitMQ.

Review the Why We Need Dynamic Secrets blog post for more info on the advantages of using dynamic secrets.

Static Secrets

Vault can be managed through the CLI, HTTP API, or UI.

CLI

Still within the bash session in the container, we can create, read, update, and delete secrets. We'll also look at how to version and roll back secrets.

Enable secrets with following command:

bash-5.1# vault secrets enable kv

Success! Enabled the kv secrets engine at: kv/

Create a new secret with a key of bar and value of precious within the kv/foo path:

bash-5.1# vault kv put kv/foo bar=precious

Success! Data written to: kv/foo

Read:

bash-5.1# vault kv get kv/foo

=== Data ===
Key    Value
---    -----
bar    precious

To work with different versions of a specific key, we'll need to upgrade to v2 of the Key/Value backend:

bash-5.1# vault kv enable-versioning kv/

Success! Tuned the secrets engine at: kv/

Add version 2 by updating the value to copper:

bash-5.1# vault kv put kv/foo bar=copper

Key              Value
---              -----
created_time     2021-09-08T18:23:14.4154928Z
deletion_time    n/a
destroyed        false
version          2

Read version 1:

bash-5.1# vault kv get -version=1 kv/foo

====== Metadata ======
Key              Value
---              -----
created_time     2021-09-08T18:22:37.2548824Z
deletion_time    n/a
destroyed        false
version          1

=== Data ===
Key    Value
---    -----
bar    precious

Read version 2:

bash-5.1# vault kv get -version=2 kv/foo

====== Metadata ======
Key              Value
---              -----
created_time     2021-09-08T18:23:14.4154928Z
deletion_time    n/a
destroyed        false
version          2

=== Data ===
Key    Value
---    -----
bar    copper

Delete the latest version (e.g., version 2):

bash-5.1# vault kv delete kv/foo

Success! Data deleted (if it existed) at: kv/foo

Delete version 1:

bash-5.1# vault kv delete -versions=1 kv/foo

Success! Data deleted (if it existed) at: kv/foo

You can undelete as well:

bash-5.1# vault kv undelete -versions=1 kv/foo

Success! Data written to: kv/undelete/foo

Delete is akin to a soft delete. If you want to remove the underlying metadata, you'll have to use the destroy command:

bash-5.1# vault kv destroy -versions=1 kv/foo

Success! Data written to: kv/destroy/foo

Review v1 and v2 to view all the available commands.

Take note of the audit log. Each of the above requests were logged!

API

You can also interact with Vault via the HTTP API. We'll make requests against v2 of the API. Open a new terminal tab, and then set the root token as an environment variable:

$ export VAULT_TOKEN=your_token_goes_here

Create a new secret called foo with a value of world:

$ curl \
    -H "X-Vault-Token: $VAULT_TOKEN" \
    -H "Content-Type: application/json" \
    -X POST \
    -d '{ "data": { "foo": "world" } }' \
    http://127.0.0.1:8200/v1/kv/data/hello

Read the secret:

$ curl \
    -H "X-Vault-Token: $VAULT_TOKEN" \
    -X GET \
    http://127.0.0.1:8200/v1/kv/data/hello

The JSON response should contain a data key with a value similar to:

"data": {
  "data":{
    "foo": "world"
  },
  "metadata": {
    "created_time": "2021-09-08T18:30:32.5140484Z",
    "deletion_time": "",
    "destroyed": false,
    "version": 1
  }
}

API de bóveda

Try adding new versions, deleting, and destroying on your own.

UI

The UI should be up at running at http://localhost:8200/ui/vault. Use the root token to login. Then, explore the Key/Value backend on your own:

interfaz de usuario de la bóveda

Policies

Thus far we've been using the root policy to interact with the API. Let's set up a policy that only has read access.

Add a new config file called app-policy.json to "vault/policies":

{
  "path": {
    "kv/data/app/*": {
      "policy": "read"
    }
  }
}

Create a new policy back in the bash session:

bash-5.1# vault policy write app /vault/policies/app-policy.json

Success! Uploaded policy: app

Then, create a new token:

bash-5.1# vault token create -policy=app

Key                  Value
---                  -----
token                s.ZOUMx3RIhVRhI4ijlZg8KXRQ
token_accessor       TT53xOxbIfGjI7l4392gjXcg
token_duration       768h
token_renewable      true
token_policies       ["app" "default"]
identity_policies    []
policies             ["app" "default"]

Within another new terminal tab (you should now have three), add the VAULT_TOKEN environment variable with the new token:

$ export VAULT_TOKEN=your_token_goes_here

Try to read the foo secret that we previously set:

$ curl \
    -H "X-Vault-Token: $VAULT_TOKEN" \
    -X GET \
    http://127.0.0.1:8200/v1/kv/data/hello

You should not have the correct permissions to view that secret:

{
  "errors":[
    "1 error occurred:\n\t* permission denied\n\n"
  ]
}

Why can't we even read it? Jump back to the policy config in vault-config.json. kv/data/app/* indicates that the policy can only read from the app path.

As you've probably already noticed, nearly everything in Vault is path-based.

Back within the bash session in the container, add a new secret to the app/test path:

bash-5.1# vault kv put kv/app/test ping=pong

Key              Value
---              -----
created_time     2021-09-08T18:40:35.2694047Z
deletion_time    n/a
destroyed        false
version          1

You should be able to view the secret using the token associated with the app policy:

$ curl \
    -H "X-Vault-Token: $VAULT_TOKEN" \
    -X GET \
    http://127.0.0.1:8200/v1/kv/data/app/test

Policies can be managed from the UI as well:

interfaz de usuario de la bóveda

Encryption as a Service

Before we look at dynamic secrets, let's quickly review the Transit backend, which can be used as an "encryption as a service" for:

  • Encrypting and decrypting data "in-transit" without storing it inside Vault
  • Easily integrating encryption into your application workflow

Back within the bash session in the container, enable Transit:

bash-5.1# vault secrets enable transit

Success! Enabled the transit secrets engine at: transit/

Configure a named encryption key:

bash-5.1# vault write -f transit/keys/foo

Success! Data written to: transit/keys/foo

Encrypt:

bash-5.1# vault write transit/encrypt/foo plaintext=$(base64 <<< "my precious")

Key           Value
---           -----
ciphertext    vault:v1:cFnk5AQLE9Mg+mZ7Ej17vRmYT5aqheikdZQ1FC4vre5jAod0L/uHDA==

Decrypt:

bash-5.1# vault write transit/decrypt/foo ciphertext=vault:v1:cFnk5AQLE9Mg+mZ7Ej17vRmYT5aqheikdZQ1FC4vre5jAod0L/uHDA==

Key          Value
---          -----
plaintext    bXkgcHJlY2lvdXMK

Decode:

bash-5.1# base64 -d <<< "bXkgcHJlY2lvdXMK"

my precious

Test it out in the UI as well:

interfaz de usuario de la bóveda

Dynamic Secrets

As mentioned, Vault supports a number of dynamic secret backends for generating secrets dynamically when needed. For example, with the AWS and Google Cloud backends, you can create access credentials based on IAM policies. The Databases backend, meanwhile, generates database credentials based on configured roles.

Dynamic Secrets:

  • are generated on demand
  • have limited access based on role
  • are leased for a period of time
  • can be revoked
  • come with an audit trail

Let's look at how to generate AWS credentials using the AWS backend.

AWS Credentials

Enable the AWS secrets backend:

bash-5.1# vault secrets enable -path=aws aws

Success! Enabled the aws secrets engine at: aws/

Authenticate:

bash-5.1# vault write aws/config/root access_key=foo secret_key=bar

Success! Data written to: aws/config/root

Make sure to replace foo and bar with your AWS access key id and secret key, respectively.

Create role:

bash-5.1# vault write aws/roles/ec2-read credential_type=iam_user policy_document=-<<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "Stmt1426528957000",
      "Effect": "Allow",
      "Action": [
        "ec2:*"
      ],
      "Resource": [
        "*"
      ]
    }
  ]
}
EOF

Success! Data written to: aws/roles/ec2-read

Here, we created a new role based on AmazonEC2ReadOnlyAccess, which is an AWS-managed policy. As the name suggests, it give users read-only access to the EC2 console; they cannot perform any actions or create new resources. You can also use an inline policy to create a custom role based on your individual needs. We'll look at an example of this shortly. Refer to the AWS Secrets Engine docs for more info.

Remember: Dynamic Secrets are generated only when they are requested (i.e., a web app requests access to S3). They are not available in the store before this.

Create a new set of credentials:

bash-5.1# vault read aws/creds/ec2-read

Key                Value
---                -----
lease_id           aws/creds/ec2-read/9KdO6J7KVBiSwOPEvwrqqALG
lease_duration     768h
lease_renewable    true
access_key         AKIAZ4DZAKZKEULSDW5A
secret_key         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
security_token     <nil>

You should now be able to see the user within the "Users" section on the IAM console on AWS:

Lo siento

Leases and Revocation

In this section, we'll take a quick look at how to define a custom lease period and revoke a secret before the end of that period.

Create a new AWS role:

bash-5.1# vault write aws/roles/foo credential_type=iam_user policy_document=-<<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "Stmt1426528957000",
      "Effect": "Allow",
      "Action": [
        "ec2:*"
      ],
      "Resource": [
        "*"
      ]
    }
  ]
}
EOF

Success! Data written to: aws/roles/foo

Take note of the lease_duration when you create a new AWS credential:

bash-5.1# vault read aws/creds/foo

Key                Value
---                -----
lease_id           aws/creds/foo/F0oBbnBIHEoz0ywVVtbuJB7r
lease_duration     768h
lease_renewable    true
access_key         AKIAZ4DZAKZKLJKB7CPX
secret_key         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
security_token     <nil>

What if you only wanted the lease period for all AWS IAM dynamic secrets to be 30 minutes?

bash-5.1# vault write aws/config/lease lease=1800s lease_max=1800s

In this example, since lease_max is the same as lease, you won't be able to renew the token. If you set the lease_max to 3600s, you'd be able to renew the lease once. For more, review the Tokens and Leases guide.

Create a new credential:

bash-5.1# vault read aws/creds/foo

Key                Value
---                -----
lease_id           aws/creds/foo/xQlJpKDS1ljE9Awz0aywXgbB
lease_duration     30m
lease_renewable    true
access_key         AKIAZ4DZAKZKJPL5OM5W
secret_key         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
security_token     <nil>

Want to quickly revoke this credential? Grab the lease_id and then run:

bash-5.1# vault lease revoke aws/creds/foo/xQlJpKDS1ljE9Awz0aywXgbB

Want to revoke all AWS creds?

bash-5.1# vault lease revoke -prefix aws/

Refer to the Lease, Renew, and Revoke guide for more info these concepts.

Consul Backend

Thus far, we've been using the Filesystem backend. This will not scale beyond a single server, so it does not take advantage of Vault's high availability. Fortunately, there are a number of other Storage backends, like the Consul backend, designed for distributed systems.

To set up Consul, start by updating the docker-compose.yml file:

version: '3.8'

services:

  vault:
    build:
      context: ./vault
      dockerfile: Dockerfile
    ports:
      - 8200:8200
    volumes:
      - ./vault/config:/vault/config
      - ./vault/policies:/vault/policies
      - ./vault/data:/vault/data
      - ./vault/logs:/vault/logs
    environment:
      - VAULT_ADDR=http://127.0.0.1:8200
      - VAULT_API_ADDR=http://127.0.0.1:8200
    command: server -config=/vault/config/vault-config.json
    cap_add:
      - IPC_LOCK
    depends_on:
      - consul

  consul:
    build:
      context: ./consul
      dockerfile: Dockerfile
    ports:
      - 8500:8500
    command: agent -server -bind 0.0.0.0 -client 0.0.0.0 -bootstrap-expect 1 -config-file=/consul/config/config.json
    volumes:
      - ./consul/config/consul-config.json:/consul/config/config.json
      - ./consul/data:/consul/data

Add a new directory in the project root called "consul", and then add a new Dockerfile to that newly created directory:

# base image
FROM alpine:3.14

# set consul version
ENV CONSUL_VERSION 1.10.2

# create a new directory
RUN mkdir /consul

# download dependencies
RUN apk --no-cache add \
      bash \
      ca-certificates \
      wget

# download and set up consul
RUN wget --quiet --output-document=/tmp/consul.zip https://releases.hashicorp.com/consul/${CONSUL_VERSION}/consul_${CONSUL_VERSION}_linux_amd64.zip && \
    unzip /tmp/consul.zip -d /consul && \
    rm -f /tmp/consul.zip && \
    chmod +x /consul/consul

# update PATH
ENV PATH="PATH=$PATH:$PWD/consul"

# add the config file
COPY ./config/consul-config.json /consul/config/config.json

# expose ports
EXPOSE 8300 8400 8500 8600

# run consul
ENTRYPOINT ["consul"]

Next, within the "consul" directory add two new directories: "config" and "data". Then, within "config", add a config file called consul-config.json:

{
  "datacenter": "localhost",
  "data_dir": "/consul/data",
  "log_level": "DEBUG",
  "server": true,
  "ui": true,
  "ports": {
    "dns": 53
  }
}

Asegúrese de revisar las opciones de configuración de los documentos de Consul para obtener más información sobre las opciones anteriores.

El directorio "cónsul" ahora debería verse así:

├── Dockerfile
├── config
│   └── consul-config.json
└── data

Salga de la sesión bash. Baje el contenedor y luego actualice el archivo de configuración de Vault:

{
  "backend": {
    "consul": {
      "address": "consul:8500",
      "path": "vault/"
    }
  },
  "listener": {
    "tcp":{
      "address": "0.0.0.0:8200",
      "tls_disable": 1
    }
  },
  "ui": true
}

Entonces, ahora estamos usando el backend de Consul en lugar del sistema de archivos. Usamos el nombre del servicio, consul, como parte de la dirección. La pathclave define la ruta en el almacén de clave/valor de Consul donde se almacenarán los datos de Vault.

Borre todos los archivos y carpetas dentro del directorio "bóveda/datos" para eliminar el backend del sistema de archivos. Cree las nuevas imágenes y haga girar los contenedores:

$ docker-compose down
$ docker-compose up -d --build

Asegúrese de que todo esté bien navegando en su navegador a http://localhost:8500/ui :

cónsul ui

Pruébelo desde la CLI o la interfaz de usuario.

CLI

Cree una nueva sesión de bash en el contenedor de Vault:

$ docker-compose exec vault bash

Entonces corre:

# Init
bash-5.1# vault operator init

# Unseal
bash-5.1# vault operator unseal

# Authenticate
bash-5.1# vault login

# Enable secrets
bash-5.1# vault secrets enable kv

# Add a new static secret
bash-5.1# vault kv put kv/foo bar=precious

# Read it back
bash-5.1# vault kv get kv/foo

interfaz de usuario

cónsul de bóveda

Observe cómo no hay archivos o carpetas dentro de "bóveda/datos". ¿Por qué crees que es esto?

¿Quiere agregar otro servidor Consul a la mezcla? Agregue un nuevo servicio a docker-compose.yml :

consul-worker:
  build:
    context: ./consul
    dockerfile: Dockerfile
  command: agent -server -join consul -config-file=/consul/config/config.json
  volumes:
    - ./consul/config/consul-config.json:/consul/config/config.json
  depends_on:
    - consul

Aquí, usamos el comando de unión para conectar este agente a un clúster existente. Observe cómo simplemente tuvimos que hacer referencia al nombre del servicio: consul.

Después:

  1. Salir de la sesión bash (si es necesario)
  2. Bajar los contenedores
  3. Limpiar el directorio de datos en "consul/data" (¿Por qué?)
  4. Haga girar los contenedores de nuevo y pruebe

cónsul ui

Conclusión

En este tutorial, repasamos cómo configurar y ejecutar Vault y Consul dentro de un contenedor Docker. Ahora debería tener una comprensión clara de cómo interactuar con Vault y realizar operaciones básicas.

Tome el código final del repositorio vault-consul-docker . Echa un vistazo a la presentación también.

Fuente:  https://testdriven.io

#vault #consul 

Cómo Configurar Y Utilizar Los Proyectos Vault Y Consul De Hashicorp
Neil  Morgan

Neil Morgan

1660603740

Como Configurar E Usar Os Projetos Vault E Consul Da Hashicorp

O tutorial a seguir detalha como configurar e usar os projetos Vault e Consul da Hashicorp para armazenar e gerenciar segredos com segurança.

Começaremos girando uma única instância do Vault em um contêiner do Docker e, em seguida, passaremos ao gerenciamento de segredos estáticos e dinâmicos junto com o recurso "criptografia como serviço" do Vault. Em seguida, adicionaremos o Consul à mistura e veremos como dimensionar o Vault.

Principais dependências:

  • Docker v20.10.8
  • Docker-Compose v1.29.2
  • Cofre v1.8.2
  • Cônsul v1.10.2

Objetivos

Ao final deste tutorial, você deverá ser capaz de:

  1. Explique o que é o Vault e por que você pode querer usá-lo
  2. Descrever a arquitetura básica do Vault juntamente com segredos dinâmicos e estáticos, os vários back-ends (armazenamento, segredo, autenticação, auditoria) e como o Vault pode ser usado como uma "criptografia como serviço"
  3. Configurar e executar o Vault e o Consul com o Docker
  4. Acelere o Vault com o back-end do sistema de arquivos
  5. Iniciar e deslacrar o Vault
  6. Autenticar no Vault
  7. Configure um back-end de auditoria para registrar todas as interações com o Vault
  8. Trabalhe com segredos estáticos e dinâmicos por meio da CLI, API HTTP e IU
  9. Crie uma política do Vault para limitar o acesso a um caminho específico
  10. Use o back-end do Transit como uma "criptografia como serviço"
  11. Configurar o Consul para trabalhar com o Vault como back-end de armazenamento para segredos
  12. Defina um período de concessão personalizado para um segredo e revogue um segredo antes do final desse período

O que é Cofre?

O Vault é uma ferramenta de código aberto usada para armazenar e gerenciar segredos com segurança.

O que é um segredo? Segredos, no contexto deste tutorial, são informações seguras ou de identificação pessoal, como credenciais de banco de dados, chaves SSH, nomes de usuário e senhas, credenciais do AWS IAM, tokens de API, números de CPF, números de cartão de crédito, apenas para citar alguns.

Reserve um momento para pensar em como sua equipe atualmente gerencia e distribui segredos:

  1. Quem tem acesso a eles?
  2. Quem os administra?
  3. Como você controla quem tem acesso a eles?
  4. Como seus aplicativos os obtêm?
  5. Como eles são atualizados?
  6. Como são revogados?

O Vault fornece respostas a essas perguntas e ajuda a resolver os seguintes problemas em relação ao gerenciamento de segredos:

ProblemasObjetivos do Vault
Os segredos estão por toda parte.O Vault é a única fonte de verdade para todos os segredos.
Eles geralmente não são criptografados.O Vault gerencia a criptografia (durante o trânsito e em repouso) imediatamente.
É difícil gerá-los dinamicamente.Os segredos podem ser gerados dinamicamente.
É ainda mais difícil arrendar e revogá-los.Os segredos podem ser alugados e revogados.
Não há trilha de auditoria.Há uma trilha de auditoria para gerar e usar segredos.

O Vault tem várias peças móveis, então pode levar algum tempo para se atualizar com a arquitetura geral. Reserve um momento para revisar o guia de arquitetura , observando os seguintes back-ends:

Processo internoUsarExemplos
ArmazenarOnde os segredos são armazenadosConsul *, Filesystem *, In-Memory, PostgreSQL, S3
SegredoLida com segredos estáticos ou dinâmicosAWS *, bancos de dados, chave/valor *, RabbitMQ, SSH
AutenticaçãoLida com autenticação e autorizaçãoAWS, Azure, Google Cloud, GitHub, Tokens *, nome de usuário e senha
AuditoriaRegistra todas as solicitações e respostasArquivo *, Syslog, Soquete

*usado neste tutorial

Com isso, vamos começar a usar o Vault.

Back-end do sistema de arquivos

Para começar a funcionar rapidamente, usaremos o back-end do Filesystem para armazenar segredos em repouso.

O back-end do sistema de arquivos deve ser usado apenas para desenvolvimento local ou implantação do Vault de servidor único, pois não suporta alta disponibilidade.

Crie um novo diretório de projeto:

$ mkdir vault-consul-docker && cd vault-consul-docker

Em seguida, adicione as seguintes pastas:

└── vault
    ├── config
    ├── data
    ├── logs
    └── policies

Adicione um Dockerfile ao diretório "vault":

# base image
FROM alpine:3.14

# set vault version
ENV VAULT_VERSION 1.8.2

# create a new directory
RUN mkdir /vault

# download dependencies
RUN apk --no-cache add \
      bash \
      ca-certificates \
      wget

# download and set up vault
RUN wget --quiet --output-document=/tmp/vault.zip https://releases.hashicorp.com/vault/${VAULT_VERSION}/vault_${VAULT_VERSION}_linux_amd64.zip && \
    unzip /tmp/vault.zip -d /vault && \
    rm -f /tmp/vault.zip && \
    chmod +x /vault

# update PATH
ENV PATH="PATH=$PATH:$PWD/vault"

# add the config file
COPY ./config/vault-config.json /vault/config/vault-config.json

# expose port 8200
EXPOSE 8200

# run vault
ENTRYPOINT ["vault"]

Em seguida, adicione um arquivo docker-compose.yml à raiz do projeto:

version: '3.8'

services:

  vault:
    build:
      context: ./vault
      dockerfile: Dockerfile
    ports:
      - 8200:8200
    volumes:
      - ./vault/config:/vault/config
      - ./vault/policies:/vault/policies
      - ./vault/data:/vault/data
      - ./vault/logs:/vault/logs
    environment:
      - VAULT_ADDR=http://127.0.0.1:8200
      - VAULT_API_ADDR=http://127.0.0.1:8200
    command: server -config=/vault/config/vault-config.json
    cap_add:
      - IPC_LOCK

Adicione um arquivo de configuração chamado vault-config.json a "vault/config":

{
  "backend": {
    "file": {
      "path": "vault/data"
    }
  },
  "listener": {
    "tcp":{
      "address": "0.0.0.0:8200",
      "tls_disable": 1
    }
  },
  "ui": true
}

Aqui, configuramos o Vault para usar o back-end do sistema de arquivos, definimos o ouvinte para o Vault, desabilitamos o TLS e habilitamos a interface do usuário do Vault . Revise os documentos para obter mais informações sobre como configurar o Vault.

Agora podemos construir a imagem e girar o contêiner:

$ docker-compose up -d --build

Puxe os logs do Docker para verificar se não houve erros na compilação:

$ docker-compose logs

Você deve ver algo semelhante a:

Attaching to vault-consul-docker_vault_1
vault_1  | ==> Vault server configuration:
vault_1  |
vault_1  |              Api Address: http://127.0.0.1:8200
vault_1  | 2021-09-08T14:48:35.014Z [INFO]  proxy environment: http_proxy="" https_proxy="" no_proxy=""
vault_1  |                      Cgo: disabled
vault_1  |          Cluster Address: https://127.0.0.1:8201
vault_1  |               Go Version: go1.16.7
vault_1  |               Listener 1: tcp (addr: "0.0.0.0:8200", cluster address: "0.0.0.0:8201", max_request_duration: "1m30s", max_request_size: "33554432", tls: "disabled")
vault_1  |                Log Level: info
vault_1  |                    Mlock: supported: true, enabled: true
vault_1  |            Recovery Mode: false
vault_1  |                  Storage: file
vault_1  |                  Version: Vault v1.8.2
vault_1  |              Version Sha: aca76f63357041a43b49f3e8c11d67358496959f
vault_1  |
vault_1  | ==> Vault server started! Log data will stream in below:
vault_1  |

Inicializando e deslacrando

Inicie uma sessão bash no contêiner em execução:

$ docker-compose exec vault bash

Dentro do shell, inicialize o Vault:

bash-5.1# vault operator init

Anote as chaves de unseal e o token raiz inicial. Você precisará fornecer três das chaves de remoção de lacre sempre que o servidor do Vault for relacrado ou reiniciado.

Por que 3 chaves? Revise o Compartilhamento Secreto de Shamir .

Agora você pode desbloquear o Vault usando três das chaves:

bash-5.1# vault operator unseal
Unseal Key (will be hidden):

Execute este comando mais duas vezes, usando chaves diferentes a cada vez. Uma vez feito, certifique- Sealedse de false:

Key             Value
---             -----
Seal Type       shamir
Initialized     true
Sealed          false
Total Shares    5
Threshold       3
Version         1.8.2
Storage Type    file
Cluster Name    vault-cluster-8fcf9d05
Cluster ID      d86e0274-ad9c-d2c1-d6ec-baeab410797b
HA Enabled      false

Usando o token raiz, agora você pode autenticar:

bash-5.1# vault login
Token (will be hidden):

Você deve ver algo semelhante a:

Success! You are now authenticated. The token information displayed below
is already stored in the token helper. You do NOT need to run "vault login"
again. Future Vault requests will automatically use this token.

Key                  Value
---                  -----
token                s.c0kYHWiOTqQvtR8JuSeTz6sZ
token_accessor       3FQJVxOY5C1brzlHHQSFaCdZ
token_duration       ∞
token_renewable      false
token_policies       ["root"]
identity_policies    []
policies             ["root"]

Tenha em mente que isso usa a política de raiz. Na produção, você desejará configurar políticas com diferentes níveis de acesso. Veremos como fazer isso em breve.

inicialização do cofre

O Vault agora está sem lacre e pronto para uso.

Auditoria

Antes de testarmos a funcionalidade, vamos habilitar um Dispositivo de Auditoria :

bash-5.1# vault audit enable file file_path=/vault/logs/audit.log

Success! Enabled the file audit device at: file/

Agora você deve poder visualizar os logs localmente em "cofre/logs". Para testar, execute o seguinte comando para visualizar todos os dispositivos de auditoria habilitados:

bash-5.1# vault audit list

Path     Type    Description
----     ----    -----------
file/    file    n/a

A solicitação e a resposta subsequente devem ser registradas em vault/logs/audit.log . Dê uma olhada.

Segredos

Existem dois tipos de segredos no Vault: estático e dinâmico .

Segredos estáticos (pense em Redis ou Memcached criptografados) têm intervalos de atualização, mas não expiram, a menos que sejam revogados explicitamente. Eles são definidos antecipadamente com o back-end de chave/valor (anteriormente o back-end "genérico") e, em seguida, compartilhados.

armazenamento secreto seguro

Os segredos dinâmicos são gerados sob demanda. Eles obrigaram os arrendamentos e geralmente expiram após um curto período de tempo. Como eles não existem até que sejam acessados, há menos exposição - portanto, os segredos dinâmicos são muito mais seguros. O Vault é fornecido com vários back-ends dinâmicos -- ou seja, AWS , Databases , Google Cloud , Consul e RabbitMQ .

Revise a postagem do blog Por que precisamos de segredos dinâmicos para obter mais informações sobre as vantagens de usar segredos dinâmicos.

Segredos estáticos

O Vault pode ser gerenciado por meio da CLI , HTTP API ou UI .

CLI

Ainda dentro da sessão bash no container, podemos criar, ler, atualizar e excluir segredos. Também veremos como versionar e reverter segredos.

Habilite segredos com o seguinte comando:

bash-5.1# vault secrets enable kv

Success! Enabled the kv secrets engine at: kv/

Crie um novo segredo com uma chave de bare um valor de preciousdentro do kv/foocaminho:

bash-5.1# vault kv put kv/foo bar=precious

Success! Data written to: kv/foo

Ler:

bash-5.1# vault kv get kv/foo

=== Data ===
Key    Value
---    -----
bar    precious

Para trabalhar com diferentes versões de uma chave específica, precisaremos atualizar para a v2 do back-end de chave/valor :

bash-5.1# vault kv enable-versioning kv/

Success! Tuned the secrets engine at: kv/

Adicione a versão 2 atualizando o valor para copper:

bash-5.1# vault kv put kv/foo bar=copper

Key              Value
---              -----
created_time     2021-09-08T18:23:14.4154928Z
deletion_time    n/a
destroyed        false
version          2

Leia a versão 1:

bash-5.1# vault kv get -version=1 kv/foo

====== Metadata ======
Key              Value
---              -----
created_time     2021-09-08T18:22:37.2548824Z
deletion_time    n/a
destroyed        false
version          1

=== Data ===
Key    Value
---    -----
bar    precious

Leia a versão 2:

bash-5.1# vault kv get -version=2 kv/foo

====== Metadata ======
Key              Value
---              -----
created_time     2021-09-08T18:23:14.4154928Z
deletion_time    n/a
destroyed        false
version          2

=== Data ===
Key    Value
---    -----
bar    copper

Exclua a versão mais recente (por exemplo, versão 2):

bash-5.1# vault kv delete kv/foo

Success! Data deleted (if it existed) at: kv/foo

Excluir versão 1:

bash-5.1# vault kv delete -versions=1 kv/foo

Success! Data deleted (if it existed) at: kv/foo

Você também pode recuperar:

bash-5.1# vault kv undelete -versions=1 kv/foo

Success! Data written to: kv/undelete/foo

Excluir é semelhante a uma exclusão reversível. Se você quiser remover os metadados subjacentes, terá que usar o comando destroy :

bash-5.1# vault kv destroy -versions=1 kv/foo

Success! Data written to: kv/destroy/foo

Revise v1 e v2 para visualizar todos os comandos disponíveis.

Anote o log de auditoria. Cada uma das solicitações acima foi registrada!

API

Você também pode interagir com o Vault por meio da API HTTP . Faremos solicitações na v2 da API. Abra uma nova guia de terminal e defina o token raiz como uma variável de ambiente:

$ export VAULT_TOKEN=your_token_goes_here

Crie um novo segredo chamado foocom um valor de world:

$ curl \
    -H "X-Vault-Token: $VAULT_TOKEN" \
    -H "Content-Type: application/json" \
    -X POST \
    -d '{ "data": { "foo": "world" } }' \
    http://127.0.0.1:8200/v1/kv/data/hello

Leia o segredo:

$ curl \
    -H "X-Vault-Token: $VAULT_TOKEN" \
    -X GET \
    http://127.0.0.1:8200/v1/kv/data/hello

A resposta JSON deve conter uma datachave com um valor semelhante a:

"data": {
  "data":{
    "foo": "world"
  },
  "metadata": {
    "created_time": "2021-09-08T18:30:32.5140484Z",
    "deletion_time": "",
    "destroyed": false,
    "version": 1
  }
}

API de cofre

Tente adicionar novas versões, excluir e destruir por conta própria.

IU

A interface do usuário deve estar em execução em http://localhost:8200/ui/vault . Use o token raiz para fazer login. Em seguida, explore o back-end de chave/valor por conta própria:

interface do cofre

Políticas

Até agora, usamos a política de raiz para interagir com a API. Vamos configurar uma política que tenha apenas acesso de leitura.

Adicione um novo arquivo de configuração chamado app-policy.json a "vault/policies":

{
  "path": {
    "kv/data/app/*": {
      "policy": "read"
    }
  }
}

Crie uma nova política de volta na sessão bash:

bash-5.1# vault policy write app /vault/policies/app-policy.json

Success! Uploaded policy: app

Em seguida, crie um novo token:

bash-5.1# vault token create -policy=app

Key                  Value
---                  -----
token                s.ZOUMx3RIhVRhI4ijlZg8KXRQ
token_accessor       TT53xOxbIfGjI7l4392gjXcg
token_duration       768h
token_renewable      true
token_policies       ["app" "default"]
identity_policies    []
policies             ["app" "default"]

Dentro de outra nova guia do terminal (agora você deve ter três), adicione a VAULT_TOKENvariável de ambiente com o novo token:

$ export VAULT_TOKEN=your_token_goes_here

Tente ler o foosegredo que definimos anteriormente:

$ curl \
    -H "X-Vault-Token: $VAULT_TOKEN" \
    -X GET \
    http://127.0.0.1:8200/v1/kv/data/hello

Você não deve ter as permissões corretas para visualizar esse segredo:

{
  "errors":[
    "1 error occurred:\n\t* permission denied\n\n"
  ]
}

Por que não podemos nem ler? Volte para a configuração de política em vault-config.json . kv/data/app/*indica que a política só pode ler do appcaminho.

Como você provavelmente já notou, quase tudo no Vault é baseado em caminhos.

De volta à sessão bash no contêiner, adicione um novo segredo ao app/testcaminho:

bash-5.1# vault kv put kv/app/test ping=pong

Key              Value
---              -----
created_time     2021-09-08T18:40:35.2694047Z
deletion_time    n/a
destroyed        false
version          1

Você deve conseguir visualizar o segredo usando o token associado à apppolítica:

$ curl \
    -H "X-Vault-Token: $VAULT_TOKEN" \
    -X GET \
    http://127.0.0.1:8200/v1/kv/data/app/test

As políticas também podem ser gerenciadas na interface do usuário:

interface do cofre

Criptografia como um serviço

Antes de analisarmos os segredos dinâmicos, vamos analisar rapidamente o back-end do Transit , que pode ser usado como uma "criptografia como serviço" para:

  • Criptografar e descriptografar dados "em trânsito" sem armazená-los no Vault
  • Integrando facilmente a criptografia ao fluxo de trabalho do seu aplicativo

De volta à sessão bash no contêiner, ative o Transit:

bash-5.1# vault secrets enable transit

Success! Enabled the transit secrets engine at: transit/

Configure uma chave de criptografia nomeada:

bash-5.1# vault write -f transit/keys/foo

Success! Data written to: transit/keys/foo

Criptografar:

bash-5.1# vault write transit/encrypt/foo plaintext=$(base64 <<< "my precious")

Key           Value
---           -----
ciphertext    vault:v1:cFnk5AQLE9Mg+mZ7Ej17vRmYT5aqheikdZQ1FC4vre5jAod0L/uHDA==

Descriptografar:

bash-5.1# vault write transit/decrypt/foo ciphertext=vault:v1:cFnk5AQLE9Mg+mZ7Ej17vRmYT5aqheikdZQ1FC4vre5jAod0L/uHDA==

Key          Value
---          -----
plaintext    bXkgcHJlY2lvdXMK

Decodificar:

bash-5.1# base64 -d <<< "bXkgcHJlY2lvdXMK"

my precious

Teste-o também na interface do usuário:

interface do cofre

Segredos dinâmicos

Conforme mencionado, o Vault oferece suporte a vários back-ends de segredo dinâmicos para gerar segredos dinamicamente quando necessário. Por exemplo, com os back-ends da AWS e do Google Cloud , você pode criar credenciais de acesso com base nas políticas do IAM. Enquanto isso, o back-end de bancos de dados gera credenciais de banco de dados com base nas funções configuradas.

Segredos dinâmicos:

  • são gerados sob demanda
  • têm acesso limitado com base na função
  • são alugados por um período de tempo
  • pode ser revogado
  • vem com uma trilha de auditoria

Vejamos como gerar credenciais da AWS usando o back-end da AWS.

Credenciais da AWS

Habilite o back-end de segredos da AWS:

bash-5.1# vault secrets enable -path=aws aws

Success! Enabled the aws secrets engine at: aws/

Autenticar:

bash-5.1# vault write aws/config/root access_key=foo secret_key=bar

Success! Data written to: aws/config/root

Certifique-se de substituir fooe barpelo ID da chave de acesso e chave secreta da AWS, respectivamente.

Criar função:

bash-5.1# vault write aws/roles/ec2-read credential_type=iam_user policy_document=-<<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "Stmt1426528957000",
      "Effect": "Allow",
      "Action": [
        "ec2:*"
      ],
      "Resource": [
        "*"
      ]
    }
  ]
}
EOF

Success! Data written to: aws/roles/ec2-read

Aqui, criamos uma nova função baseada em , que é uma políticaAmazonEC2ReadOnlyAccess gerenciada pela AWS . Como o nome sugere, ele dá aos usuários acesso somente leitura ao console do EC2; eles não podem executar nenhuma ação ou criar novos recursos. Você também pode usar uma política em linha para criar uma função personalizada com base em suas necessidades individuais. Veremos um exemplo disso em breve. Consulte os documentos do AWS Secrets Engine para obter mais informações.

Lembre -se : Segredos dinâmicos são gerados apenas quando solicitados (ou seja, um aplicativo da Web solicita acesso ao S3). Eles não estão disponíveis na loja antes disso.

Crie um novo conjunto de credenciais:

bash-5.1# vault read aws/creds/ec2-read

Key                Value
---                -----
lease_id           aws/creds/ec2-read/9KdO6J7KVBiSwOPEvwrqqALG
lease_duration     768h
lease_renewable    true
access_key         AKIAZ4DZAKZKEULSDW5A
secret_key         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
security_token     <nil>

Agora você deve conseguir ver o usuário na seção "Usuários" no console do IAM na AWS:

Sinto muito

Arrendamentos e Revogação

Nesta seção, veremos rapidamente como definir um período de concessão personalizado e revogar um segredo antes do final desse período.

Crie uma nova função da AWS:

bash-5.1# vault write aws/roles/foo credential_type=iam_user policy_document=-<<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "Stmt1426528957000",
      "Effect": "Allow",
      "Action": [
        "ec2:*"
      ],
      "Resource": [
        "*"
      ]
    }
  ]
}
EOF

Success! Data written to: aws/roles/foo

Observe lease_durationquando você cria uma nova credencial da AWS:

bash-5.1# vault read aws/creds/foo

Key                Value
---                -----
lease_id           aws/creds/foo/F0oBbnBIHEoz0ywVVtbuJB7r
lease_duration     768h
lease_renewable    true
access_key         AKIAZ4DZAKZKLJKB7CPX
secret_key         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
security_token     <nil>

E se você quisesse que o período de concessão de todos os segredos dinâmicos do AWS IAM fosse de 30 minutos?

bash-5.1# vault write aws/config/lease lease=1800s lease_max=1800s

Neste exemplo, como lease_maxé igual a lease, você não poderá renovar o token. Se você definir lease_maxcomo 3600s, poderá renovar a concessão uma vez. Para saber mais, consulte o guia Tokens e Leases .

Crie uma nova credencial:

bash-5.1# vault read aws/creds/foo

Key                Value
---                -----
lease_id           aws/creds/foo/xQlJpKDS1ljE9Awz0aywXgbB
lease_duration     30m
lease_renewable    true
access_key         AKIAZ4DZAKZKJPL5OM5W
secret_key         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
security_token     <nil>

Quer revogar rapidamente esta credencial? Pegue o lease_ide depois execute:

bash-5.1# vault lease revoke aws/creds/foo/xQlJpKDS1ljE9Awz0aywXgbB

Quer revogar todas as credenciais da AWS?

bash-5.1# vault lease revoke -prefix aws/

Consulte o guia Lease, Renew e Revoke para obter mais informações sobre esses conceitos.

Back-end do cônsul

Até agora, estamos usando o backend Filesystem . Isso não será dimensionado além de um único servidor, portanto, não aproveita a alta disponibilidade do Vault. Felizmente, existem vários outros back-ends de armazenamento , como o back-end Consul , projetados para sistemas distribuídos.

Para configurar o Consul , comece atualizando o arquivo docker-compose.yml :

version: '3.8'

services:

  vault:
    build:
      context: ./vault
      dockerfile: Dockerfile
    ports:
      - 8200:8200
    volumes:
      - ./vault/config:/vault/config
      - ./vault/policies:/vault/policies
      - ./vault/data:/vault/data
      - ./vault/logs:/vault/logs
    environment:
      - VAULT_ADDR=http://127.0.0.1:8200
      - VAULT_API_ADDR=http://127.0.0.1:8200
    command: server -config=/vault/config/vault-config.json
    cap_add:
      - IPC_LOCK
    depends_on:
      - consul

  consul:
    build:
      context: ./consul
      dockerfile: Dockerfile
    ports:
      - 8500:8500
    command: agent -server -bind 0.0.0.0 -client 0.0.0.0 -bootstrap-expect 1 -config-file=/consul/config/config.json
    volumes:
      - ./consul/config/consul-config.json:/consul/config/config.json
      - ./consul/data:/consul/data

Adicione um novo diretório na raiz do projeto chamado "consul" e adicione um novo Dockerfile nesse diretório recém-criado:

# base image
FROM alpine:3.14

# set consul version
ENV CONSUL_VERSION 1.10.2

# create a new directory
RUN mkdir /consul

# download dependencies
RUN apk --no-cache add \
      bash \
      ca-certificates \
      wget

# download and set up consul
RUN wget --quiet --output-document=/tmp/consul.zip https://releases.hashicorp.com/consul/${CONSUL_VERSION}/consul_${CONSUL_VERSION}_linux_amd64.zip && \
    unzip /tmp/consul.zip -d /consul && \
    rm -f /tmp/consul.zip && \
    chmod +x /consul/consul

# update PATH
ENV PATH="PATH=$PATH:$PWD/consul"

# add the config file
COPY ./config/consul-config.json /consul/config/config.json

# expose ports
EXPOSE 8300 8400 8500 8600

# run consul
ENTRYPOINT ["consul"]

Em seguida, dentro do diretório "consul" adicione dois novos diretórios: "config" e "data". Em seguida, dentro de "config", adicione um arquivo de configuração chamado consul-config.json :

{
  "datacenter": "localhost",
  "data_dir": "/consul/data",
  "log_level": "DEBUG",
  "server": true,
  "ui": true,
  "ports": {
    "dns": 53
  }
}

Certifique-se de revisar as opções de configuração dos documentos do Consul para obter mais informações sobre as opções acima.

O diretório "consul" agora deve se parecer com:

├── Dockerfile
├── config
│   └── consul-config.json
└── data

Saia da sessão do bash. Desative o contêiner e atualize o arquivo de configuração do Vault:

{
  "backend": {
    "consul": {
      "address": "consul:8500",
      "path": "vault/"
    }
  },
  "listener": {
    "tcp":{
      "address": "0.0.0.0:8200",
      "tls_disable": 1
    }
  },
  "ui": true
}

Então, agora estamos usando o backend Consul em vez do Filesystem. Usamos o nome do serviço, consul, como parte do endereço. A pathchave define o caminho no armazenamento de chave/valor do Consul onde os dados do Vault serão armazenados.

Limpe todos os arquivos e pastas dentro do diretório "vault/data" para remover o backend do sistema de arquivos. Construa as novas imagens e gire os contêineres:

$ docker-compose down
$ docker-compose up -d --build

Certifique-se de que tudo está bem navegando em seu navegador para http://localhost:8500/ui :

cônsul ui

Teste isso na CLI ou na interface do usuário.

CLI

Crie uma nova sessão bash no contêiner do Vault:

$ docker-compose exec vault bash

Então corra:

# Init
bash-5.1# vault operator init

# Unseal
bash-5.1# vault operator unseal

# Authenticate
bash-5.1# vault login

# Enable secrets
bash-5.1# vault secrets enable kv

# Add a new static secret
bash-5.1# vault kv put kv/foo bar=precious

# Read it back
bash-5.1# vault kv get kv/foo

IU

cônsul do cofre

Observe como não há arquivos ou pastas dentro de "cofre/dados". Por que você acha que é isso?

Quer adicionar outro servidor Consul à mistura? Adicione um novo serviço ao docker-compose.yml :

consul-worker:
  build:
    context: ./consul
    dockerfile: Dockerfile
  command: agent -server -join consul -config-file=/consul/config/config.json
  volumes:
    - ./consul/config/consul-config.json:/consul/config/config.json
  depends_on:
    - consul

Aqui, usamos o comando join para conectar esse agente a um cluster existente. Observe como simplesmente tivemos que referenciar o nome do serviço: consul.

Então:

  1. Sair da sessão bash (se necessário)
  2. Derrube os recipientes
  3. Limpe o diretório de dados em "consul/data" (Por quê?)
  4. Gire os contêineres de volta e teste

cônsul ui

Conclusão

Neste tutorial, vimos como configurar e executar o Vault e o Consul dentro de um contêiner do Docker. Agora você deve ter uma compreensão clara de como interagir com o Vault e realizar operações básicas.

Pegue o código final do repositório vault-consul-docker . Confira também a apresentação .

Fonte:  https://testdrive.io

#vault #consul 

Como Configurar E Usar Os Projetos Vault E Consul Da Hashicorp

Consul: A Distributed, Highly Available, & Data Center Aware Solution

Consul

Consul is a distributed, highly available, and data center aware solution to connect and configure applications across dynamic, distributed infrastructure.

Consul provides several key features:

Multi-Datacenter - Consul is built to be datacenter aware, and can support any number of regions without complex configuration.

Service Mesh/Service Segmentation - Consul Connect enables secure service-to-service communication with automatic TLS encryption and identity-based authorization. Applications can use sidecar proxies in a service mesh configuration to establish TLS connections for inbound and outbound connections without being aware of Connect at all.

Service Discovery - Consul makes it simple for services to register themselves and to discover other services via a DNS or HTTP interface. External services such as SaaS providers can be registered as well.

Health Checking - Health Checking enables Consul to quickly alert operators about any issues in a cluster. The integration with service discovery prevents routing traffic to unhealthy hosts and enables service level circuit breakers.

Key/Value Storage - A flexible key/value store enables storing dynamic configuration, feature flagging, coordination, leader election and more. The simple HTTP API makes it easy to use anywhere.

Consul runs on Linux, macOS, FreeBSD, Solaris, and Windows and includes an optional browser based UI. A commercial version called Consul Enterprise is also available.

Please note: We take Consul's security and our users' trust very seriously. If you believe you have found a security issue in Consul, please responsibly disclose by contacting us at security@hashicorp.com.

Quick Start

A few quick start guides are available on the Consul website:

Documentation

Full, comprehensive documentation is available on the Consul website:

https://www.consul.io/docs

Contributing

Thank you for your interest in contributing! Please refer to CONTRIBUTING.md for guidance. For contributions specifically to the browser based UI, please refer to the UI's README.md for guidance.



Download Details:

Author: Hashicorp
Source Code: https://github.com/hashicorp/consul 
License: MPL-2.0 license

#go #golang #kubernetes #consul #vault 

Consul: A Distributed, Highly Available, & Data Center Aware Solution

Top 14 Most Important Kubernetes Service Meshes

Service meshes solve the traffic management, security, and observability challenges introduced by microservices and distributed architecture. IstioLinkerd, and Consul are three matured tools with highly customizable and advanced features. At the same time, other tools highlighted in this article also offer lightweight and efficient capabilities to manage traffic management in microservices.

Microservice, Kubernetes, and distributed architecture have become a norm for scalable applications and enterprise business solutions. With distributed services, managing the communication and networking between each service is difficult — and service meshes prove to help solve it.

Service Meshes uses a proxy to provide a dedicated infrastructure layer for facilitating and securing service-to-service communication between microservices.

Service meshes’ typical architecture uses a sidecar container in each service to provide seamless communication, configuration, and security. To achieve this, many of the service meshes use an open source sidecar proxyEnvoy, while some others use custom-built proxies or monolithic architecture.

Let’s explore some of the top open source and commercial service mesh implementations available in the service mesh tooling market.

  1. Istio
  2. Linkerd
  3. AWS App Mesh
  4. Consul Connect
  5. Kuma
  6. Traefik Mesh
  7. Apache ServiceComb
  8. Network Service Mesh (NSM)
  9. Kiali Operator
  10. NGINX Service Mesh
  11. Aspen Mesh
  12. Open Service Mesh (OSM)
  13. Grey Matter
  14. OpenShift Service Mesh

#kubernetes #linkerd #istio #consul

Top 14 Most Important Kubernetes Service Meshes

Consul deployment patterns – A brief overview

While delving into service mesh, key-value store, or a service discovery solution in cloud-native space, you would have definitely come across Consul. Consul, developed by HashiCorp is a multi-purpose solution which primarily provides the following features:

  • Service discovery and Service Mesh features with Kubernetes
  • Secure communication and observability between the services
  • Automate load-balancing
  • Key-Value store
  • Consul watches

This blog post briefly explains what are the deployment patterns for the Consul to watch configuration changes stored in the Key-Value store. It will explain how to discover and synchronize with the services running out of the Kubernetes cluster.

#devops #servicemesh #consul

Consul deployment patterns – A brief overview
Owen  Lemke

Owen Lemke

1620441720

Consul Deployment Patterns: A Brief Overview

We provide some insights into deploying HashiCorp’s Consul cloud solution for monitoring Key-Value changes, Hybrid Workloads, and Service Mesh.

If you’ve ever delved into a service mesh, key-value store, or service discovery solution in the cloud-native space, you have definitely come across Consul. Consul, developed by HashiCorp, is a multi-purpose solution which primarily provides the following features:

  • Service discovery and Service Mesh features with Kubernetes.
  • Secure communication and observability between the services.
  • Automate load-balancing.
  • Key-Value store.
  • Consul watches.

This blog post briefly explains the deployment patterns for Consul to use when making configuration changes that are stored in the Key-Value store. It will explain how to discover and synchronize with the services running out of the Kubernetes cluster. We will also see how to enable Service Mesh features with Consul. We broadly categorize Consul deployment patterns as in-cluster patterns (Consul deployed in a Kubernetes cluster) and hybrid patterns (Consul deployed outside a Kubernetes cluster).

#cloud #kubernetes #cloud native #service mesh #observability #consul #deployment pattern

Consul Deployment Patterns: A Brief Overview
Mike  Kozey

Mike Kozey

1618574820

Service Discovery: Consul Support in Judge-d (Part 1)

We introduce you (both in theory and in practice) to the subject of service discovery, Consul as a sample tool implementing it, and how it can be used with Judge-d.

Abstract

Service discovery is a mechanism allowing to automatically detect services in complex ecosystems utilizing microservices architecture. Consul is one of the most popular tools offering service discovery. Therefore, it is a perfect match for Judge-d, an open source tool providing contract testing, to become one of the available sources of information about the state of the environment, which is under Judge-d jurisdiction.

In this article, we would like to introduce you (both in theory and in practice) to the subject of service discovery, Consul as a sample tool implementing it, and how it can be used with Judge-d as a source of truth about the environment where contract tests are required.

#tutorial #microservices #consul #judge d

Service Discovery: Consul Support in Judge-d (Part 1)
Lia  Haley

Lia Haley

1598803020

Building a Service Mesh with Consul

Load balancers aren’t efficient in a dynamic environment where we scale services up or down. Consul uses a registry to keep a real-time list of services, their location, and their health. Services query the registry to discover the location of upstream services and then connect directly. This allows services to scale up/down and gracefully handle failure

Installation

To Install Consul on Ubuntu

apt-get update
cd /usr/local/bin
wget https://releases.hashicorp.com/consul/1.2.2/consul_1.2.2_linux_amd64.zip
wget https://releases.hashicorp.com/consul/1.2.2/consul_1.2.2_SHA256SUMS
sha256sum consul_1.2.2_linux_amd64.zip
grep "consul_1.2.2_linux_amd64.zip" consul_1.2.2_SHA256SUMS
rm consul_1.2.2_SHA256SUMS
unzip *.zip
rm *.zip

## Refer to https://www.consul.io/docs/agent/options.html#ports for required ports and optional
ufw allow 8300
ufw allow 8301
ufw allow 8302
ufw allow 8400
ufw allow 8500
ufw allow 8600

mkdir /tmp/consul_services
mkdir /tmp/consul

Run Consul for Development

Create a system service for consul nano /etc/systemd/system/consul.service

[Unit]
Description=Consul
Documentation=https://www.consul.io/

[Service]
ExecStart=/usr/local/bin/consul agent -dev -ui -client $CLIENT_SERVER_IP_HOST
ExecReload=/bin/kill -HUP $MAINPID
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

Then

systemctl daemon-reload
systemctl start consul.service

#scaling #service discovery #consul #amazon web services

Building a Service Mesh with Consul
Norbert  Ernser

Norbert Ernser

1595345760

Getting Started With Consul Service Discovery

Consul is a tool for discovering and configuring services in your infrastructure. It provides several key features like service discovery, health checking, KV store and multi datacenter support.

Consul is designed to be friendly to both the DevOps community and application developers, making it perfect for modern, elastic infrastructures.

Install Consul & Build Servers

First lets create four servers and install apache on three of them or any other services since we will do a service check in this article.

Let’s say we have the four servers as follow:

  • Server Name is consul-01 and has a public IP 111.111.111.111
  • Server Name is consul-02 and has a public IP 222.222.222.222
  • Server Name is consul-03 and has a public IP 333.333.333.333
  • Server Name is consul-04 and has a public IP 444.444.444.444

Now we need to install consul on all these servers:

apt-get update
apt-get install unzip
cd /usr/local/bin
wget https://releases.hashicorp.com/consul/1.0.2/consul_1.0.2_linux_amd64.zip
unzip *.zip
rm *.zip

Feel free to use the appropriate consul version, Here we use 1.0.2. Also consul use some ports so you need to open these ports or you can configure these ports but this out of the scope of this part right now. For now let’s just open them and later we will talk about why these ports needed and how to configure:

sudo ufw allow 8300
sudo ufw allow 8301
sudo ufw allow 8302
sudo ufw allow 8400
sudo ufw allow 8500
sudo ufw allow 8600

Once you finihed, you can call consul command and get a help stuff:

$ consul
Usage: consul [--version] [--help] <command> [<args>]

Available commands are:
    agent          Runs a Consul agent
    catalog        Interact with the catalog

#linux #microservices #scaling #consul #health-checking #kv-support #microservices #service-discovery

Getting Started With Consul Service Discovery